Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-7358

Deployments performance degradation in 1.4

    Details

    • Sprint:
      Marathon Sprint 1.10-5, Marathon Sprint 1.10-6, Marathon Sprint 1.10-7, Marathon Sprint 1.10-9, Marathon Sprint 1.10-10, Marathon Sprint 1.11-1, Marathon Sprint 1.11-2, Marathon Sprint 1.11-3
    • Story Points:
      13

      Description

      Marathon 1.4.3 has huge degradation in handling groups and tasks. In 1.3.10 deploying 1k apps or more in one request works fine. No hiccups or slowness unfortunately in Marathon 1.4.3 it's not working. Deployment fails due to timeouts. API returns 503 with no information about deployment ID.

       Marathon 1.4.3

      ➜ ~ curl -X PUT -d@y.json localhost:8080/v2/groups -v
      * Trying 127.0.0.1...
      * Connected to localhost (127.0.0.1) port 8080 (#0)
      > PUT /v2/groups HTTP/1.1
      > Host: localhost:8080
      > User-Agent: curl/7.47.0
      > Accept: */*
      > Content-Length: 646099
      > Content-Type: application/x-www-form-urlencoded
      > Expect: 100-continue
      >
      < HTTP/1.1 100 Continue
      * We are completely uploaded and fine
      < HTTP/1.1 503 Service Unavailable
      < Date: Fri, 12 May 2017 19:07:10 GMT
      < X-Marathon-Leader: http://localhost:8080
      < Cache-Control: no-cache, no-store, must-revalidate
      < Pragma: no-cache
      < Expires: 0
      < Content-Type: application/json
      < Transfer-Encoding: chunked
      < Server: Jetty(9.3.z-SNAPSHOT)
      <
      * Connection #0 to host localhost left intact
      {"message":"Futures timed out after [10000 milliseconds]"}%
      

      Marathon 1.3.10

       ➜ ~ curl -X PUT -d@y.json localhost:8080/v2/groups -v
      * Trying 127.0.0.1...
      * Connected to localhost (127.0.0.1) port 8080 (#0)
      > PUT /v2/groups HTTP/1.1
      > Host: localhost:8080
      > User-Agent: curl/7.47.0
      > Accept: */*
      > Content-Length: 646099
      > Content-Type: application/x-www-form-urlencoded
      > Expect: 100-continue
      >
      < HTTP/1.1 100 Continue
      * We are completely uploaded and fine
      < HTTP/1.1 200 OK
      < Date: Fri, 12 May 2017 19:14:54 GMT
      < X-Marathon-Leader: http://localhost:8080
      < Cache-Control: no-cache, no-store, must-revalidate
      < Pragma: no-cache
      < Expires: 0
      < Content-Type: application/json; qs=2
      < Transfer-Encoding: chunked
      < Server: Jetty(9.3.z-SNAPSHOT)
      <
      * Connection #0 to host localhost left intact
      {"version":"2017-05-12T19:14:55.771Z","deploymentId":"5caa235f-afa1-4108-abd3-baec027828e4"}% 
      

      y.json is created from following template using http://beta.json-generator.com/V1Aehlyl7

      {"id": "/",
        "groups":
      [{
        'repeat(10)': {
        "id": '{{surname().toLowerCase()}}',
        "groups": [{
        'repeat(10)': {
        "id" : '{{surname().toLowerCase()}}',
        "apps":[{
          'repeat(10)': {
            "id": '{{firstName().toLowerCase()}}',
            "cmd": "python -m SimpleHTTPServer $PORT0",
            "cpus": 0.01,
            "mem": 0,
            "disk": 0,
            "instances": '{{integer(1, 1)}}',
            "labels": {
              "consul": '{{company().toLowerCase()}}'
            },
            "healthChecks": [
              {
                "protocol": "HTTP",
                "path": "/",
                "portIndex": 0,
                "gracePeriodSeconds": 30,
                "intervalSeconds": 5,
                "timeoutSeconds": 20,
                "maxConsecutiveFailures": 3
              }
            ]
      	}
        }]
        }}]
      }}]
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                junterstein Johannes Unterstein
                Reporter:
                janisz janisz
                Team:
                Orchestration Team
                Watchers:
                Aleksey Dukhovniy, Bekir Dogan, Chmielewski, Ioannis Charalampidis, janisz, Johannes Unterstein, Marco Monaco, Matthias Eichstedt, ndigati, tgermain, Tim Harper, Tobias Mueller
              • Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: