Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-7101

Event stream message deployment_success represents the entire root, which is enormous

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Event Stream
    • Labels:

      Description

      While troubleshooting an issue with Marathon consuming massive CPU resources, we determined it was falling behind processing the event stream.  While analyzing the event stream, we observed that the deployment_success message was 1.8MB in size and appeared to represent the root of the entire Marathon system.    We logged 88 such messages in a 10 minute time period when no deployments were occurring, to our knowledge, and I suspect Marathon's high CPU utilization is due to trying to parse these messages.

      For reference, here is the beginning of one such message, annotated:

       

      edited to add:   deployment_info events also seem to have the highest node as the payload.

      event: deployment_success

      {

        "id": "6b91ca71-21ef-485a-b81c-73275272d7d5",

        "plan": {

          "id": "6b91ca71-21ef-485a-b81c-73275272d7d5",

          "original": {

            "id": "/",      <<< this right here just seems unnecessary and excessive.

            "apps": [

       

            ],

            "groups": [

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                carltonbrown carltonbrown
                Reporter:
                carltonbrown carltonbrown
                Team:
                Orchestration Team
                Watchers:
                carltonbrown, Jason Gilanfarr (Inactive)
              • Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: