Uploaded image for project: 'DC/OS'
  1. DC/OS
  2. DCOS_OSS-3506

Marathon-LB crash on empty or corrupted response from apps API

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: High
    • Resolution: Invalid
    • Affects Version/s: DC/OS 1.8.6
    • Fix Version/s: None
    • Component/s: marathon-lb

      Description

      Hi.

      We are running Marathon-LB 1.6.0 on DC/OS 1.8.6
      Recently we've been experiencing issues with Marathon-LB. Looks like it crashes on an empty or malformed response from ` master.mesos:8080/v2/apps?embed=apps.tasks` endpoint.

      2018-03-14 20:05:11,023 marathon_lb: skipping reload: config/map unchanged
      2018-03-14 20:05:11,023 marathon_lb: updating tasks finished, took 0.2349224090576172 seconds
      2018-03-14 20:05:47,440 marathon_lb: fetching apps
      2018-03-14 20:05:50,068 marathon_lb: SSE Active, trying fetch events from http://master.mesos:8080/v2/events
      2018-03-14 20:05:52,458 marathon_lb: GET http://master.mesos:8080/v2/apps?embed=apps.tasks
      2018-03-14 20:05:52,459 marathon_lb: Unexpected error!
      Traceback (most recent call last):
        File "/marathon-lb/marathon_lb.py", line 1510, in do_reset
          self.__apps = get_apps(self.__marathon)
        File "/marathon-lb/marathon_lb.py", line 1220, in get_apps
          apps = marathon.list()
        File "/marathon-lb/marathon_lb.py", line 214, in list
          params={'embed': 'apps.tasks'})["apps"]
        File "/marathon-lb/marathon_lb.py", line 201, in api_req
          verify=self.__verify, **kwargs).json()
        File "/marathon-lb/marathon_lb.py", line 192, in api_req_raw
          if 'message' in response.json():
        File "/usr/local/lib/python3.5/dist-packages/requests/models.py", line 866, in json
          return complexjson.loads(self.text, **kwargs)
        File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
          return _default_decoder.decode(s)
        File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
          obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
          raise JSONDecodeError("Expecting value", s, err.value) from None
      json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
      2018-03-14 20:05:52,460 marathon_lb: starting event processor thread
      2018-03-14 20:05:55,082 marathon_lb: fetching apps
      Received killTask for task marathon-lb.4ea5f45e-2373-11e8-9744-661d6e9bf771
      

      Could you please take a look at it? Thanks.

        Attachments

          Activity

            People

            • Assignee:
              jasonkoelker Jason Koelker
              Reporter:
              aleks-m aleks-m
              Team:
              Networking Team
              Watchers:
              aleks-m, Deepak Goel, Justin Lee (Inactive), Orlando Hohmeier
            • Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: