Contract change of marathon api call http://marathon.mesos:8080/v2/apps?embed=apps.tasks breaks marathon-lb app frontend-backend mapping. A being-deployed instance could be treated as healthy and put into haproxy.cfg with new host:port and causes haproxy Layer4 error and prevents the access to the service until the next health check as specified in the marathon service definition.
Working Version (DC/OS 1.8.6, Marathon 1.3.2, marathon-lb v1.12.1)
Breaking Version (DC/OS 1.11.2, Marathon 1.6.392, marathon-lb v1.12.1)
for an unhealthy instance, api "tasks/task/healthCheckResults" entry is changed from
in marathon 1.3.2
in marathon 1.6.392
This contract change breaks the marathon-lb marathon_lb.py
if 'healthCheckResults' not in task: at line 1524 and 1638 and cause unhealthy instance treated as healthy.
The possible fix is to change the line to
if 'healthCheckResults' not in task or not task['healthCheckResults']:
to handle both no-key and empty list case.
Please let us know if this is new bug that we should fix in the mentioned version.