Details

    • Sprint:
      Networking Team 1.12 Sprint 2, Networking Team 1.12 Sprint 3, Networking Team 1.12 Sprint 4, Networking Team 1.12 Sprint 5, Networking Team 1.12 Sprint 6, Networking Team 1.12 Sprint 7
    • Story Points:
      1

      Description

      My Public Agent with MarathonLB seems to fail a health check and then restarts but it never restarts it stays unhealthy. 

      DCOS - 1.10

      MarathonLB - 1.11.1 

      Error logs: 

      [/marathon-lb /marathon-lb/run] 80,443,9090,9091,10000,10001,10002,10003,10004,10005,10006,10007,10008,10009,10010,10011,10012,10013,10014,10015,10016,10017,10018,10019,10020,10021,10022,10023,10024,10025,10026,10027,10028,10029,10030,10031,10032,10033,10034,10035,10036,10037,10038,10039,10040,10041,10042,10043,10044,10045,10046,10047,10048,10049,10050,10051,10052,10053,10054,10055,10056,10057,10058,10059,10060,10061,10062,10063,10064,10065,10066,10067,10068,10069,10070,10071,10072,10073,10074,10075,10076,10077,10078,10079,10080,10081,10082,10083,10084,10085,10086,10087,10088,10089,10090,10091,10092,10093,10094,10095,10096,10097,10098,10099,10100 > /marathon-lb/service/haproxy/env/PORTS
      [/marathon-lb /marathon-lb/run] setting sysctl params to: net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_fin_timeout=30 net.ipv4.tcp_max_syn_backlog=10240 net.ipv4.tcp_max_tw_buckets=400000 net.ipv4.tcp_max_orphans=60000 net.core.somaxconn=10000
      net.ipv4.tcp_tw_reuse = 1
      net.ipv4.tcp_fin_timeout = 30
      net.ipv4.tcp_max_syn_backlog = 10240
      net.ipv4.tcp_max_tw_buckets = 400000
      net.ipv4.tcp_max_orphans = 60000
      net.core.somaxconn = 10000
      [/marathon-lb /marathon-lb/run] Created /marathon-lb/service/lb/run with contents:
      [/marathon-lb /marathon-lb/run] #!/bin/sh
      exec 2>&1
      cd /marathon-lb
      exec /marathon-lb/marathon_lb.py     --syslog-socket /dev/null     --haproxy-config /marathon-lb/haproxy.cfg     --ssl-certs "/etc/ssl/cert.pem"     --command "sv reload /marathon-lb/service/haproxy"     --sse -m http://marathon.mesos:8080 --health-check --haproxy-map --max-reload-retries 10 --reload-interval 10 --group external
      [/marathon-lb/service/haproxy ./run] Reloading haproxy
      [/marathon-lb/service/haproxy ./run] Dropping SYN packets with addFirewallRules
      2017-10-20 22:10:48,048 marathon_lb: setting default value for HAPROXY_BACKEND_HTTP_OPTIONS
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL_ONLY_WITH_PATH
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HEAD
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_BACKEND_HSTS_OPTIONS
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_TCP_BACKEND_ACL_ALLOW_DENY
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL_WITH_AUTH_AND_PATH
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_AUTH_REQUEST_ONLY
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_FRONTEND_HEAD
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_MAP_HTTP_FRONTEND_APPID_ACL
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_BACKEND_SERVER_OPTIONS
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL_ONLY_WITH_PATH_AND_AUTH
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ROUTING_ONLY_WITH_PATH_AND_AUTH
      2017-10-20 22:10:48,049 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_APPID_ACL
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTP_BACKEND_PROXYPASS_GLUE
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_USERLIST_HEAD
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL_WITH_AUTH
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_MAP_HTTP_FRONTEND_ACL_ONLY
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_MAP_HTTPS_FRONTEND_ACL
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_FRONTEND_BACKEND_GLUE
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL_WITH_PATH
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTP_BACKEND_REVPROXY_GLUE
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ROUTING_ONLY_WITH_PATH
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_BACKEND_SERVER_TCP_HEALTHCHECK_OPTIONS
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL_WITH_AUTH_AND_PATH
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_BACKEND_TCP_HEALTHCHECK_OPTIONS
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_BACKEND_STICKY_OPTIONS
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_BACKEND_REDIRECT_HTTP_TO_HTTPS
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_BACKEND_HTTP_HEALTHCHECK_OPTIONS
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL_ONLY_WITH_PATH
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_APPID_HEAD
      2017-10-20 22:10:48,050 marathon_lb: setting default value for HAPROXY_HTTP_BACKEND_ACL_ALLOW_DENY
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_BACKEND_REDIRECT_HTTP_TO_HTTPS_WITH_PATH
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_HEAD
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_AUTH_ACL_ONLY
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_BACKEND_REDIR
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_BACKEND_HEAD
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ROUTING_ONLY_WITH_AUTH
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ROUTING_ONLY
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_BACKEND_NETWORK_ALLOWED_ACL
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ACL_WITH_PATH
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL_ONLY
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_TCP_BACKEND_NETWORK_ALLOWED_ACL
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL_WITH_AUTH
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_BACKEND_SERVER_HTTP_HEALTHCHECK_OPTIONS
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_MAP_HTTP_FRONTEND_ACL
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTPS_FRONTEND_ROUTING_ONLY_WITH_PATH_AND_AUTH
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_HEAD
      2017-10-20 22:10:48,051 marathon_lb: setting default value for HAPROXY_HTTP_FRONTEND_ACL
      2017-10-20 22:10:48,052 marathon_lb: starting event processor thread
      2017-10-20 22:10:48,052 marathon_lb: SSE Active, trying fetch events from http://marathon.mesos:8080/v2/events
      2017-10-20 22:10:48,052 marathon_lb: fetching apps
      2017-10-20 22:10:48,059 marathon_lb: received event of type event_stream_attached
      2017-10-20 22:10:48,062 marathon_lb: GET http://marathon.mesos:8080/v2/apps?embed=apps.tasks
      2017-10-20 22:10:48,064 marathon_lb: got apps ['/marathon-lb', '/marathon-lb-internal', '/portworx']
      2017-10-20 22:10:48,070 marathon_lb: generating config
      2017-10-20 22:10:48,070 marathon_lb: HAProxy dir is /marathon-lb
      2017-10-20 22:10:48,070 marathon_lb: reading running config from /marathon-lb/haproxy.cfg
      2017-10-20 22:10:48,070 marathon_lb: couldn't open config file for reading
      2017-10-20 22:10:48,070 marathon_lb: running config/map is different from generated config - reloading
      2017-10-20 22:10:48,071 marathon_lb: writing temp file /tmp/tmpjpcgeej3 that will replace /marathon-lb/domain2backend.map
      2017-10-20 22:10:48,071 marathon_lb: writing temp file /tmp/tmp5mp5s3qx that will replace /marathon-lb/app2backend.map
      2017-10-20 22:10:48,071 marathon_lb: writing temp file /tmp/tmp4obi7vwr that will replace /marathon-lb/haproxy.cfg
      2017-10-20 22:10:48,071 marathon_lb: checking config with command: ['haproxy', '-f', '/tmp/tmp4obi7vwr', '-c']
      [WARNING] 292/221048 (152) : Can't open server state file '/var/state/haproxy/global': No such file or directory
      Configuration file is valid
      2017-10-20 22:10:48,077 marathon_lb: moving temp file /tmp/tmpjpcgeej3 to /marathon-lb/domain2backend.map
      2017-10-20 22:10:48,077 marathon_lb: moving temp file /tmp/tmp5mp5s3qx to /marathon-lb/app2backend.map
      2017-10-20 22:10:48,077 marathon_lb: moving temp file /tmp/tmp4obi7vwr to /marathon-lb/haproxy.cfg
      2017-10-20 22:10:48,078 marathon_lb: reloading using sv reload /marathon-lb/service/haproxy
      2017-10-20 22:10:48,080 marathon_lb: Unable to get haproxy pids: Command 'pidof haproxy' returned non-zero exit status 1
      ok: run: /marathon-lb/service/haproxy: (pid 33) 1s
      2017-10-20 22:10:48,083 marathon_lb: Unable to get haproxy pids: Command 'pidof haproxy' returned non-zero exit status 1
      2017-10-20 22:10:48,083 marathon_lb: Waiting for new haproxy pid (old pids: [set()], new_pids: [set()])...
      [/marathon-lb/service/haproxy ./run] addFirewallRules done
      [/marathon-lb/service/haproxy ./run] Saving the current HAProxy state
      [/marathon-lb/service/haproxy ./run] Done saving the current HAProxy state
      cat: /tmp/haproxy.pid: No such file or directory
      [/marathon-lb/service/haproxy ./run] LATEST_HAPROXY_PID: []
      [/marathon-lb/service/haproxy ./run] /marathon-lb/haproxy_wrapper.py /usr/local/sbin/haproxy -D -p /tmp/haproxy.pid -f /marathon-lb/haproxy.cfg -sf  200>&-
      2017-10-20 22:10:48,162 haproxy_wrapper: create_haproxy_pipe called
      2017-10-20 22:10:48,162 haproxy_wrapper: create_haproxy_pipe done
      2017-10-20 22:10:48,162 haproxy_wrapper: wait_on_haproxy_pipe called
      [WARNING] 292/221048 (162) : Can't read first line of the server state file '/var/state/haproxy/global'
      [ALERT] 292/221048 (162) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #2 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #2 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #2 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221048 (162) : sendmsg logger #2 failed: No such file or directory (errno=2)
      2017-10-20 22:10:48,168 haproxy_wrapper: close_and_swallow called
      2017-10-20 22:10:48,168 haproxy_wrapper: close_and_swallow successful
      2017-10-20 22:10:48,168 haproxy_wrapper: close_and_swallow called
      2017-10-20 22:10:48,168 haproxy_wrapper: close_and_swallow swallow OSError: [Errno 9] Bad file descriptor
      2017-10-20 22:10:48,168 haproxy_wrapper: wait_on_haproxy_pipe done (False)
      [/marathon-lb/service/haproxy ./run] exit code: 0
      [/marathon-lb/service/haproxy ./run] Removing firewall rules with removeFirewallRules
      2017-10-20 22:10:48,186 marathon_lb: new pids: [{163}]
      2017-10-20 22:10:48,186 marathon_lb: reload finished, took 0.10827231407165527 seconds
      2017-10-20 22:10:48,186 marathon_lb: updating tasks finished, took 0.1342484951019287 seconds
      [/marathon-lb/service/haproxy ./run] removeFirewallRules done
      [/marathon-lb/service/haproxy ./run] Reloading haproxy
      [/marathon-lb/service/haproxy ./run] Reload finished
      [/marathon-lb/service/haproxy ./run] Dropping SYN packets with addFirewallRules
      [/marathon-lb/service/haproxy ./run] addFirewallRules done
      [/marathon-lb/service/haproxy ./run] Saving the current HAProxy state
      [/marathon-lb/service/haproxy ./run] Done saving the current HAProxy state
      [/marathon-lb/service/haproxy ./run] LATEST_HAPROXY_PID: [163]
      [/marathon-lb/service/haproxy ./run] /marathon-lb/haproxy_wrapper.py /usr/local/sbin/haproxy -D -p /tmp/haproxy.pid -f /marathon-lb/haproxy.cfg -sf 163 200>&-
      2017-10-20 22:10:49,902 haproxy_wrapper: create_haproxy_pipe called
      2017-10-20 22:10:49,902 haproxy_wrapper: create_haproxy_pipe done
      2017-10-20 22:10:49,902 haproxy_wrapper: wait_on_haproxy_pipe called
      [ALERT] 292/221049 (493) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #2 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #2 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #2 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #1 failed: No such file or directory (errno=2)
      [ALERT] 292/221049 (493) : sendmsg logger #2 failed: No such file or directory (errno=2)
      2017-10-20 22:10:49,908 haproxy_wrapper: close_and_swallow called
      2017-10-20 22:10:49,908 haproxy_wrapper: close_and_swallow successful
      2017-10-20 22:10:49,908 haproxy_wrapper: close_and_swallow called
      2017-10-20 22:10:49,908 haproxy_wrapper: close_and_swallow swallow OSError: [Errno 9] Bad file descriptor
      2017-10-20 22:10:49,908 haproxy_wrapper: wait_on_haproxy_pipe done (False)
      [/marathon-lb/service/haproxy ./run] exit code: 0
      [/marathon-lb/service/haproxy ./run] Removing firewall rules with removeFirewallRules
      [/marathon-lb/service/haproxy ./run] removeFirewallRules done
      [/marathon-lb/service/haproxy ./run] Reload finished
      2017-10-20 22:10:52,866 marathon_lb: received event of type event_stream_detached
      2017-10-20 22:10:59,939 marathon_lb: received event of type deployment_info
      2017-10-20 22:10:59,939 marathon_lb: received event of type group_change_success
      2017-10-20 22:10:59,945 marathon_lb: received event of type status_update_event
      2017-10-20 22:10:59,945 marathon_lb: received event of type instance_changed_event
      2017-10-20 22:10:59,945 marathon_lb: fetching apps
      2017-10-20 22:10:59,957 marathon_lb: GET http://marathon.mesos:8080/v2/apps?embed=apps.tasks
      2017-10-20 22:10:59,959 marathon_lb: got apps ['/marathon-lb', '/portworx']
      2017-10-20 22:10:59,966 marathon_lb: generating config
      2017-10-20 22:10:59,967 marathon_lb: HAProxy dir is /marathon-lb
      2017-10-20 22:10:59,967 marathon_lb: reading running config from /marathon-lb/haproxy.cfg
      2017-10-20 22:10:59,967 marathon_lb: reading map config from /marathon-lb/domain2backend.map
      2017-10-20 22:10:59,967 marathon_lb: reading map config from /marathon-lb/app2backend.map
      2017-10-20 22:10:59,967 marathon_lb: skipping reload: config/map unchanged
      2017-10-20 22:10:59,968 marathon_lb: updating tasks finished, took 0.022618770599365234 seconds
      2017-10-20 22:11:00,105 marathon_lb: received event of type status_update_event
      2017-10-20 22:11:00,106 marathon_lb: fetching apps
      2017-10-20 22:11:00,112 marathon_lb: received event of type instance_changed_event
      2017-10-20 22:11:00,112 marathon_lb: received event of type remove_health_check_event
      2017-10-20 22:11:00,129 marathon_lb: received event of type deployment_step_success
      2017-10-20 22:11:00,134 marathon_lb: GET http://marathon.mesos:8080/v2/apps?embed=apps.tasks
      2017-10-20 22:11:00,135 marathon_lb: got apps ['/marathon-lb', '/portworx']
      2017-10-20 22:11:00,140 marathon_lb: generating config
      2017-10-20 22:11:00,140 marathon_lb: HAProxy dir is /marathon-lb
      2017-10-20 22:11:00,141 marathon_lb: reading running config from /marathon-lb/haproxy.cfg
      2017-10-20 22:11:00,143 marathon_lb: received event of type deployment_success
      2017-10-20 22:11:00,143 marathon_lb: received event of type app_terminated_event
      2017-10-20 22:11:00,144 marathon_lb: reading map config from /marathon-lb/domain2backend.map
      2017-10-20 22:11:00,144 marathon_lb: reading map config from /marathon-lb/app2backend.map
      2017-10-20 22:11:00,144 marathon_lb: skipping reload: config/map unchanged
      2017-10-20 22:11:00,144 marathon_lb: updating tasks finished, took 0.038243770599365234 seconds
      2017-10-20 22:11:04,559 marathon_lb: received event of type event_stream_detached
      2017-10-20 22:11:51,510 marathon_lb: received event of type failed_health_check_event
      2017-10-20 22:11:56,530 marathon_lb: received event of type failed_health_check_event
      2017-10-20 22:12:01,549 marathon_lb: received event of type failed_health_check_event
      2017-10-20 22:12:01,550 marathon_lb: received event of type unhealthy_instance_kill_event
      2017-10-20 22:12:01,563 marathon_lb: received event of type status_update_event
      2017-10-20 22:12:01,563 marathon_lb: received event of type instance_changed_event
      2017-10-20 22:12:01,564 marathon_lb: fetching apps
      2017-10-20 22:12:01,584 marathon_lb: GET http://marathon.mesos:8080/v2/apps?embed=apps.tasks
      
      

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              prozlach Pawel Rozlach
              Reporter:
              vitosans vitosans
              Team:
              Networking Team
              Watchers:
              Avinash Sridharan (Inactive), Daniil Bastrich, Deepak Goel, Jeffrey Zampieron, jmparra, Judith Malnick (Inactive), Julien Eid (Inactive), Karsten Jeschkies, Pawel Rozlach, vitosans
            • Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: