Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-7940

Disable / Skip Connection Pooling for HTTP Health Checks

    Details

      Description

      Background

      Marathon uses Akka HTTP for health checks. By default, we are using the default connection pool settings, which settings can be seen here:

      https://doc.akka.io/docs/akka-http/10.0.10/scala/http/configuration.html (search for host-connection-pool)

      This includes keeping TCP connections alive for further HTTP requests. This is an optimization that makes sense when a client is making many requests to the same HTTP service. However, for Marathon, this behavior is very unlikely to be desired as health checks are periodic (most every 15 seconds to 1 minutes), and can lead to strange behavior. We've seen at least one case where a host networking issue occurred and a socket would not close, leading to a pool becoming backed up and unusable.

      Proposed Solution

      Ideally, we can use Source.single(request).via(Http().outgoingConnection(...)).runWith(Sink.head) to skip all connection pooling logic, and force a new connection to be created with every request.

      Alternatively, the method singleRequest on HttpExt ( Http().singleRequest() ) accepts a connectionPoolSettings parameter, which we should be able to use to effectively use to cause connections to become closed immediately by setting the idleTimeout to 0 seconds.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                nikitamelkozerov Nikita Melkozerov (Inactive)
                Reporter:
                tharper Tim Harper
                Team:
                Orchestration Team
                Watchers:
                Marco Monaco, Marcus Alvarez, Tim Harper
              • Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: