Marathon uses Akka HTTP for health checks. By default, we are using the default connection pool settings, which settings can be seen here:
https://doc.akka.io/docs/akka-http/10.0.10/scala/http/configuration.html (search for host-connection-pool)
This includes keeping TCP connections alive for further HTTP requests. This is an optimization that makes sense when a client is making many requests to the same HTTP service. However, for Marathon, this behavior is very unlikely to be desired as health checks are periodic (most every 15 seconds to 1 minutes), and can lead to strange behavior. We've seen at least one case where a host networking issue occurred and a socket would not close, leading to a pool becoming backed up and unusable.
Ideally, we can use Source.single(request).via(Http().outgoingConnection(...)).runWith(Sink.head) to skip all connection pooling logic, and force a new connection to be created with every request.
Alternatively, the method singleRequest on HttpExt ( Http().singleRequest() ) accepts a connectionPoolSettings parameter, which we should be able to use to effectively use to cause connections to become closed immediately by setting the idleTimeout to 0 seconds.