Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-8398

Limit Akka Streams parallelism throughout Marathon codebase

    Details

    • Type: Task
    • Status: Resolved
    • Priority: High
    • Resolution: Done
    • Affects Version/s: Marathon 1.4.12, Marathon 1.5.10, Marathon 1.7, Marathon 1.6.535
    • Fix Version/s: DC/OS 1.12.0
    • Component/s: Unknown
    • Labels:
      None

      Description

      There is a few places where parallelism is still unbounded:

      $ rg Int.MaxValue src/main | grep mapAsync
      src/main/scala/mesosphere/marathon/api/v2/PodsResource.scala:    val future = Source(podSystem.ids()).mapAsync(Int.MaxValue) { id =>
      src/main/scala/mesosphere/marathon/core/group/impl/GroupManagerImpl.scala:    groupRepository.rootVersions().mapAsync(Int.MaxValue) { version =>
      src/main/scala/mesosphere/marathon/core/storage/repository/impl/PersistenceStoreRepository.scala:  override def all(): Source[V, NotUsed] = ids().mapAsync(Int.MaxValue)(get).collect { case Some(x) => x }
      src/main/scala/mesosphere/marathon/core/storage/store/impl/BasePersistenceStore.scala:    Source(list).mapAsync[Option[Serialized]](Int.MaxValue) {
      src/main/scala/mesosphere/marathon/core/storage/store/impl/BasePersistenceStore.scala:    }.mapAsync(Int.MaxValue) { marshaled =>
      

      It should be addressed on master and in all the supported Marathon versions.

      There are two hardcoded variables that are used throughout the Marathon codebase:

      src/main/scala/mesosphere/marathon/storage/migration/Migration.scala:  val maxConcurrency = 8
      src/main/scala/mesosphere/marathon/core/storage/repository/Repository.scala:  val maxConcurrency = 8
      

      They should be replaced with an internal tuning command-line argument with the default value of 8.

        Attachments

          Activity

            People

            • Assignee:
              tharper Tim Harper
              Reporter:
              ivanchernetsky Ivan Chernetsky
              Team:
              Orchestration Team
              Watchers:
              Ivan Chernetsky, Mergebot, Tim Harper
            • Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: