Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: DC/OS 1.12.0, DC/OS 1.13.0
    • Component/s: metronome
    • Labels:
      None

      Description

      Metronome logs uncovered a race condition during application initialization.

      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: [info] m.m.c.l.i.LeadershipCoordinatorActor - All actors suspended:
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/offerMatcherLaunchTokens#-1298145355]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/offerMatcherManager#565143515]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/expungeOverdueLostTasks#1837852759]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/reconciliationActor#-471519071]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/killOverdueStagedTasks#-1879923897]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/instanceTracker#-7392271]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: * Actor[akka://metronome/user/taskKillServiceActor#663466314]
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: [info] m.m.c.e.CuratorElectionStream$ - Will do leader election through zk://xxxxxxxx:xxxxxxxx@zk-1.zk:2181,zk-2.zk:2181,zk-3.zk:2181,zk-4.zk:2181,zk-5.zk:2181/metronome/leader
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: Oops, cannot start the server.
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: java.lang.IllegalArgumentException: requirement failed: already started: reviveOffersWhenWanted
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at scala.Predef$.require(Predef.scala:277)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at mesosphere.marathon.core.leadership.LeadershipModuleImpl.startWhenLeader(LeadershipModule.scala:43)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at mesosphere.marathon.core.flow.FlowModule.maybeOfferReviver(FlowModule.scala:43)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.scheduler.SchedulerModule.launchQueueModule$lzycompute(SchedulerModule.scala:182)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.scheduler.SchedulerModule.launchQueueModule(SchedulerModule.scala:177)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobsModule.jobRunModule$lzycompute(JobsModule.scala:46)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobsModule.jobRunModule(JobsModule.scala:45)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobsModule.jobSpecModule$lzycompute(JobsModule.scala:57)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobsModule.jobSpecModule(JobsModule.scala:52)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobComponents.apiModule$lzycompute(JobApplicationLoader.scala:62)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobComponents.apiModule(JobApplicationLoader.scala:61)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobComponents.router(JobApplicationLoader.scala:91)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponents.httpRequestHandler(Application.scala:321)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponents.httpRequestHandler$(Application.scala:321)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponentsFromContext.httpRequestHandler$lzycompute(ApplicationLoader.scala:122)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponentsFromContext.httpRequestHandler(ApplicationLoader.scala:122)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponents.application(Application.scala:324)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponents.application$(Application.scala:323)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponentsFromContext.application$lzycompute(ApplicationLoader.scala:122)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.api.BuiltInComponentsFromContext.application(ApplicationLoader.scala:122)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at dcos.metronome.JobApplicationLoader.load(JobApplicationLoader.scala:44)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.core.server.ProdServerStart$.start(ProdServerStart.scala:51)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.core.server.ProdServerStart$.main(ProdServerStart.scala:25)
      Oct 05 21:29:29 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: at play.core.server.ProdServerStart.main(ProdServerStart.scala)
      Oct 05 21:29:30 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 bash[7584]: waiting...
      Oct 05 21:29:35 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 systemd[1]: dcos-metronome.service: main process exited, code=exited, status=255/n/a
      Oct 05 21:29:35 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 systemd[1]: Unit dcos-metronome.service entered failed state.
      Oct 05 21:29:35 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 systemd[1]: dcos-metronome.service failed.
      Oct 05 21:29:50 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 systemd[1]: dcos-metronome.service holdoff time over, scheduling restart.
      Oct 05 21:29:50 dcos-e2e-8958ec50-d542-4e9c-972c-f79c769733df-master-0 systemd[1]: Starting DC/OS Jobs (Metronome): job orchestration...
      

        Attachments

          Activity

            People

            • Assignee:
              tharper Tim Harper
              Reporter:
              alenavarkockova Alena Varkockova
              Team:
              Orchestration Team
              Watchers:
              Alena Varkockova, Matthias Eichstedt, Mergebot, Tim Harper
            • Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: