Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-7972

Upgrading 1.4.5 to 1.5.4 the migration fails.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Medium
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Persistence
    • Labels:

      Description

      Upgrading 1.4.5-1.0.654.ubuntu1404 to 1.5.4 the migration fails.

       

      [2017-12-18 00:12:39,046] ERROR Fatal error while starting leadership of Some(MarathonSchedulerService [RUNNING]). Exiting now (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
      mesosphere.marathon.StoreCommandFailedException: Migration is already in progress; /migration-in-progress node already exists
      at mesosphere.marathon.core.storage.store.impl.zk.ZkPersistenceStore$stateMachine$macro$714$1.apply(ZkPersistenceStore.scala:364)
      at mesosphere.marathon.core.storage.store.impl.zk.ZkPersistenceStore$stateMachine$macro$714$1.apply(ZkPersistenceStore.scala:359)
      at scala.concurrent.impl.CallbackRunnable.run_aroundBody0(Promise.scala:36)
      at scala.concurrent.impl.CallbackRunnable$AjcClosure1.run(Promise.scala:1)
      at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
      at kamon.scala.instrumentation.FutureInstrumentation$$anonfun$aroundExecution$1.apply(FutureInstrumentation.scala:45)
      at kamon.trace.Tracer$.withContext(TracerModule.scala:58)
      at kamon.scala.instrumentation.FutureInstrumentation.aroundExecution(FutureInstrumentation.scala:44)
      at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:35)
      at mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2$$anonfun$run$1.apply$mcV$sp(ExecutionContexts.scala:20)
      at mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2$$anonfun$run$1.apply(ExecutionContexts.scala:20)
      at mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2$$anonfun$run$1.apply(ExecutionContexts.scala:20)
      at mesosphere.marathon.core.async.package$.propagateContext(package.scala:15)
      at mesosphere.marathon.core.async.ContextPropagatingExecutionContext$$anon$1$$anon$2.run(ExecutionContexts.scala:20)
      at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
      Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
      at mesosphere.marathon.core.storage.store.impl.zk.ZkFuture.processResult(ZkFuture.scala:30)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:749)
      at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:522)
      at org.apache.curator.framework.imps.CreateBuilderImpl.sendBackgroundResponse(CreateBuilderImpl.java:580)
      at org.apache.curator.framework.imps.CreateBuilderImpl.access$900(CreateBuilderImpl.java:44)
      at org.apache.curator.framework.imps.CreateBuilderImpl$6.processResult(CreateBuilderImpl.java:534)
      at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:613)
      at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
      [2017-12-18 00:12:39,046] INFO Stopping the election service (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
      [2017-12-18 00:12:39,052] INFO backgroundOperationsLoop exiting (org.apache.curator.framework.imps.CuratorFrameworkImpl:Curator-Framework-0)
      [2017-12-18 00:12:39,068] INFO Session: 0x15c6eaee54702a7 closed (org.apache.zookeeper.ZooKeeper:pool-3-thread-1)
      [2017-12-18 00:12:39,068] INFO EventThread shut down for session: 0x15c6eaee54702a7 (org.apache.zookeeper.ClientCnxn:pool-3-thread-1-EventThread)
      [2017-12-18 00:12:39,070] INFO Stopping MarathonSchedulerService [RUNNING]'s leadership (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
      [2017-12-18 00:12:39,070] INFO Lost leadership (mesosphere.marathon.MarathonSchedulerService:pool-3-thread-1)
      [2017-12-18 00:12:39,073] INFO Stopped MarathonSchedulerService [RUNNING]'s leadership (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
      [2017-12-18 00:12:39,073] INFO Terminating due to leadership abdication or failure (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-3-thread-1)
      [2017-12-18 00:12:39,073] INFO Now standing by. Closing existing handles and rejecting new.
      

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ivanchernetsky Ivan Chernetsky
                Reporter:
                justinholmes Justin Holmes
                Team:
                Orchestration Team
                Watchers:
                Artem Harutyunyan, Ivan Chernetsky, Justin Holmes, pgillet
              • Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: