Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-1703

Multiple instances of a Task with persistent volume do not dispatch to different nodes when unique hostname constraint is set

    Details

      Description

      I'm running Marathon 1.1.1 on Mesos 0.28.1.

      I have a cluster deployed initially with 3 nodes. Originally, i have a marathon app which has set instance to 3 and "constraints" : [["hostname", "UNIQUE"]].
      It works as expected that 3 tasks running on 3 different nodes in the cluster.

      After that, i want to set the marathon app as persistent task such that when my cluster scale to more than 3 nodes (e.g. scale up to 5 nodes), the marathon app can still attach to original running node if the marathon app is killed/restarted. To achieve this this, i update the marathon app json to add persistent volume as follow:
      {
      ...
      "volumes": [
      {
      "containerPath": "data",
      "mode": "RW",
      "persistent":

      { "size": 10 }

      }
      ]
      ...
      }
      The issue is, when i re-deploy the marathon app with persistent volume again, the 3 tasks may dispatch to same node but not spread to 3 nodes. It violates the unique hostname constraints.

      When i look at the marathon log, i find that it tries to reserve and create the volume before launching the task. Below are partial log and from last 3 line, it tries to reserve on node-1b 2 times and node-1a 1 time. It seems that it does not consider the unique hostname constraints yet and hence reserve on node-1b 2 times.

      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,601] INFO Request ReserveAndCreateVolumes for task 'ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3', version '2016-05-06T04:13:14.226Z'. 3 tasksToLaunch, 0 in flight, 0 confirmed.  not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-17)
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,607] INFO Request ReserveAndCreateVolumes for task 'ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3', version '2016-05-06T04:13:14.226Z'. 3 tasksToLaunch, 1 in flight, -1 confirmed. instance count delta 1. not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-7)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,612] INFO Request ReserveAndCreateVolumes for task 'ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3', version '2016-05-06T04:13:14.226Z'. 3 tasksToLaunch, 2 in flight, -2 confirmed. instance count delta 2. not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,614] INFO Finished processing a49a6f78-be84-4473-bc7a-94d0864671bd-O8. Matched 1 ops after 2 passes. cpus(*) 3.6; mem(*) 14094.0; disk(*) 97257.0; ports(*) 31000->32000 left. (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-17)
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,616] INFO Finished processing a49a6f78-be84-4473-bc7a-94d0864671bd-O7. Matched 2 ops after 3 passes. cpus(*) 3.2; mem(*) 13582.0; disk(*) 97247.0; ports(*) 31000->32000 left. (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,621] INFO Processing Reserve(Reserved(task [ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3],AgentInfo(node-01b,Some(a49a6f78-be84-4473-bc7a-94d0864671bd-S1),Buffer(name: "CPUFlags"
      
      ........
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,621] INFO Processing Reserve(Reserved(task [ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3],AgentInfo(node-01a,Some(a49a6f78-be84-4473-bc7a-94d0864671bd-S0),Buffer(name: "CPUFlags"
      
      .....
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,667] INFO Processing Reserve(Reserved(task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3],AgentInfo(node-01b,Some(a49a6f78-be84-4473-bc7a-94d0864671bd-S1),Buffer(name: "CPUFlags"
      

      And later on, the tasks are launched at node-01a two times which is not expected because of unique hostname constraint.

      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.671176   838 master.cpp:3104] Processing ACCEPT call for offers: [ a49a6f78-be84-4473-bc7a-94d0864671bd-O8 ] on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a) for framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.671234   838 master.cpp:2820] Authorizing principal 'marathon' to reserve resources 'cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):10'
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.671315   838 master.cpp:2892] Authorizing principal 'marathon' to create volumes
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.671634   838 master.cpp:3331] Applying RESERVE operation for resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):10 from framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 to slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.671833   838 master.cpp:6560] Sending checkpointed resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):10 to slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.672073   838 master.cpp:3433] Applying CREATE operation for volumes disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da806118-1340-11e6-b6ac-0242d42d37a3:data]:10 from framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 to slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.672217   838 master.cpp:6560] Sending checkpointed resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da806118-1340-11e6-b6ac-0242d42d37a3:data]:10 to slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.672672   838 hierarchical.cpp:653] Updated allocation of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 from cpus(*):4; mem(*):14606; disk(*):97267; ports(*):[31000-32000] to cpus(*):3.6; mem(*):14094; disk(*):97257; ports(*):[31000-32000]; cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):10
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.673059   838 hierarchical.cpp:653] Updated allocation of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 from cpus(*):3.6; mem(*):14094; disk(*):97257; ports(*):[31000-32000]; cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):10 to cpus(*):3.6; mem(*):14094; disk(*):97257; ports(*):[31000-32000]; cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marat
      
      ......
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,671] INFO Task launch for 'task [ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3]' was accepted. 3 tasksToLaunch, 2 in flight, -2 confirmed. instance count delta 3. not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-9)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,685] INFO receiveTaskUpdate: updating status of task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3] (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-13)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,688] INFO Task launch for 'task [ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3]' was accepted. 3 tasksToLaunch, 1 in flight, -1 confirmed. instance count delta 3. not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-6)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,688] INFO Task launch for 'task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3]' was accepted. 3 tasksToLaunch, 0 in flight, 0 confirmed. instance count delta 3. not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-6)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.808794   837 master.cpp:3104] Processing ACCEPT call for offers: [ a49a6f78-be84-4473-bc7a-94d0864671bd-O10 ] on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a) for framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.808843   837 master.cpp:2776] Authorizing framework principal 'marathon' to launch task ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3 as user 'root'
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.809844   837 master.hpp:177] Adding task ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3 with resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da806118-1340-11e6-b6ac-0242d42d37a3:data]:10 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 (node-01a)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.809947   837 master.cpp:3589] Launching task ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3 of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 with resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da806118-1340-11e6-b6ac-0242d42d37a3:data]:10 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0 at slave(1)@10.10.1.64:5051 (node-01a)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,816] INFO receiveTaskUpdate: updating status of task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3] (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-4)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.818338   837 master.cpp:3104] Processing ACCEPT call for offers: [ a49a6f78-be84-4473-bc7a-94d0864671bd-O9 ] on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1 at slave(1)@10.10.2.157:5051 (node-01b) for framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.818383   837 master.cpp:2776] Authorizing framework principal 'marathon' to launch task ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3 as user 'root'
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.818456   837 master.cpp:2776] Authorizing framework principal 'marathon' to launch task ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3 as user 'root'
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.819499   837 master.hpp:177] Adding task ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3 with resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da81246a-1340-11e6-b6ac-0242d42d37a3:data]:10 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1 (node-01b)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.819574   837 master.cpp:3589] Launching task ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3 of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 with resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da81246a-1340-11e6-b6ac-0242d42d37a3:data]:10 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1 at slave(1)@10.10.2.157:5051 (node-01b)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.819982   837 master.hpp:177] Adding task ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3 with resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da7bcd36-1340-11e6-b6ac-0242d42d37a3:data]:10 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1 (node-01b)
      
      May  6 04:13:14 ip-10-10-3-29 mesos-master[707]: I0506 04:13:14.820063   837 master.cpp:3589] Launching task ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3 of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 with resources cpus(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3}):0.4; mem(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3}):512; disk(middlewares, marathon, {marathon_framework_id: a49a6f78-be84-4473-bc7a-94d0864671bd-0001, marathon_task_id: ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3})[ceph-monitor#data#da7bcd36-1340-11e6-b6ac-0242d42d37a3:data]:10 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1 at slave(1)@10.10.2.157:5051 (node-01b)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,818] INFO Task launch for 'task [ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3]' was accepted. 0 tasksToLaunch, 0 in flight, 3 confirmed.  not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-4)
      
      May  6 04:13:14 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:14,818] INFO Task launch for 'task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3]' was accepted. 0 tasksToLaunch, 0 in flight, 3 confirmed.  not backing off (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-4)
      
      May  6 04:13:15 ip-10-10-3-29 mesos-master[707]: I0506 04:13:15.736728   832 master.cpp:5324] Sending 2 offers to framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378
      
      May  6 04:13:15 ip-10-10-3-29 mesos-master[707]: I0506 04:13:15.739001   834 master.cpp:3641] Processing DECLINE call for offers: [ a49a6f78-be84-4473-bc7a-94d0864671bd-O12 ] for framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378
      
      May  6 04:13:15 ip-10-10-3-29 mesos-master[707]: I0506 04:13:15.739146   834 master.cpp:3641] Processing DECLINE call for offers: [ a49a6f78-be84-4473-bc7a-94d0864671bd-O11 ] for framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,545] INFO Received status update for task ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$9250fa1f:Thread-20)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,574] INFO receiveTaskUpdate: updating status of task [ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3] (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-16)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,576] INFO Sending event notification for task [ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3] of app [/ceph-monitor]: TASK_RUNNING (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$e2136a36:marathon-akka.actor.default-dispatcher-6)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,579] INFO POSTing to all endpoints. (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-16)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,580] INFO Sending POST to:http://127.0.0.1:8088/api/v1/marathon_event/ (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-13)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,581] INFO New task task [ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3] changed during app /ceph-monitor scaling, 1 ready 1 healthy need 3 (mesosphere.marathon.upgrade.TaskStartActor:marathon-akka.actor.default-dispatcher-13)
      
      May  6 04:13:16 ip-10-10-3-29 mesos-master[707]: I0506 04:13:16.588955   833 master.cpp:3918] Processing ACKNOWLEDGE call 55189e14-7646-439c-9ef1-2fd8166ccf6d for task ceph-monitor.da806119-1340-11e6-b6ac-0242d42d37a3 of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S0
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,590] INFO Received status update for task ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$9250fa1f:Thread-21)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,592] INFO Received status update for task ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3: TASK_RUNNING () (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$9250fa1f:Thread-22)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,609] INFO Sending event notification for task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3] of app [/ceph-monitor]: TASK_RUNNING (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$e2136a36:marathon-akka.actor.default-dispatcher-12)
      
      May  6 04:13:16 ip-10-10-3-29 mesos-master[707]: I0506 04:13:16.616158   833 master.cpp:3918] Processing ACKNOWLEDGE call 86bc104e-92f6-44db-a317-49909dbff81f for task ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3 of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,613] INFO Sending event notification for task [ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3] of app [/ceph-monitor]: TASK_RUNNING (mesosphere.marathon.core.task.update.impl.steps.PostToEventStreamStepImpl$$EnhancerByGuice$$e2136a36:marathon-akka.actor.default-dispatcher-15)
      
      May  6 04:13:16 ip-10-10-3-29 mesos-master[707]: I0506 04:13:16.618156   833 master.cpp:3918] Processing ACKNOWLEDGE call 4ed579b5-ef54-4348-a9c4-6c9580a1070d for task ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3 of framework a49a6f78-be84-4473-bc7a-94d0864671bd-0001 (marathon) at scheduler-4a428741-d2fe-4bb6-ad21-403d6412e526@10.10.3.188:37378 on slave a49a6f78-be84-4473-bc7a-94d0864671bd-S1
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,630] INFO receiveTaskUpdate: updating status of task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3] (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,630] INFO POSTing to all endpoints. (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,630] INFO New task task [ceph-monitor.da7c1b57-1340-11e6-b6ac-0242d42d37a3] changed during app /ceph-monitor scaling, 2 ready 2 healthy need 3 (mesosphere.marathon.upgrade.TaskStartActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,630] INFO Sending POST to:http://127.0.0.1:8088/api/v1/marathon_event/ (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,630] INFO receiveTaskUpdate: updating status of task [ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3] (mesosphere.marathon.core.launchqueue.impl.AppTaskLauncherActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,631] INFO POSTing to all endpoints. (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,631] INFO New task task [ceph-monitor.da81246b-1340-11e6-b6ac-0242d42d37a3] changed during app /ceph-monitor scaling, 3 ready 3 healthy need 3 (mesosphere.marathon.upgrade.TaskStartActor:marathon-akka.actor.default-dispatcher-17)
      
      May  6 04:13:16 ip-10-10-3-29 marathon[777]: [2016-05-06 04:13:16,631] INFO Successfully started 3 instances of /ceph-monitor (mesosphere.marathon.upgrade.TaskStartActor:marathon-akka.actor.default-dispatcher-17)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                adukhovniy Aleksey Dukhovniy
                Reporter:
                GitHub_IvanTing IvanTing (Inactive)
                Team:
                Orchestration Team
                Watchers:
                Adam Bordelon (Inactive), Asit Trivedi, Cathy Daw, Jason Gilanfarr (Inactive), Ken Sipe, Kevin Lu, Matthias Eichstedt, Srikanth Viswanathan, Tim Harper, Tobias Mueller
              • Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: