Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-3459

Marathon 0.8.1 change leader frequently

    Details

    • Type: Task
    • Status: Resolved
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We upgrade to 0.8.1 few days ago from 0.8.0. We found Marathon frequently change leader after upgrade. Below is logs from Marathon and Zookeeper when change leader.

      Marathon logs:

      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,819] INFO Already running 1 instances of /84bae127-7d46-4829-9518-9620129ce4ac. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,821] INFO Already running 1 instances of /0c8d3b90-ffba-44d3-8c9a-36f2c2e81da0. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,823] INFO Already running 0 instances of /09c0aa5f-ea17-4a28-a493-70ed42ab09e2. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,825] INFO Already running 1 instances of /beb2ef98-05f5-402a-8b01-6eaa50ae8888. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,828] INFO Already running 0 instances of /b465913b-682d-4c53-add0-29120a7bf895. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,830] INFO Already running 1 instances of /2813daaa-4439-4a9c-a3c6-8feac23d218a. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,834] INFO Already running 1 instances of /4c2283ff-979e-4fb6-af92-0c3b00632dbb. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,844] INFO Already running 1 instances of /4f5398fd-2f62-4196-9ab2-d70599b4cf9a. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,851] INFO Already running 1 instances of /1aa1ec62-8217-4068-ae0a-6663d466aeac. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,857] INFO Already running 0 instances of /20050d74-b799-46d3-8a83-d6886314cd7b. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,866] INFO Already running 1 instances of /1f1404aa-967a-4210-96ae-e677e69c829a. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,884] INFO Already running 1 instances of /1481e6b8-19ac-433d-aaf6-2783a8489ab7. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,888] INFO Already running 0 instances of /5271955d-78e4-4d09-99fa-d8698176a5d6. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,895] INFO Already running 0 instances of /92d08645-a9d3-48a7-ab15-ffad1974ecc3. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,901] INFO Already running 0 instances of /f471d891-cbe1-4bd6-a9a7-a270eebe8b7d. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,903] INFO Already running 1 instances of /f7c3d059-c1e3-44a6-b884-326e25d3875c. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:39 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:39,906] INFO Already running 0 instances of /74e3ff71-7d1f-431f-aa82-c9f86aa5efbe. Not scaling. (mesosphere.marathon.SchedulerActions:554)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: Exception in thread "Thread-108148" java.util.concurrent.TimeoutException: Failed to wait for future within timeout
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.__fetch_get_timeout(Native Method)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.access$400(AbstractState.java:34)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:69)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:42)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:22)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:21)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.package$.blocking(package.scala:54)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.lang.Thread.run(Thread.java:745)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: I0611 09:52:40.005492 16162 sched.cpp:1320] Asked to abort the driver
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: I0611 09:52:40.013629 16162 sched.cpp:777] Aborting framework '20150327-042900-1459822508-5050-13001-0000'
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,014] INFO Driver future completed. Executing optional abdication command. (mesosphere.marathon.MarathonSchedulerService:191)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,016] INFO Defeated (Leader Interface) (mesosphere.marathon.MarathonSchedulerService:245)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,016] INFO Will offer leadership after 500 milliseconds backoff (mesosphere.marathon.MarathonSchedulerService:333)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,016] INFO Defeat leadership (mesosphere.marathon.MarathonSchedulerService:284)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [INFO] [06/11/2015 09:52:40.016] [marathon-akka.actor.default-dispatcher-386] [akka://marathon/user/MarathonScheduler/$a] Suspending scheduler actor
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,017] INFO Stopping driver (mesosphere.marathon.MarathonSchedulerService:220)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: I0611 09:52:40.017220 16151 sched.cpp:1286] Asked to stop the driver
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [INFO] [06/11/2015 09:52:40.017] [marathon-akka.actor.default-dispatcher-391] [akka://marathon/user/$b] POSTing to all endpoints.
      un 11 09:52:40 ip-172-31-3-87 marathon[16116]: [INFO] [06/11/2015 09:52:40.017] [marathon-akka.actor.default-dispatcher-389] [akka://marathon/user/MarathonScheduler/$a/UpgradeManager] Removing 1ca26094-74dc-4017-a817-4829ddb5743d from list of running deployments
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: I0611 09:52:40.018920 16157 sched.cpp:752] Stopping framework '20150327-042900-1459822508-5050-13001-0000'
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,320] WARN  (mesosphere.marathon.api.MarathonExceptionMapper:28)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: java.util.concurrent.TimeoutException: Failed to wait for future within timeout
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.__fetch_get_timeout(Native Method)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.access$400(AbstractState.java:34)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:69)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:42)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:22)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:21)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.package$.blocking(package.scala:54)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.lang.Thread.run(Thread.java:745)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,321] INFO 172.31.8.66 -  -  [11/Jun/2015:09:52:38 +0000] "GET /v2/apps/6bb34a8a-602d-4a4f-9ea8-b2f88c9bad78/ HTTP/1.1" 504 54 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-40-generic" (mesosphere.chaos.http.ChaosRequestLog:15)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,535] INFO Using HA and therefore offering leadership (mesosphere.marathon.MarathonSchedulerService:340)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,539] INFO Set group member ID to member_0000019497 (com.twitter.common.zookeeper.Group:426)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,540] INFO Candidate /marathon/leader/member_0000019497 waiting for the next leader election, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:165)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,566] WARN  (mesosphere.marathon.api.MarathonExceptionMapper:28)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: java.util.concurrent.TimeoutException: Failed to wait for future within timeout
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.__fetch_get_timeout(Native Method)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.access$400(AbstractState.java:34)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:69)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:42)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:22)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:21)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.package$.blocking(package.scala:54)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
      un 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: #011at java.lang.Thread.run(Thread.java:745)
      Jun 11 09:52:40 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:40,569] INFO 172.31.8.66 -  -  [11/Jun/2015:09:52:38 +0000] "GET /v2/apps/1419314c-44b8-4e35-95c2-90ceaa062e9f/ HTTP/1.1" 504 54 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-40-generic" (mesosphere.chaos.http.ChaosRequestLog:15)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,287] WARN  (mesosphere.marathon.api.MarathonExceptionMapper:28)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: java.util.concurrent.TimeoutException: Failed to wait for future within timeout
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.__fetch_get_timeout(Native Method)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState.access$400(AbstractState.java:34)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:69)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at org.apache.mesos.state.AbstractState$1.get(AbstractState.java:42)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:22)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1$$anonfun$apply$1.apply(BackToTheFuture.scala:21)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.package$.blocking(package.scala:54)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at mesosphere.util.BackToTheFuture$$anonfun$futureToFutureOption$1.apply(BackToTheFuture.scala:20)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: #011at java.lang.Thread.run(Thread.java:745)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,288] INFO 172.31.11.94 -  -  [11/Jun/2015:09:52:38 +0000] "GET /v2/apps/f1d75759-d38c-412f-b86c-c9e9bf9a1aee/ HTTP/1.1" 504 54 "-" "python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-40-generic" (mesosphere.chaos.http.ChaosRequestLog:15)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,335] INFO Expunging orphaned task with key task:2dc64d8a-2927-4b09-9d6e-6c7a7f8230fc:2dc64d8a-2927-4b09-9d6e-6c7a7f8230fc.c6c1b3f6-101c-11e5-a6be-02416b28d26a (mesosphere.marathon.tasks.TaskTracker:178)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [INFO] [06/11/2015 09:52:41.338] [pool-2-thread-742] [akka://marathon/user/$b] Sending POST to:https://mirana.alauda.club:8443/api/v1/appevents/
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,341] INFO Setting framework ID to 20150327-042900-1459822508-5050-13001-0000 (mesosphere.marathon.MarathonSchedulerService:75)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,348] INFO Expunging orphaned task with key task:36131715-690c-45f4-b1a2-a3c5ad40b76b:36131715-690c-45f4-b1a2-a3c5ad40b76b.c8a87557-101c-11e5-a6be-02416b28d26a (mesosphere.marathon.tasks.TaskTracker:178)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,354] INFO Expunging orphaned task with key task:38b06007-51c7-4230-83a0-887394bd562e:38b06007-51c7-4230-83a0-887394bd562e.47af613b-101d-11e5-a6be-02416b28d26a (mesosphere.marathon.tasks.TaskTracker:178)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,356] ERROR Current member ID member_0000019434 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,357] ERROR Current member ID member_0000019404 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,358] ERROR Current member ID member_0000019386 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,359] INFO Expunging orphaned task with key task:45be490a-8dd2-4d29-8a90-7b3ab75cb8a1:45be490a-8dd2-4d29-8a90-7b3ab75cb8a1.c6c18ce5-101c-11e5-a6be-02416b28d26a (mesosphere.marathon.tasks.TaskTracker:178)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,359] ERROR Current member ID member_0000019374 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,360] ERROR Current member ID member_0000019431 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,361] ERROR Current member ID member_0000019482 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,362] ERROR Current member ID member_0000019485 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      Jun 11 09:52:41 ip-172-31-3-87 marathon[16116]: [2015-06-11 09:52:41,363] ERROR Current member ID member_0000019452 is not a candidate for leader, current voting: [member_0000019496, member_0000019497, member_0000019495] (com.twitter.common.zookeeper.CandidateImpl:144)
      
      

      Zookeeper logs:

      
      

      2015-06-11 09:51:08,340 - INFO [Thread-22:NIOServerCnxn@1001] - Closed socket connection for client /127.0.0.1:36800 (no session established for client)
      2015-06-11 09:51:08,572 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x56508a83 zxid:0xbd0004a726 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:51:08,575 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x56508a84 zxid:0xbd0004a727 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:51:08,607 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x56508a88 zxid:0xbd0004a729 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:51:08,611 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x56508a89 zxid:0xbd0004a72a txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:51:16,289 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@476] - Processed session termination for sessionid: 0x24d8e029dd5009e
      2015-06-11 09:51:16,289 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception
      EndOfStreamException: Unable to read additional data from client sessionid 0x24d8e029dd5009e, likely client has closed socket
      at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
      at java.lang.Thread.run(Thread.java:745)
      2015-06-11 09:51:16,289 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1001] - Closed socket connection for client /172.31.3.87:36278 which had sessionid 0x24d8e029dd5009e
      2015-06-11 09:51:16,290 - ERROR [CommitProcessor:2:NIOServerCnxn@180] - Unexpected Exception:
      java.nio.channels.CancelledKeyException
      at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
      at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
      at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153)
      at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076)
      at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
      at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:641)
      at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74)
      2015-06-11 09:51:27,386 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x5650a409 zxid:0xbd0004a731 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:51:27,388 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x5650a40a zxid:0xbd0004a732 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:51:29,349 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x5650a612 zxid:0xbd0004a736 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:51:29,352 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x5650a613 zxid:0xbd0004a737 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:51:31,253 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x5650a815 zxid:0xbd0004a739 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:51:31,255 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x34d8e02ec7e0058 type:create cxid:0x5650a816 zxid:0xbd0004a73a txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:52:39,985 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.31.11.94:59674
      2015-06-11 09:52:39,985 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] - Connection request from old client /172.31.11.94:59674; will be dropped if server is in r-o mode
      2015-06-11 09:52:39,986 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /172.31.11.94:59674
      2015-06-11 09:52:39,987 - INFO [CommitProcessor:2:ZooKeeperServer@595] - Established session 0x24d8e029dd500a0 with negotiated timeout 10000 for client /172.31.11.94:59674
      2015-06-11 09:52:45,139 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570b8ab2 zxid:0xbd0004a754 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:52:45,141 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570b8ab3 zxid:0xbd0004a755 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:02,199 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba04f zxid:0xbd0004a75c txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:02,202 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba050 zxid:0xbd0004a75d txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:02,224 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba057 zxid:0xbd0004a75f txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:52:39,985 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.31.11.94:59674
      2015-06-11 09:52:39,985 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] - Connection request from old client /172.31.11.94:59674; will be dropped if server is in r-o mode
      2015-06-11 09:52:39,986 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@839] - Client attempting to establish new session at /172.31.11.94:59674
      2015-06-11 09:52:39,987 - INFO [CommitProcessor:2:ZooKeeperServer@595] - Established session 0x24d8e029dd500a0 with negotiated timeout 10000 for client /172.31.11.94:59674
      2015-06-11 09:52:45,139 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570b8ab2 zxid:0xbd0004a754 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:52:45,141 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570b8ab3 zxid:0xbd0004a755 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:02,199 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba04f zxid:0xbd0004a75c txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:02,202 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba050 zxid:0xbd0004a75d txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:02,224 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba057 zxid:0xbd0004a75f txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:02,226 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba058 zxid:0xbd0004a760 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:02,346 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba067 zxid:0xbd0004a762 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:02,348 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba068 zxid:0xbd0004a763 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:03,214 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba266 zxid:0xbd0004a765 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:03,216 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba267 zxid:0xbd0004a766 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:03,241 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba26e zxid:0xbd0004a768 txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:03,243 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba26f zxid:0xbd0004a769 txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:03,271 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba276 zxid:0xbd0004a76b txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:03,279 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba277 zxid:0xbd0004a76c txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:04,234 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba477 zxid:0xbd0004a76e txntype:-1 reqpath:n/a Error Path:/marathon Error:KeeperErrorCode = NodeExists for /marathon
      2015-06-11 09:53:04,236 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14d8e024fff004b type:create cxid:0x570ba478 zxid:0xbd0004a76f txntype:-1 reqpath:n/a Error Path:/marathon/state Error:KeeperErrorCode = NodeExists for /marathon/state
      2015-06-11 09:53:04,260 - INFO [ProcessThread(sid:2 cport:-1)::PrepRequestProces...

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              GitHub_mingqi Mingqi Shao (Inactive)
              Team:
              Orchestration Team
              Watchers:
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: