Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-2563

"MesosAppIntegrationTest.correctly version pods" is flaky

    Details

      Description

      Reported by Jan Schlicht

      On some runs in the CI, this test case fails with the following error message:

      > The code passed to eventually never returned normally. Attempted 5913 times over 1.5001815416666666 minutes. Last failure message: requirement failed: No events matched <event deployment_success to arrive>.

      Stacktrace:

           org.scalatest.exceptions.TestFailedDueToTimeoutException: The code passed to eventually never returned normally. Attempted 5913 times over 1.5001815416666666 minutes. Last failure message: requirement failed: No events matched <event deployment_success to arrive>.
            at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:421)
            at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:439)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.eventually(MesosAppIntegrationTest.scala:16)
            at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:337)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.eventually(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.setup.MarathonTest$class.waitForEventMatching(MarathonTest.scala:461)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.waitForEventMatching(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.setup.MarathonTest$class.waitForEventWith(MarathonTest.scala:442)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.waitForEventWith(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.setup.MarathonTest$class.waitForEvent(MarathonTest.scala:437)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.waitForEvent(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.MesosAppIntegrationTest$$anonfun$9.apply(MesosAppIntegrationTest.scala:236)
            at mesosphere.marathon.integration.MesosAppIntegrationTest$$anonfun$9.apply(MesosAppIntegrationTest.scala:216)
            at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
            at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
            at org.scalatest.Transformer.apply(Transformer.scala:22)
            at org.scalatest.Transformer.apply(Transformer.scala:20)
            at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
            at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
            at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)
            at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
            at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
            at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
            at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
            at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfter$$super$runTest(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:203)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfterEach$$super$runTest(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
            at mesosphere.FunTest.runTest(UnitTest.scala:97)
            at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
            at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
            at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
            at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
            at scala.collection.immutable.List.foreach(List.scala:381)
            at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
            at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
            at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
            at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
            at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
            at org.scalatest.Suite$class.run(Suite.scala:1147)
            at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
            at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
            at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
            at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
            at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfter$$super$run(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:258)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfterAll$$super$run(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
            at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
            at mesosphere.FunTest.run(UnitTest.scala:97)
            at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
            at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
            at sbt.ForkMain$Run$2.call(ForkMain.java:296)
            at sbt.ForkMain$Run$2.call(ForkMain.java:286)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
            Cause: java.lang.IllegalArgumentException: requirement failed: No events matched <event deployment_success to arrive>
            at scala.Predef$.require(Predef.scala:224)
            at mesosphere.marathon.integration.setup.MarathonTest$$anonfun$waitForEventMatching$1.apply(MarathonTest.scala:462)
            at mesosphere.marathon.integration.setup.MarathonTest$$anonfun$waitForEventMatching$1.apply(MarathonTest.scala:461)
            at org.scalatest.concurrent.Eventually$class.makeAValiantAttempt$1(Eventually.scala:395)
            at org.scalatest.concurrent.Eventually$class.tryTryAgain$1(Eventually.scala:409)
            at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:439)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.eventually(MesosAppIntegrationTest.scala:16)
            at org.scalatest.concurrent.Eventually$class.eventually(Eventually.scala:337)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.eventually(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.setup.MarathonTest$class.waitForEventMatching(MarathonTest.scala:461)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.waitForEventMatching(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.setup.MarathonTest$class.waitForEventWith(MarathonTest.scala:442)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.waitForEventWith(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.setup.MarathonTest$class.waitForEvent(MarathonTest.scala:437)
            at mesosphere.marathon.integration.MesosAppIntegrationTest.waitForEvent(MesosAppIntegrationTest.scala:16)
            at mesosphere.marathon.integration.MesosAppIntegrationTest$$anonfun$9.apply(MesosAppIntegrationTest.scala:236)
            at mesosphere.marathon.integration.MesosAppIntegrationTest$$anonfun$9.apply(MesosAppIntegrationTest.scala:216)
            at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
            at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
            at org.scalatest.Transformer.apply(Transformer.scala:22)
            at org.scalatest.Transformer.apply(Transformer.scala:20)
            at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
            at org.scalatest.TestSuite$class.withFixture(TestSuite.scala:196)
            at org.scalatest.FunSuite.withFixture(FunSuite.scala:1560)
            at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:183)
            at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
            at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:196)
            at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
            at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:196)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfter$$super$runTest(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:203)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfterEach$$super$runTest(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:221)
            at mesosphere.FunTest.runTest(UnitTest.scala:97)
            at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
            at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:229)
            at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:396)
            at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:384)
            at scala.collection.immutable.List.foreach(List.scala:381)
            at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
            at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:379)
            at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
            at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:229)
            at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
            at org.scalatest.Suite$class.run(Suite.scala:1147)
            at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
            at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
            at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:233)
            at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
            at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:233)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfter$$super$run(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:258)
            at mesosphere.FunTest.org$scalatest$BeforeAndAfterAll$$super$run(UnitTest.scala:97)
            at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:213)
            at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:210)
            at mesosphere.FunTest.run(UnitTest.scala:97)
            at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
            at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:480)
            at sbt.ForkMain$Run$2.call(ForkMain.java:296)
            at sbt.ForkMain$Run$2.call(ForkMain.java:286)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
      

      The test fails to deploy a pod, the logs this

      Unable to find source-code formatter for language: info. Available languages are: actionscript, html, java, javascript, none, sql, xhtml, xml
       [08:53:23 MesosAppIntegrationTest] Event CallbackEvent(deployment_step_success,Map(plan -> Map(version -> 2016-11-29T08:53:23.460Z, id -> 2ba5ebde-ba6b-45f6-affb-1683e389a176, target -> Map(groups -> List(), version -> 2016-11-29T08:53:23.460Z, id -> /, pods -> List(Map(networks -> List(Map(mode -> host, labels -> Map())), scheduling -> Map(backoff -> Map(backoff -> 1, backoffFactor -> 1.15, maxLaunchDelay -> 3600), upgrade -> Map(minimumHealthCapacity -> 1, maximumOverCapacity -> 1), placement -> Map(constraints -> List(), acceptedResourceRoles -> List())), secrets -> Map(), executorResources -> Map(cpus -> 0.1, mem -> 32, disk -> 10), scaling -> Map(kind -> fixed, instances -> 1, maxInstances -> null), volumes -> List(), version -> 2016-11-29T08:53:23.46Z, id -> /simplepod, labels -> Map(), containers -> List(Map(artifacts -> List(), name -> task1, endpoints -> List(), labels -> Map(), environment -> Map(), resources -> Map(cpus -> 0.1, mem -> 32, disk -> 0, gpus -> 0), exec -> Map(command -> Map(shell -> sleep 1000)), volumeMounts -> List())), environment -> Map())), dependencies -> List(), apps -> List()), original -> Map(groups -> List(), version -> 2016-11-29T08:53:22.200Z, id -> /, pods -> List(), dependencies -> List(), apps -> List()), steps -> List(Map(actions -> List(Map(action -> StartPod, pod -> /simplepod))), Map(actions -> List(Map(action -> ScalePod, pod -> /simplepod))))), currentStep -> Map(actions -> List(Map(action -> ScalePod, pod -> /simplepod))), eventType -> deployment_step_success, timestamp -> 2016-11-29T08:53:23.659Z)) did not match criteria skipping to next event
      INFO [08:53:23 ClientCnxn] Opening socket connection to server 127.0.0.1/127.0.0.1:63024. Will not attempt to authenticate using SASL (unknown error)
      WARN [08:53:23 ClientCnxn] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
      java.net.ConnectException: Connection refused
      	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
      	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:356)
      	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
      

      while waiting for deployment_success. This warning gets repeated over a hundred times. These warning are from Zookeeper and probably halt the deployment. No idea why this is occurring in this particular test case, because the test setup here is similar to some other test cases in MesosAppIntegrationTest.

        Attachments

          Activity

            People

            • Assignee:
              jan Jan Schlicht
              Reporter:
              marco.monaco Marco Monaco
              Team:
              Orchestration Team
              Watchers:
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: