Details

    • Type: Bug
    • Status: Accepted
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: metronome
    • Labels:

      Description

      There is a timeout error 500 that occurs when requesting job list resulting in an HTTP 500

       

      How to duplicate:

      1. Create a simple job (that requires more CPU that is possible on a node)
        1. {"id":"large","labels":{},"run":{"cpus":10,"mem":128,"disk":0,"cmd":"sleep inf","env":{},"placement":{"constraints":[]},"artifacts":[],"maxLaunchDelay":3600,"volumes":[],"restart":{"policy":"NEVER"},"secrets":{}}}
      2. Trigger that job 1200 times
        1. for run in {1..1000}
          do
          curl -X POST -k -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/service/metronome/v1/jobs/large/runs
          done
      3. bounce metronome
        1. sudo systemctl restart dcos-metronome
      1. hit job list (like UI) 
        1. curl -k -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/service/metronome/v1/jobs?embed=activeRuns&embed=schedules&embed=historySummary

       

      10 secs later... 

      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]: [error] d.m.a.ErrorHandler - Error serving /v1/jobs
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka://application/user/JobRunService#1204363722]] after [10000 ms]. Sender[null] sent message of type "dcos.metronome.jobrun.impl.JobRunServiceActor$ListRuns".
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
      
      Aug 21 19:13:29 ip-10-0-6-206.us-west-2.compute.internal metronome[10870]:         at java.lang.Thread.run(Thread.java:748)

       

      Eventually the stdout will show the result of the query.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ken Ken Sipe
                Team:
                Orchestration Team
                Watchers:
                Brahim Atchi, Dustin Nemes, Ken Sipe, Matthias Eichstedt, Nikita Melkozerov (Inactive), Sivaram Kannan, Vishnu Mohan (Inactive)
              • Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: