Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-8084

`POST /v2/apps/:app_id/restart` to non-leader request has no content-body but requires content-type to be set

    Details

    • Story Points:
      2
    • Build artifact:
      Marathon-v1.7.174

      Description

      Overview

      The behavior for posting to the /v2/apps/{app_id}/restart endpoint is different when you post to the leader versus the non-leader.

      Against the leader, the following works perfectly fine:

      $ curl -X POST localhost:8080/v2/apps/sleeper/restart -v
      *   Trying ::1...
      * TCP_NODELAY set
      * Connected to localhost (::1) port 8080 (#0)
      > POST /v2/apps/sleeper/restart HTTP/1.1
      > Host: localhost:8080
      > User-Agent: curl/7.54.0
      > Accept: */*
      >
      < HTTP/1.1 200 OK
      < Date: Wed, 07 Mar 2018 04:30:24 GMT
      < X-Marathon-Leader: http://10.0.75.1:8080
      < Cache-Control: no-cache, no-store, must-revalidate
      < Pragma: no-cache
      < Expires: 0
      < Marathon-Deployment-Id: 0caacee4-33e8-4c06-a9ca-95c43bdc4d5a
      < Content-Type: application/json; qs=2
      < Transfer-Encoding: chunked
      < Server: Jetty(9.3.z-SNAPSHOT)
      <
      * Connection #0 to host localhost left intact
      {"version":"2018-03-07T04:30:24.994Z","deploymentId":"0caacee4-33e8-4c06-a9ca-95c43bdc4d5a"}
      

      however, when you post against a non-leader:

      19:30 $ curl -X POST localhost:8081/v2/apps/sleeper/restart -v
      *   Trying ::1...
      * TCP_NODELAY set
      * Connected to localhost (::1) port 8081 (#0)
      > POST /v2/apps/sleeper/restart HTTP/1.1
      > Host: localhost:8081
      > User-Agent: curl/7.54.0
      > Accept: */*
      >
      < HTTP/1.1 200 OK
      < Date: Fri, 09 Mar 2018 02:30:22 GMT
      < Content-Length: 0
      <
      * Connection #0 to host localhost left intact
      

      What's so terrible here, is that we don't get an error message, and we get a 200 okay response. If you look at the logs, things did not go so well:

      (on non-leader)

      [2018-03-08 19:30:22,493] INFO  Proxying request to POST http://10.0.75.1:8080/v2/apps/sleeper/restart from 10.0.75.1:8081 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder:qtp655013820-58)
      [2018-03-08 19:30:22,495] WARN  /v2/apps/sleeper/restart (org.eclipse.jetty.servlet.ServletHandler:qtp655013820-58)
      java.lang.RuntimeException: while proxying
      	at mesosphere.marathon.api.LeaderProxyFilter.doFilter(LeaderProxyFilter.scala:117)
      	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
      	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
      	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
      	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
      	at org.eclipse.jetty.server.Server.handle(Server.java:517)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Server returned HTTP response code: 415 for URL: http://10.0.75.1:8080/v2/apps/sleeper/restart
      	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
      	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
      	at mesosphere.marathon.api.JavaUrlConnectionRequestForwarder.forward(LeaderProxyFilter.scala:298)
      	at mesosphere.marathon.api.LeaderProxyFilter$$anonfun$doFilter$2.apply(LeaderProxyFilter.scala:113)
      	at mesosphere.marathon.api.LeaderProxyFilter$$anonfun$doFilter$2.apply(LeaderProxyFilter.scala:111)
      	at scala.Option.foreach(Option.scala:257)
      	at mesosphere.marathon.api.LeaderProxyFilter.doFilter(LeaderProxyFilter.scala:111)
      	... 26 common frames omitted
      [2018-03-08 19:30:22,495] WARN  //localhost:8081/v2/apps/sleeper/restart (org.eclipse.jetty.server.HttpChannel:qtp655013820-58)
      java.lang.RuntimeException: while proxying
      	at mesosphere.marathon.api.LeaderProxyFilter.doFilter(LeaderProxyFilter.scala:117)
      	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
      	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
      	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
      	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
      	at org.eclipse.jetty.server.Server.handle(Server.java:517)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Server returned HTTP response code: 415 for URL: http://10.0.75.1:8080/v2/apps/sleeper/restart
      	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
      	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
      	at mesosphere.marathon.api.JavaUrlConnectionRequestForwarder.forward(LeaderProxyFilter.scala:298)
      	at mesosphere.marathon.api.LeaderProxyFilter$$anonfun$doFilter$2.apply(LeaderProxyFilter.scala:113)
      	at mesosphere.marathon.api.LeaderProxyFilter$$anonfun$doFilter$2.apply(LeaderProxyFilter.scala:111)
      	at scala.Option.foreach(Option.scala:257)
      	at mesosphere.marathon.api.LeaderProxyFilter.doFilter(LeaderProxyFilter.scala:111)
      	... 26 common frames omitted
      

      On the leader:

      [2018-03-08 19:30:22,494] WARN  Invalid Request (mesosphere.marathon.api.MarathonExceptionMapper:qtp2013570947-51)
      javax.ws.rs.WebApplicationException: null
      	at com.sun.jersey.server.impl.uri.rules.TerminatingRule.accept(TerminatingRule.java:66)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
      	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
      	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
      	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
      	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
      	at mesosphere.marathon.api.CacheDisablingFilter.doFilter(CacheDisablingFilter.scala:19)
      	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
      	at mesosphere.marathon.api.CORSFilter.doFilter(CORSFilter.scala:41)
      	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
      	at mesosphere.marathon.api.LimitConcurrentRequestsFilter.pass(LimitConcurrentRequestsFilter.scala:31)
      	at mesosphere.marathon.api.LimitConcurrentRequestsFilter$$anonfun$3$$anonfun$apply$2.apply(LimitConcurrentRequestsFilter.scala:16)
      	at mesosphere.marathon.api.LimitConcurrentRequestsFilter$$anonfun$3$$anonfun$apply$2.apply(LimitConcurrentRequestsFilter.scala:16)
      	at mesosphere.marathon.api.LimitConcurrentRequestsFilter.doFilter(LimitConcurrentRequestsFilter.scala:35)
      	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
      	at mesosphere.marathon.api.LeaderProxyFilter.doFilter(LeaderProxyFilter.scala:99)
      	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
      	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
      	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
      	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1158)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1090)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
      	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:240)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109)
      	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:375)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:119)
      	at org.eclipse.jetty.server.Server.handle(Server.java:517)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:242)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:261)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
      	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:75)
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:213)
      	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:147)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
      	at java.lang.Thread.run(Thread.java:748)
      [2018-03-08 19:30:22,495] INFO  10.0.75.1 - - [09/Mar/2018:02:30:22 +0000] "POST //10.0.75.1:8080/v2/apps/sleeper/restart HTTP/1.1" 415 36 "-" "curl/7.54.0"  (mesosphere.chaos.http.ChaosRequestLog:qtp2013570947-51)
      

      If I specify a Content-Type of application/json, then the restart command works, but still, the response is empty.

      $ curl -X POST localhost:8081/v2/apps/sleeper/restart   -H 'Content-Type: application/json' -v
      *   Trying ::1...
      * TCP_NODELAY set
      * Connected to localhost (::1) port 8081 (#0)
      > POST /v2/apps/sleeper/restart HTTP/1.1
      > Host: localhost:8081
      > User-Agent: curl/7.54.0
      > Accept: */*
      > Content-Type: application/json
      >
      < HTTP/1.1 200 OK
      < Date: Fri, 09 Mar 2018 02:36:32 GMT
      < Content-Length: 0
      <
      * Connection #0 to host localhost left intact
      

      Non-leader logs:

      [2018-03-08 19:36:32,642] INFO  Proxying request to POST http://10.0.75.1:8080/v2/apps/sleeper/restart from 10.0.75.1:8081 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder:qtp655013820-58)
      [2018-03-08 19:36:32,676] WARN  //localhost:8081/v2/apps/sleeper/restart (org.eclipse.jetty.server.HttpChannel:qtp655013820-58)
      scala.util.control.BreakControl: null
      

      leader logs:

      [2018-03-08 19:36:32,642] INFO  Proxying request to POST http://10.0.75.1:8080/v2/apps/sleeper/restart from 10.0.75.1:8081 (mesosphere.marathon.api.JavaUrlConnectionRequestForwarder:qtp655013820-58)
      [2018-03-08 19:36:32,676] WARN  //localhost:8081/v2/apps/sleeper/restart (org.eclipse.jetty.server.HttpChannel:qtp655013820-58)
      scala.util.control.BreakControl: null
      
      [2018-03-08 19:39:32,753] INFO  Upgrade root group version:2018-03-09T02:39:32.753Z with force:false (mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-29)
      [2018-03-08 19:39:32,754] INFO  [/sleeper]: restart detected for app (oldVersion FullVersionInfo(2018-03-09T02:36:32.644Z,2018-03-09T02:36:32.644Z,2018-03-09T02:29:40.340Z)) (mesosphere.marathon.upgrade.GroupVersioningUtil$:ForkJoinPool-3-worker-29)
      [2018-03-08 19:39:32,755] INFO  Computed new deployment plan:
      DeploymentPlan id=3943cf75-939e-453a-ad81-23f3740fd022,2018-03-09T02:39:32.753Z
      step 1:
        * Restart(App(/sleeper, cmd="sleep 3600"List())))
       (mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-29)
      [2018-03-08 19:39:32,773] INFO  Deploy plan with force=false:
      DeploymentPlan id=3943cf75-939e-453a-ad81-23f3740fd022,2018-03-09T02:39:32.753Z
      step 1:
        * Restart(App(/sleeper, cmd="sleep 3600"List())))
        (mesosphere.marathon.MarathonSchedulerService:ForkJoinPool-3-worker-29)
      [2018-03-08 19:39:32,774] INFO  Received new deployment plan 3943cf75-939e-453a-ad81-23f3740fd022, no conflicts detected (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-15)
      [2018-03-08 19:39:32,781] INFO  Stored new deployment plan 3943cf75-939e-453a-ad81-23f3740fd022 (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-15)
      [2018-03-08 19:39:32,781] INFO  Launching DeploymentActor for 3943cf75-939e-453a-ad81-23f3740fd022 (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-15)
      [2018-03-08 19:39:32,782] INFO  Removing 3943cf75-939e-453a-ad81-23f3740fd022 from list of running deployments (mesosphere.marathon.core.deployment.impl.DeploymentManagerActor:marathon-akka.actor.default-dispatcher-15)
      [2018-03-08 19:39:32,782] INFO  Deployment 3943cf75-939e-453a-ad81-23f3740fd022:2018-03-09T02:39:32.753Z of / finished (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-11)
      [2018-03-08 19:39:32,798] INFO  Updated groups/apps/pods according to plan 3943cf75-939e-453a-ad81-23f3740fd022 (mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-25)
      [2018-03-08 19:39:32,798] INFO  Deployment 3943cf75-939e-453a-ad81-23f3740fd022:2018-03-09T02:39:32.753Z for / acknowledged. Waiting to get processed (mesosphere.marathon.core.group.impl.GroupManagerImpl:ForkJoinPool-3-worker-29)
      [2018-03-08 19:39:32,799] INFO  10.0.75.1 - - [09/Mar/2018:02:39:32 +0000] "POST //10.0.75.1:8080/v2/apps/sleeper/restart HTTP/1.1" 200 92 "-" "curl/7.54.0" 4 (mesosphere.chaos.http.ChaosRequestLog:qtp2013570947-49)
      [2018-03-08 19:39:32,800] INFO  Completed empty scan in PT0.013S (mesosphere.marathon.storage.repository.GcActor:marathon-akka.actor.default-dispatcher-7)
      

      Acceptance criteria

      Sleeper app definition

      {"id": "/sleeper", "cmd": "sleep 3600", "instances": 0, "cpus": 0.05, "mem": 128}
      

      Given two instances of Marathon (1 leader, one standby proxy) with the sleeper app definition loaded
      When I run curl -X POST {non_leader}/v2/apps/sleeper/restart
      Then I should see a response of 200
      And I should see a deployment plan ID returned
      And the marathon leaders should log that it restarted the app

      Original issue

      When using the v2 api to restart an app, no body is required for the `POST` request, but it seems that if `content-type` is not set, a `415 (Unsupported media type)` error may be returned. This behavior seems inappropriate considering there is actually no content (`Content-Length: 0`). Enabling basic auth seems to prevent the 415. I assume the content-type is being rewritten at some point in those requests.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ivanchernetsky Ivan Chernetsky
                Reporter:
                jfanjoy jfanjoy
                Team:
                Orchestration Team
                Watchers:
                Ivan Chernetsky, jfanjoy, Marco Monaco, Mergebot, Tim Harper
              • Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: