Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-3638

Marathon mount volume for docker using -v/hostpath:/containerpath:RW encounter failure, if the /hostpath is a mounted path.

    Details

    • Type: Task
    • Status: Resolved
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Hi Experts,

      I met a problem about using mounted path on host and mount it to docker container at the same time.

      I launched an app on marathon, but found it's not executed right by mesos.

      The app json is like this:

      {
          "id": "jenkins-docker-template",
          "cpus": 0.2,
          "mem": 256,
          "instances": 1,
          "container": {
              "type": "DOCKER",
              "docker": {
                  "image": "docker-registry1:5000/jenkins",
                  "network": "BRIDGE",
                  "portMappings": [
                      {
                          "containerPort": 8080,
                          "hostPort": 0,
                          "protocol": "tcp"
                      },
                      {
                          "containerPort": 50000,
                          "hostPort": 0,
                          "protocol": "tcp"
                      }
                  ],
                  "privileged": false
              },
              "volumes": [
                      {
                          "containerPath": "/JenkinsBackups",
                          "hostPath": "/JenkinsBackups",
                          "mode": "RW"
                      }
                  ]
          },
      
          "constraints": [
              ["os", "LIKE", "linux"],
              ["docker", "LIKE", "yes"]
          ],
      
          "env": {
              "BACKUP_PATH": "jenkins-template"
          }
      }
      

      In Mesos slave log, it's saying [8] System error: operation not supported"

      I0917 23:02:04.998103  3006 slave.cpp:1144] Got assigned task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b for framework 20150614-225841-303307530-5050-22568-0001
      I0917 23:02:04.999068  3006 slave.cpp:1254] Launching task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b for framework 20150614-225841-303307530-5050-22568-0001
      I0917 23:02:05.025038  3006 slave.cpp:4208] Launching executor jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001 in work directory '/tmp/mesos/slaves/20150812-175543-353639178-5050-5642-S5/frameworks/20150614-225841-303307530-5050-22568-0001/executors/jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b/runs/cb7240f8-ddc4-48d1-a8e7-804616e8567c'
      I0917 23:02:05.026161  3006 slave.cpp:1401] Queuing task 'jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b' for executor jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework '20150614-225841-303307530-5050-22568-0001
      I0917 23:02:05.041221  3008 docker.cpp:626] Starting container 'cb7240f8-ddc4-48d1-a8e7-804616e8567c' for task 'jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b' (and executor 'jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b') of framework '20150614-225841-303307530-5050-22568-0001'
      E0917 23:02:05.897095  3006 slave.cpp:3112] Container 'cb7240f8-ddc4-48d1-a8e7-804616e8567c' for executor 'jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b' of framework '20150614-225841-303307530-5050-22568-0001' failed to start: Failed to 'docker run -d -c 307 -m 301989888 -e MARATHON_APP_VERSION=2015-09-18T03:01:54.605Z -e HOST=mesos-slave2.labs.microstrategy.com -e BACKUP_PATH=jenkins-template -e PORT_50000=31002 -e MESOS_TASK_ID=jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b -e PORT=31001 -e PORTS=31001,31002 -e PORT1=31002 -e PORT_8080=31001 -e MARATHON_APP_ID=/jenkins-template -e PORT0=31001 -e MESOS_SANDBOX=/mnt/mesos/sandbox -v /JenkinsBackups:/JenkinsBackups:rw -v /tmp/mesos/slaves/20150812-175543-353639178-5050-5642-S5/frameworks/20150614-225841-303307530-5050-22568-0001/executors/jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b/runs/cb7240f8-ddc4-48d1-a8e7-804616e8567c:/mnt/mesos/sandbox --net bridge -p 31001:8080/tcp -p 31002:50000/tcp --name mesos-cb7240f8-ddc4-48d1-a8e7-804616e8567c docker-registry1:5000/jenkins': exit status = exited with status 1 stderr = time="2015-09-17T23:02:05-04:00" level=fatal msg="Error response from daemon: Cannot start container 45b78bf612e042d29619f42ff3df38632abf4830d93db6c1e8d222500e65db67: [8] System error: operation not supported" 
      E0917 23:02:05.897999  3011 slave.cpp:3207] Termination of executor 'jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b' of framework '20150614-225841-303307530-5050-22568-0001' failed: Unknown container: cb7240f8-ddc4-48d1-a8e7-804616e8567c
      I0917 23:02:05.905081  3011 slave.cpp:2531] Handling status update TASK_FAILED (UUID: 95fee899-09cf-499a-bac5-9b7745162e9c) for task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001 from @0.0.0.0:0
      W0917 23:02:05.911139  3011 docker.cpp:886] Ignoring updating unknown container: cb7240f8-ddc4-48d1-a8e7-804616e8567c
      I0917 23:02:05.911301  3011 status_update_manager.cpp:317] Received status update TASK_FAILED (UUID: 95fee899-09cf-499a-bac5-9b7745162e9c) for task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001
      I0917 23:02:05.912236  3011 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_FAILED (UUID: 95fee899-09cf-499a-bac5-9b7745162e9c) for task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001
      E0917 23:02:05.906175  3009 slave.cpp:3461] Failed to unmonitor container for executor jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001: Not monitored
      I0917 23:02:05.914896  3011 slave.cpp:2776] Forwarding the update TASK_FAILED (UUID: 95fee899-09cf-499a-bac5-9b7745162e9c) for task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001 to master@10.27.20.18:5050
      I0917 23:02:05.924022  3011 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 95fee899-09cf-499a-bac5-9b7745162e9c) for task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001
      I0917 23:02:05.924109  3011 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_FAILED (UUID: 95fee899-09cf-499a-bac5-9b7745162e9c) for task jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b of framework 20150614-225841-303307530-5050-22568-0001
      I0917 23:02:05.926256  3011 slave.cpp:3332] Cleaning up executor 'jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b' of framework 20150614-225841-303307530-5050-22568-0001
      

      I pick the docker command in it, like this, run it in CLI, still met the same error.

      docker run -d -c 307 -m 301989888 -e MARATHON_APP_VERSION=2015-09-18T03:01:54.605Z -e HOST=mesos-slave2.labs.microstrategy.com -e BACKUP_PATH=jenkins-template -e PORT_50000=31002 -e MESOS_TASK_ID=jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b -e PORT=31001 -e PORTS=31001,31002 -e PORT1=31002 -e PORT_8080=31001 -e MARATHON_APP_ID=/jenkins-template -e PORT0=31001 -e MESOS_SANDBOX=/mnt/mesos/sandbox -v /JenkinsBackups:/JenkinsBackups:rw -v /tmp/mesos/slaves/20150812-175543-353639178-5050-5642-S5/frameworks/20150614-225841-303307530-5050-22568-0001/executors/jenkins-template.a4306ec0-5db1-11e5-bb92-00505601432b/runs/cb7240f8-ddc4-48d1-a8e7-804616e8567c:/mnt/mesos/sandbox --net bridge -p 31001:8080/tcp -p 31002:50000/tcp --name mesos-cb7240f8-ddc4-48d1-a8e7-804616e85ccc docker-registry1:5000/jenkins
      

      Please notice there is one param: -v /JenkinsBackups:/JenkinsBackups:rw

      After I change it to -v /JenkinsBackups:/JenkinsBackups without the ":rw", it works fine, the container is started.

      I try to removing this "mode:RW" line from marathon json app, it looks like there are some validation process inside of marathon, so I got validation error and cannot deploy the app.

      {
          "id": "jenkins-docker-template",
          "cpus": 0.2,
          "mem": 256,
          "instances": 1,
          "container": {
              "type": "DOCKER",
              "docker": {
                  "image": "docker-registry1:5000/jenkins",
                  "network": "BRIDGE",
                  "portMappings": [
                      {
                          "containerPort": 8080,
                          "hostPort": 0,
                          "protocol": "tcp"
                      },
                      {
                          "containerPort": 50000,
                          "hostPort": 0,
                          "protocol": "tcp"
                      }
                  ],
                  "privileged": false
              },
              "volumes": [
                      {
                          "containerPath": "/JenkinsBackups",
                          "hostPath": "/JenkinsBackups"
                      }
                  ]
          },
      
          "constraints": [
              ["os", "LIKE", "linux"],
              ["docker", "LIKE", "yes"]
          ],
      
          "env": {
              "BACKUP_PATH": "jenkins-template"
          }
      }
      

      The error message is like this:

      {
        "message": "Invalid JSON: JsResultException(errors:List((/container/volumes(0)/mode,List(ValidationError( is not a valid mode,WrappedArray())))))"
      }
      

      Is there any way I can remove the ":rw" in marathon app json configuration?

      I have to use this way, because that's how we backup all the jenkins server. the /JenkinsBackups folder is mounted on every mesos slave, the source is from some large disk and our mesos slave machine don't have such large disk.

      Thank you!
      Ryanivanka

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              GitHub_ryanivanka ryanivanka (Inactive)
              Team:
              Orchestration Team
              Watchers:
            • Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: