Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-1710

Persistent Docker volumes are not actually persistent

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Medium
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Mesos Integration
    • Labels:

      Description

      Launched postgres container from example in documentation. It starts and claims to have a persistent volume attached:

      <img width="1433" alt="screen shot 2016-05-26 at 16 22 07" src="https://cloud.githubusercontent.com/assets/3029571/15579801/1ae9d06c-235e-11e6-8958-4e9536366dd5.png">

      ip-10-0-70-103 volumes # docker ps
      CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS                     NAMES
      3a35d1a989ad        postgres:latest                          "/docker-entrypoint.s"   11 minutes ago      Up 11 minutes       0.0.0.0:31561->5432/tcp   mesos-fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0.fd156da2-bf90-40dc-b32e-968bbfd6da4c
      49cffa0a9952        docker.intenthq.com/mesos-slave:latest   "mesos-slave"            2 weeks ago         Up 2 weeks                                    mesos-slave
      

      Upon inspection it doesn't look like anything persistent gets mounted into /pgdata volume as specified in containerPath of the app:

      ip-10-0-70-103 volumes # docker inspect 3a35d1a989ad
      [
      {
          "Id": "3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076",
          "Created": "2016-05-26T15:12:01.349808731Z",
          "Path": "/docker-entrypoint.sh",
          "Args": [
              "postgres"
          ],
          "State": {
              "Status": "running",
              "Running": true,
              "Paused": false,
              "Restarting": false,
              "OOMKilled": false,
              "Dead": false,
              "Pid": 9888,
              "ExitCode": 0,
              "Error": "",
              "StartedAt": "2016-05-26T15:12:01.618798614Z",
              "FinishedAt": "0001-01-01T00:00:00Z"
          },
          "Image": "bb897848921416c050d1e4b9be040c76fc3869611638791698775e05caa5046a",
          "ResolvConfPath": "/var/lib/docker/containers/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/resolv.conf",
          "HostnamePath": "/var/lib/docker/containers/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/hostname",
          "HostsPath": "/var/lib/docker/containers/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/hosts",
          "LogPath": "/var/lib/docker/containers/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076-json.log",
          "Name": "/mesos-fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0.fd156da2-bf90-40dc-b32e-968bbfd6da4c",
          "RestartCount": 0,
          "Driver": "overlay",
          "ExecDriver": "native-0.2",
          "MountLabel": "system_u:object_r:svirt_lxc_file_t:s0:c576,c580",
          "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c576,c580",
          "AppArmorProfile": "",
          "ExecIDs": null,
          "HostConfig": {
              "Binds": [
                  "/var/lib/mesos/slave/slaves/fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0/frameworks/1a6352a6-d690-41a2-967e-07342bba56d2-0000/executors/postgres.e21f6bf2-234e-11e6-9931-a2bd870b1e9c/runs/fd156da2-bf90-40dc-b32e-968bbfd6da4c:/mnt/mesos/sandbox"
              ],
              "ContainerIDFile": "",
              "LxcConf": [],
              "Memory": 536870912,
              "MemoryReservation": 0,
              "MemorySwap": 1073741824,
              "KernelMemory": 0,
              "CpuShares": 1024,
              "CpuPeriod": 0,
              "CpusetCpus": "",
              "CpusetMems": "",
              "CpuQuota": 0,
              "BlkioWeight": 0,
              "OomKillDisable": false,
              "MemorySwappiness": -1,
              "Privileged": false,
              "PortBindings": {
                  "5432/tcp": [
                      {
                          "HostIp": "",
                          "HostPort": "31561"
                      }
                  ]
              },
              "Links": null,
              "PublishAllPorts": false,
              "Dns": [],
              "DnsOptions": [],
              "DnsSearch": [],
              "ExtraHosts": null,
              "VolumesFrom": null,
              "Devices": [],
              "NetworkMode": "bridge",
              "IpcMode": "",
              "PidMode": "",
              "UTSMode": "",
              "CapAdd": null,
              "CapDrop": null,
              "GroupAdd": null,
              "RestartPolicy": {
                  "Name": "no",
                  "MaximumRetryCount": 0
              },
              "SecurityOpt": null,
              "ReadonlyRootfs": false,
              "Ulimits": null,
              "LogConfig": {
                  "Type": "json-file",
                  "Config": {}
              },
              "CgroupParent": "",
              "ConsoleSize": [
                  0,
                  0
              ],
              "VolumeDriver": ""
          },
          "GraphDriver": {
              "Name": "overlay",
              "Data": {
                  "LowerDir": "/var/lib/docker/overlay/bb897848921416c050d1e4b9be040c76fc3869611638791698775e05caa5046a/root",
                  "MergedDir": "/var/lib/docker/overlay/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/merged",
                  "UpperDir": "/var/lib/docker/overlay/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/upper",
                  "WorkDir": "/var/lib/docker/overlay/3a35d1a989ad340b2e573fb2538efc1a5a320d2f2122187a3529664c2cf7d076/work"
              }
          },
          "Mounts": [
              {
                  "Source": "/var/lib/mesos/slave/slaves/fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0/frameworks/1a6352a6-d690-41a2-967e-07342bba56d2-0000/executors/postgres.e21f6bf2-234e-11e6-9931-a2bd870b1e9c/runs/fd156da2-bf90-40dc-b32e-968bbfd6da4c",
                  "Destination": "/mnt/mesos/sandbox",
                  "Mode": "",
                  "RW": true
              },
              {
                  "Name": "3397f2de382920654da0ae92c06ad46633eeb415788c831bde2765bfbe91255b",
                  "Source": "/var/lib/docker/volumes/3397f2de382920654da0ae92c06ad46633eeb415788c831bde2765bfbe91255b/_data",
                  "Destination": "/var/lib/postgresql/data",
                  "Driver": "local",
                  "Mode": "",
                  "RW": true
              }
          ],
          "Config": {
              "Hostname": "3a35d1a989ad",
              "Domainname": "",
              "User": "",
              "AttachStdin": false,
              "AttachStdout": true,
              "AttachStderr": true,
              "ExposedPorts": {
                  "5432/tcp": {}
              },
              "Tty": false,
              "OpenStdin": false,
              "StdinOnce": false,
              "Env": [
                  "MARATHON_APP_VERSION=2016-05-26T15:11:50.854Z",
                  "HOST=ip-10-0-70-103.eu-west-1.compute.internal",
                  "MARATHON_APP_RESOURCE_CPUS=1.0",
                  "PORT_5432=31561",
                  "POSTGRES_PASSWORD=password",
                  "MARATHON_APP_DOCKER_IMAGE=postgres:latest",
                  "MESOS_TASK_ID=postgres.e21f6bf2-234e-11e6-9931-a2bd870b1e9c",
                  "PORT=31561",
                  "MARATHON_APP_RESOURCE_MEM=512.0",
                  "PORTS=31561",
                  "PGDATA=pgdata",
                  "MARATHON_APP_RESOURCE_DISK=0.0",
                  "MARATHON_APP_LABELS=",
                  "MARATHON_APP_ID=/postgres",
                  "PORT0=31561",
                  "MESOS_SANDBOX=/mnt/mesos/sandbox",
                  "MESOS_CONTAINER_NAME=mesos-fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0.fd156da2-bf90-40dc-b32e-968bbfd6da4c",
                  "PATH=/usr/lib/postgresql/9.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                  "GOSU_VERSION=1.7",
                  "LANG=en_US.utf8",
                  "PG_MAJOR=9.5",
                  "PG_VERSION=9.5.3-1.pgdg80+1"
              ],
              "Cmd": [
                  "postgres"
              ],
              "Image": "postgres:latest",
              "Volumes": {
                  "/var/lib/postgresql/data": {}
              },
              "WorkingDir": "",
              "Entrypoint": [
                  "/docker-entrypoint.sh"
              ],
              "OnBuild": null,
              "Labels": {},
              "StopSignal": "SIGTERM"
          },
          "NetworkSettings": {
              "Bridge": "",
              "SandboxID": "7bd1bda768cbe66ab10b0e61480fe82d9c03f0e4b65db711ad27e5b4acae4ec7",
              "HairpinMode": false,
              "LinkLocalIPv6Address": "",
              "LinkLocalIPv6PrefixLen": 0,
              "Ports": {
                  "5432/tcp": [
                      {
                          "HostIp": "0.0.0.0",
                          "HostPort": "31561"
                      }
                  ]
              },
              "SandboxKey": "/var/run/docker/netns/7bd1bda768cb",
              "SecondaryIPAddresses": null,
              "SecondaryIPv6Addresses": null,
              "EndpointID": "a59277c050fb2eeacb0f9a0ad0d10b2f1e02770317f176c3a36f216125b13272",
              "Gateway": "172.17.0.1",
              "GlobalIPv6Address": "",
              "GlobalIPv6PrefixLen": 0,
              "IPAddress": "172.17.0.2",
              "IPPrefixLen": 16,
              "IPv6Gateway": "",
              "MacAddress": "02:42:ac:11:00:02",
              "Networks": {
                  "bridge": {
                      "EndpointID": "a59277c050fb2eeacb0f9a0ad0d10b2f1e02770317f176c3a36f216125b13272",
                      "Gateway": "172.17.0.1",
                      "IPAddress": "172.17.0.2",
                      "IPPrefixLen": 16,
                      "IPv6Gateway": "",
                      "GlobalIPv6Address": "",
                      "GlobalIPv6PrefixLen": 0,
                      "MacAddress": "02:42:ac:11:00:02"
                  }
              }
          }
      }
      ]
      

      So that means that any data that is created in pgdata will disappear after container is re-launched:

      Creating a file:

      ip-10-0-70-103 volumes # docker exec -it 3a35d1a989ad touch /pgdata/me
      ip-10-0-70-103 volumes # docker exec -it 3a35d1a989ad ls /pgdata/
      base    me   pg_commit_ts  pg_hba.conf    pg_logical    pg_notify    pg_serial     pg_stat      pg_subtrans  pg_twophase  pg_xlog           postgresql.conf  postmaster.pid
      global  pg_clog  pg_dynshmem   pg_ident.conf  pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION   postgresql.auto.conf  postmaster.opts
      

      Suspending the app:
      <img width="1424" alt="screen shot 2016-05-26 at 16 30 13" src="https://cloud.githubusercontent.com/assets/3029571/15580116/58091c90-235f-11e6-8504-269254a111d7.png">

      And after scaling back to 1 instance, the volume gets re-attached, but the it's a new volume (the file that was created vanished):

      ip-10-0-70-103 volumes # docker ps
      CONTAINER ID        IMAGE                                    COMMAND                  CREATED              STATUS              PORTS                     NAMES
      a1187eec97d4        postgres:latest                          "/docker-entrypoint.s"   About a minute ago   Up About a minute   0.0.0.0:31561->5432/tcp   mesos-fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0.d168d347-61ca-4973-b341-0fedcef3909c
      49cffa0a9952        docker.intenthq.com/mesos-slave:latest   "mesos-slave"            2 weeks ago          Up 2 weeks                                    mesos-slave
      ip-10-0-70-103 volumes # docker exec -it a1187eec97d4 ls /pgdata
      base    pg_clog       pg_dynshmem  pg_ident.conf  pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION  postgresql.auto.conf  postmaster.opts
      global  pg_commit_ts  pg_hba.conf  pg_logical     pg_notify pg_serial    pg_stat       pg_subtrans  pg_twophase  pg_xlog     postgresql.conf       postmaster.pid
      

      Moreover, the file exists only in overlayFS layer.

      Here's a snippet from stdout of the app:

      --container="mesos-fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0.d168d347-61ca-4973-b341-0fedcef3909c" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slave/slaves/fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0/frameworks/1a6352a6-d690-41a2-967e-07342bba56d2-0000/executors/postgres.e21f6bf2-234e-11e6-9931-a2bd870b1e9c/runs/d168d347-61ca-4973-b341-0fedcef3909c" --stop_timeout="0ns"
      --container="mesos-fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0.d168d347-61ca-4973-b341-0fedcef3909c" --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" --logbufsecs="0" --logging_level="INFO" --mapped_directory="/mnt/mesos/sandbox" --quiet="false" --sandbox_directory="/var/lib/mesos/slave/slaves/fde0506a-d6e1-4cd4-98a0-834de3fbc694-S0/frameworks/1a6352a6-d690-41a2-967e-07342bba56d2-0000/executors/postgres.e21f6bf2-234e-11e6-9931-a2bd870b1e9c/runs/d168d347-61ca-4973-b341-0fedcef3909c" --stop_timeout="0ns"
      Registered docker executor on ip-10-0-70-103.eu-west-1.compute.internal
      Starting task postgres.e21f6bf2-234e-11e6-9931-a2bd870b1e9c
      The files belonging to this database system will be owned by user "postgres".
      This user must also own the server process.
      
      ...skipped...
      
      PostgreSQL init process complete; ready for start up.
      

      Am I missing anything in configuration?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              GitHub_liquid-sky Evgeny Lazarev (Inactive)
              Team:
              Orchestration Team
              Watchers:
              drcrallen, Jason Gilanfarr (Inactive)
            • Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: