Uploaded image for project: 'DC/OS'
  1. DC/OS
  2. DCOS_OSS-1230

Need a Proper Reported Failure for bad Protocol

    Details

    • Type: Task
    • Status: Open
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: networking
    • Labels:
      None

      Description

      When creating a `container/bridge` network... the underlying implementation is iptables (for our env).  which limits the protocol options.   Marathon does NOT want to limit the protocols because 1) we shouldn't bubble up the underlying impl and 2) customers may have their own preferred protocol (or use `iptables-extensions`).   In the case that a protocol (like 'http') is used, the launch attempts fail, but the failure can not be determined by Marathon.

      We are looking for a way for users to get failure info / notification without digging through logs.

      {
      "id": "/pod-container-bridge",
      "containers": [
      {
      "name": "simple-docker",
      "resources": {
      "cpus": 1,
      "mem": 128,
      "disk": 0,
      "gpus": 0
      },
      "image": {
      "kind": "DOCKER",
      "id": "nginx"
      },
      "endpoints": [
      {
      "name": "web",
      "containerPort": 80,
      "hostPort" : 0,
      "protocol": [
      "http"
      ]
      }
      ]
      }
      ],
      "networks": [ { "mode": "container/bridge"} ]
      }

      Here is a pod spec to use.

       

      current generated marathon logs

      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: [2017-06-09 18:28:46,875] INFO  Acknowledge status update for task pod-container-bridge.instance-7954e8d4-4d41-11e7-81a2-0e765f9edf17.simple-docker: TASK_FAILED (Failed to launch container: Collect failed: The CNI plugin '/opt/mesosphere/active/mesos/libexec/mesos/mesos-cni-port-mapper' failed to attach container 3660f0e4-32db-4a86-94b7-9ed895c6327b to CNI network 'mesos-bridge': stdout='{"cniVersion":"0.3.0","code":103,"msg":"Failed to add DNAT rule with tag: Resource temporarily unavailable"}
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: ', stderr='Delegate CNI plugin '/opt/mesosphere/active/cni/bridge' executed successfully for ADD command: {"dns":{},"ip4":{"gateway":"172.31.254.1","ip":"172.31.254.2\/24","routes":[{"dst":"0.0.0.0\/0"},{"dst":"0.0.0.0\/0","gw":"172.31.254.1"}]}}
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + iptables -w -t nat --list UCR-DEFAULT-BRIDGE
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: iptables: No chain/target/match by that name.
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + '[' 1 -ne 0 ']'
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + iptables -w -t nat -N UCR-DEFAULT-BRIDGE
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + iptables -w -t nat -A PREROUTING -m addrtype --dst-type LOCAL -j UCR-DEFAULT-BRIDGE
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + iptables -w -t nat -A OUTPUT '!' -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j UCR-DEFAULT-BRIDGE
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + iptables -w -t nat -C UCR-DEFAULT-BRIDGE '!' -i ucr-br0 -p http -m http --dport 4204 -j DNAT --to-destination 172.31.254.2:80 -m comment --comment 'container_id: 3660f0e4-32db-4a86-94b7-9ed895c6327b'
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: iptables v1.4.21: unknown protocol "http" specified
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: Try `iptables -h' or 'iptables --help' for more information.
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: + iptables -t nat -A UCR-DEFAULT-BRIDGE '!' -i ucr-br0 -p http -m http --dport 4204 -j DNAT --to-destination 172.31.254.2:80 -m comment --comment 'container_id: 3660f0e4-32db-4a86-94b7-9ed895c6327b'
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: iptables v1.4.21: unknown protocol "http" specified
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: Try `iptables -h' or 'iptables --help' for more information.
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: ') (mesosphere.marathon.core.task.update.impl.TaskStatusUpdateProcessorImpl:ForkJoinPool-3-worker-7)
      Jun 09 18:28:46 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: [2017-06-09 18:28:46,875] INFO  add 1 instances to 0 instances to launch (mesosphere.marathon.core.launchqueue.impl.TaskLauncherActor:marathon-akka.actor.default-dispatcher-7)
      Jun 09 18:28:48 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: [2017-06-09 18:28:48,034] INFO  activating matcher ActorOfferMatcher(Actor[akka://marathon/user/launchQueue/1/0-pod-container-bridge#2131453442]). (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-5)
      Jun 09 18:28:48 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: [2017-06-09 18:28:48,034] INFO  Received offers WANTED notification (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-5)
      Jun 09 18:28:48 ip-10-0-4-214.us-west-2.compute.internal marathon[3264]: [2017-06-09 18:28:48,034] INFO  => Schedule next revive at 2017-06-09T18:28:51.184Z in 3150 milliseconds, adhering to --min_revive_offers_interval 5000 (ms) (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-5)

       

       

        Attachments

          Activity

            People

            • Assignee:
              dgoel Deepak Goel
              Reporter:
              ken Ken Sipe
              Team:
              Networking Team
              Watchers:
              Avinash Sridharan (Inactive), James DeFelice, Ken Sipe
            • Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: