Uploaded image for project: 'Marathon'
  1. Marathon
  2. MARATHON-6984

Error in Framework Authentication #5311

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Medium
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Docs
    • Labels:

      Description

      Hello to All,I've got a question in marathon framework authentication.
      I'm following these two docs:
      http://mesos.apache.org/documentation/latest/authentication/
      http://mesosphere.github.io/marathon/docs/framework-authentication.html

      The mesos-slave can connect to the master by authentication but marathon framework can not .

      I have three mesos-masters and use zookeeper for HA. Here is the configuration of the first mesos-master:
      [root@centos7-01 mesos-master]# pwd
      /etc/mesos-master

      [root@centos7-01 mesos-master]# ls -l
      total 16
      rw-rr- 1 root root 0 Mar 6 22:18 ?authenticate
      rw-rr- 1 root root 0 Mar 6 22:19 ?authenticate_agents
      rw-rr- 1 root root 0 Mar 6 22:40 ?authenticate_http_readonly
      rw-rr- 1 root root 0 Mar 6 22:40 ?authenticate_http_readwrite
      rw-rr- 1 root root 43 Mar 6 22:23 credentials
      rw-rr- 1 root root 14 Feb 3 23:29 hostname
      rw-rw-r- 1 root root 2 Feb 3 23:23 quorum
      rw-rw-r- 1 root root 15 Nov 16 09:46 work_dir

      [root@centos7-01 mesos-master]# cat credentials
      /home/mesosuser/master_key/credentials.bak

      [root@centos7-01 mesos-master]# cat /home/mesosuser/master_key/credentials.bak
      {
      "credentials" : [
      {
      "principal": "principal1",
      "secret": "secret1"
      },
      {
      "principal": "marathon",
      "secret": "marathonpassword"
      }
      ]
      }

      mesos-slaves connect to mesos-master successfully by using the first principal (principal1,secret1) .

      Here is the marathon configuration:
      [root@centos7-01 conf]# pwd
      /etc/marathon/conf

      [root@centos7-01 conf]# ls -l
      total 32
      rw-rr- 1 root root 0 Mar 3 23:09 ?disable_http
      rw-rr- 1 root root 14 Feb 3 23:29 hostname
      rw-rr- 1 root root 15 Feb 14 22:15 http_credentials
      rw-rr- 1 root root 0 Feb 21 21:27 ?leader_proxy_ssl_ignore_hostname
      rw-rr- 1 root root 68 Feb 3 23:33 master
      rw-rr- 1 root root 0 Mar 7 19:18 ?mesos_authentication
      rw-rr- 1 root root 9 Mar 7 19:18 mesos_authentication_principal
      rw-rr- 1 root root 50 Mar 7 20:16 mesos_authentication_secret_file
      rw-rr- 1 root root 9 Feb 15 19:36 ssl_keystore_password
      rw-rr- 1 root root 36 Feb 16 22:56 ssl_keystore_path
      rw-rr- 1 root root 71 Feb 3 23:34 zk

      [root@centos7-01 conf]# cat mesos_authentication_principal
      marathon

      [root@centos7-01 conf]# cat mesos_authentication_secret_file
      /home/mesosuser/marathon_framwork_key/marathonkey

      [root@centos7-01 conf]# cat /home/mesosuser/marathon_framwork_key/marathonkey
      marathonpassword

      [root@centos7-01 conf]# service marathon restart
      Redirecting to /bin/systemctl restart marathon.service

      Marathon failed to register with the mesos-master because of authentication failure.

      log:
      Mar 07 20:51:31 centos7-01 marathon[18315]: 2017-03-07 20:51:31,106:18293(0x7f4052583700):ZOO_INFO@log_env@747: Client environment:user.name=(null)
      Mar 07 20:51:31 centos7-01 marathon[18315]: 2017-03-07 20:51:31,107:18293(0x7f4052583700):ZOO_INFO@log_env@755: Client environment:user.home=/root
      Mar 07 20:51:31 centos7-01 marathon[18315]: 2017-03-07 20:51:31,107:18293(0x7f4052583700):ZOO_INFO@log_env@767: Client environment:user.dir=/
      Mar 07 20:51:31 centos7-01 marathon[18315]: 2017-03-07 20:51:31,107:18293(0x7f4052583700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=192.168.1.101:2181,192.168.1.102:2181,192.168.1.103:2181 sessionTimeout
      Mar 07 20:51:31 centos7-01 marathon[18315]: 2017-03-07 20:51:31,108:18293(0x7f4050376700):ZOO_INFO@check_events@1728: initiated connection to server [192.168.1.102:2181]
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,110] INFO reconcile [/demo/tomcatlb] with latest version [2017-03-03T12:09:04.293Z] (mesosphere.marathon.core.health.impl.MarathonHealthCheckManager:ForkJoinPoo
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,111] INFO reconcile [/demo/vmstat] with latest version [2017-02-22T12:50:50.681Z] (mesosphere.marathon.core.health.impl.MarathonHealthCheckManager:ForkJoinPool-
      Mar 07 20:51:31 centos7-01 marathon[18315]: 2017-03-07 20:51:31,117:18293(0x7f4050376700):ZOO_INFO@check_events@1775: session establishment complete on server [192.168.1.102:2181], sessionId=0x25aa40f676b0172, negotiated time
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.118002 18417 group.cpp:340] Group process (zookeeper-group(1)@192.168.1.101:24664) connected to ZooKeeper
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.118038 18417 group.cpp:828] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.118048 18417 group.cpp:418] Trying to create path '/mesos' in ZooKeeper
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,115] INFO Creating tombstone for old twitter commons leader election (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-1-thread-1)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,117] INFO addAllFor [/demo/tomcatlb] version [2017-03-03T12:09:04.293Z] (mesosphere.marathon.core.health.impl.MarathonHealthCheckManager:ForkJoinPool-2-worker-7
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,118] INFO addAllFor [/demo/vmstat] version [2017-02-22T12:50:50.681Z] (mesosphere.marathon.core.health.impl.MarathonHealthCheckManager:ForkJoinPool-2-worker-15)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,118] INFO Starting scheduler actor (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-11)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,119] INFO Scheduler actor ready (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-11)
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.125054 18417 detector.cpp:152] Detected a new leader: (id='102')
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.125149 18417 group.cpp:697] Trying to get '/mesos/json.info_0000000102' in ZooKeeper
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.125790 18417 zookeeper.cpp:259] A new leading master (UPID=master@192.168.1.101:5050) is detected
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.125836 18417 sched.cpp:330] New master detected at master@192.168.1.101:5050
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.125964 18417 sched.cpp:396] Authenticating with master master@192.168.1.101:5050
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.125972 18417 sched.cpp:403] Using default CRAM-MD5 authenticatee
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.126215 18415 authenticatee.cpp:97] Initializing client SASL
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.127928 18415 authenticatee.cpp:121] Creating new client SASL connection
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.128970 18416 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.128996 18416 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.131512 18416 authenticatee.cpp:259] Received SASL authentication step
      Mar 07 20:51:31 centos7-01 marathon[18315]: E0307 20:51:31.133486 18416 sched.cpp:496] Master master@192.168.1.101:5050 refused authentication
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.133502 18416 sched.cpp:1171] Got error 'Master refused authentication'
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.133505 18416 sched.cpp:2029] Asked to abort the driver
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,143] WARN Error: Master refused authentication
      Mar 07 20:51:31 centos7-01 marathon[18314]: In case Mesos does not allow registration with the current frameworkId, delete the ZooKeeper Node: /marathon/state/framework:id
      Mar 07 20:51:31 centos7-01 marathon[18314]: CAUTION: if you remove this node, all tasks started with the current frameworkId will be orphaned! (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$52061705:Thread-14)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,144] ERROR Committing suicide! (mesosphere.marathon.MarathonScheduler$$EnhancerByGuice$$52061705:Thread-14)
      Mar 07 20:51:31 centos7-01 marathon[18315]: I0307 20:51:31.147397 18416 sched.cpp:1217] Aborting framework a98fd49b-5f86-4659-9946-8fffafeab5fd-0032
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,147] INFO Driver future completed with result=Success(()). (mesosphere.marathon.MarathonSchedulerService$$EnhancerByGuice$$3a689db7:ForkJoinPool-2-worker-5)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,148] INFO Abdicating leadership while leading (reoffer=true) (mesosphere.marathon.core.election.impl.CuratorElectionService:ForkJoinPool-2-worker-5)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,150] INFO Call postDriverRuns callbacks on EntityStoreCache(MarathonStore(app), EntityStoreCache(MarathonStore(group), EntityStoreCache(MarathonStore(deploy
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,150] INFO Defeated (LeaderLatchListener Interface). New leader: - (mesosphere.marathon.core.election.impl.CuratorElectionService:pool-1-thread-1)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,150] INFO Finished postDriverRuns callbacks (mesosphere.marathon.MarathonSchedulerService$$EnhancerByGuice$$3a689db7:ForkJoinPool-2-worker-5)
      Mar 07 20:51:31 centos7-01 marathon[18314]: [2017-03-07 20:51:31,167] INFO Shutting down services (mesosphere.marathon.Main$:shutdownHook1)
      Mar 07 20:51:31 centos7-01 systemd[1]: marathon.service: main process exited, code=exited, status=137/n/a
      Mar 07 20:51:31 centos7-01 systemd[1]: Unit marathon.service entered failed state.
      Mar 07 20:51:31 centos7-01 systemd[1]: marathon.service failed.

      I have no idea about this question.Any help will be appreciated.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              marco.monaco Marco Monaco
              Team:
              Orchestration Team
              Watchers:
              Jason Gilanfarr (Inactive), Marco Monaco, mosyang
            • Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: