Uploaded image for project: 'DC/OS'
  1. DC/OS
  2. DCOS_OSS-4568

test_metrics.deploy_and_cleanup_dcos_package Framework hello-world still running

    Details

    • Sprint:
      Observability Team Sprint 35, Observability Team Sprint 36, Observability Team Sprint 39
    • Story Points:
      2
    • Transition Due Date:

      Description

      In CI test_metrics.test_executor_metrics_metadata test failed with

      test_metrics.py:234 (test_executor_metrics_metadata)
      dcos_api_session = <dcos_test_utils.dcos_api.DcosApiSession object at 0x7fcd5bcac320>
      
          @pytest.mark.skipif(expanded_config.get('security') == 'strict',
                              reason="Framework disabled for strict mode")
          def test_executor_metrics_metadata(dcos_api_session):
              """Test that executor metrics have expected metadata/labels"""
              with deploy_and_cleanup_dcos_package(dcos_api_session, 'hello-world', '2.2.0-0.42.2', 'hello-world'):
                  node = get_task_hostname(dcos_api_session, 'marathon', 'hello-world')
          
                  @retrying.retry(wait_fixed=2000, stop_max_delay=300 * 1000)
                  def check_executor_metrics_metadata():
                      response = get_metrics_prom(dcos_api_session, node)
                      for line in response.text.splitlines():
                          if '#' in line:
                              continue
                          # ignore metrics from hello-world task started by marathon by checking
                          # for absence of 'marathon' string.
                          if 'cpus_nr_periods' in line and 'marathon' not in line:
                              assert 'service_name="hello-world"' in line
                              assert 'task_name=""' in line  # this is an executor, not a task
                              # hello-world executors can be named "hello" or "world"
                              assert ('executor_name="hello"' in line or 'executor_name="world"' in line)
      >           check_executor_metrics_metadata()
      
      test_metrics.py:255: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      ../python--f5920be9c457fe7ef17c62100f366d4cbdba5963/lib/python3.6/contextlib.py:88: in __exit__
          next(self.gen)
      test_metrics.py:186: in deploy_and_cleanup_dcos_package
          wait_for_package_teardown()
      ../../lib/python3.6/site-packages/retrying.py:49: in wrapped_f
          return Retrying(*dargs, **dkw).call(f, *args, **kw)
      ../../lib/python3.6/site-packages/retrying.py:212: in call
          raise attempt.get()
      ../../lib/python3.6/site-packages/retrying.py:247: in get
          six.reraise(self.value[0], self.value[1], self.value[2])
      ../../lib/python3.6/site-packages/six.py:686: in reraise
          raise value
      ../../lib/python3.6/site-packages/retrying.py:200: in call
          attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      
          @retrying.retry(wait_fixed=5000, stop_max_delay=150 * 1000)
          def wait_for_package_teardown():
              state_response = dcos_api_session.get('/state', host=dcos_api_session.masters[0], port=5050)
              assert state_response.status_code == 200
              state = state_response.json()
          
              for framework in state['frameworks']:
                  if framework['name'] == framework_name:
      >               raise Exception('Framework {} still running'.format(framework_name))
      E               Exception: Framework hello-world still running
      
      test_metrics.py:185: Exception
      

        Attachments

          Activity

            People

            • Assignee:
              gracedo Grace Do
              Reporter:
              jonathangiddy Jonathan Giddy
              Team:
              Observability Team
              Watchers:
              Adam Dangoor, Alex Rukletsov (Inactive), Branden Rolston, Carter Gawron, Charles Provencher, Daniel Baker, Deepak Goel, Gastón Kleiman, Gauri Powale, Gilbert Song, Grace Do, Greg Mann, Gustav Paul, Jan-Philip Gehrcke, Jonathan Giddy, Julian Gieseke, Kevin Klues (Inactive), Lisa Gunn, Matthias Eichstedt, Mergebot, Orlando Hohmeier, Philip Norman, Senthil Kumaran, Sergey Urbanovich, Till Toenshoff, Tim Weidner
            • Watchers:
              26 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: