ivancich-2019-11-13_17:59:09-rgw-wip-move-some-bucket-listing-logic-to-cls-distro-basic-smithi

rgw multisite, rgw multisite pubsub, test failures

Command failed on smithi134 with status 1: 'cd /home/ubuntu/cephtest/tempest && /home/ubuntu/cephtest/tox-venv/bin/tox -e venv --notest'

Command failed on smithi198 with status 1: 'cd /home/ubuntu/cephtest/hadoop/hadoop-tools/hadoop-aws/ && rm src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java && /home/ubuntu/cephtest/apache-maven-3.6.2/bin/mvn test -Dtest=S3a*,TestS3A*'

rgw multisite pubsub, test failures

Command failed on smithi007 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '(tempest.api.object_storage)(?!.*test_list_containers_reverse_order.*)(?!.*test_list_container_contents_with_end_marker.*)(?!.*test_delete_non_empty_container.*)(?!.*test_container_synchronization.*)(?!.*test_get_object_after_expiration_time.*)(?!.*test_create_object_with_transfer_encoding.*)'"

Command failed on smithi164 with status 1: 'cd /home/ubuntu/cephtest/hadoop/hadoop-tools/hadoop-aws/ && rm src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAWSCredentialsProvider.java && /home/ubuntu/cephtest/apache-maven-3.6.2/bin/mvn -Dit.test=ITestS3A* -Dparallel-tests -Dscale -Dfs.s3a.scale.test.timeout=1200 -Dfs.s3a.scale.test.huge.filesize=256M verify'

rgw multisite, rgw multisite pubsub, test failures

Command failed on smithi102 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '(tempest.api.object_storage)(?!.*test_list_containers_reverse_order.*)(?!.*test_list_container_contents_with_end_marker.*)(?!.*test_delete_non_empty_container.*)(?!.*test_container_synchronization.*)(?!.*test_get_object_after_expiration_time.*)(?!.*test_create_object_with_transfer_encoding.*)'"

{'smithi105.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 102, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 236, in _get_loop_items\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/loader.py", line 562, in get\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/loader.py", line 538, in _load_module_source\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/lookup/items.py\'\n', '_ansible_no_log': None, 'stdout': ''}}

The command was not found or was not executable: ansible-playbook.

{'smithi005.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 102, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 236, in _get_loop_items\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/loader.py", line 562, in get\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/loader.py", line 538, in _load_module_source\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/lookup/items.py\'\n', '_ansible_no_log': None, 'stdout': ''}, 'smithi006.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 102, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 236, in _get_loop_items\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/loader.py", line 562, in get\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/loader.py", line 538, in _load_module_source\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/lookup/items.py\'\n', '_ansible_no_log': None, 'stdout': ''}}

{'smithi162.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 110, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 370, in _run_loop\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 46, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 809, in _execute_module\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 203, in _configure_module\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1015, in modify_module\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/modules/system/user.py\'\n', '_ansible_no_log': False, 'stdout': ''}, 'smithi113.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 110, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 370, in _run_loop\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/normal.py", line 46, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 809, in _execute_module\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 203, in _configure_module\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1015, in modify_module\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/modules/system/user.py\'\n', '_ansible_no_log': False, 'stdout': ''}}

e_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u"The conditional check 'skip_packaging|default(false)|bool != true' failed. The error was: template error while templating string: no filter named 'bool'. String: {% if skip_packaging|default(false)|bool != true %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/tasks/yum/packages.yml': line 50, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Install packages\n ^ here\n")Failure object was: {'smithi068.front.sepia.ceph.com': {'msg': u"The conditional check 'skip_packaging|default(false)|bool != true' failed. The error was: template error while templating string: no filter named 'bool'. String: {% if skip_packaging|default(false)|bool != true %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/tasks/yum/packages.yml': line 50, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Install packages\n ^ here\n", '_ansible_no_log': False}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u'tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)RepresenterError: ('cannot represent an object', u"The conditional check 'skip_packaging|default(false)|bool != true' failed. The error was: template error while templating string: no filter named 'bool'. String: {% if skip_packaging|default(false)|bool != true %} True {% else %} False {% endif %}\n\nThe error appears to be in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/tasks/yum/packages.yml': line 50, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Install packages\n ^ here\n")

{'Failure object was': {'smithi205.front.sepia.ceph.com': {'msg': 'u"the connection plugin \'ssh\' was not found"', '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 309, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/__init__.py", line 281, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 29, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 227, in represent_dict return self.represent_mapping(u\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 125, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 68, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/yaml/representer.py", line 251, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'smithi188.front.sepia.ceph.com': {'msg': 'Unexpected failure during module execution.', 'exception': 'Traceback (most recent call last):\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 110, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 370, in _run_loop\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/shell.py", line 27, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/command.py", line 24, in run\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 809, in _execute_module\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/plugins/action/__init__.py", line 203, in _configure_module\n File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/executor/module_common.py", line 1015, in modify_module\nIOError: [Errno 2] No such file or directory: \'/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/ansible/modules/commands/command.py\'\n', '_ansible_no_log': False, 'stdout': ''}, 'RepresenterError': '(\'cannot represent an object\', u"the connection plugin \'ssh\' was not found")'}

Command failed on smithi180 with status 1: "cd /home/ubuntu/cephtest/tempest && source .tox/venv/bin/activate && tempest run --workspace-path /home/ubuntu/cephtest/tempest/workspace.yaml --workspace rgw --regex '(tempest.api.object_storage)(?!.*test_list_containers_reverse_order.*)(?!.*test_list_container_contents_with_end_marker.*)(?!.*test_delete_non_empty_container.*)(?!.*test_container_synchronization.*)(?!.*test_get_object_after_expiration_time.*)(?!.*test_create_object_with_transfer_encoding.*)'"

"2019-11-13T19:57:49.711816+0000 mon.a (mon.0) 25 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log