Name Machine Type Up Locked Locked Since Locked By OS Type OS Version Arch Description
smithi053.front.sepia.ceph.com smithi True True 2021-10-19 02:34:01.809442 scheduled_yuriw@teuthology ubuntu 20.04 x86_64 /home/teuthworker/archive/yuriw-2021-10-18_19:04:28-rados-wip-yuri3-testing-2021-10-18-0907-pacific-distro-basic-smithi/6449645
Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
running 6449645 2021-10-18 19:11:10 2021-10-19 02:34:01 2021-10-19 09:01:54 6:28:48 smithi master ubuntu 20.04 rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/few objectstore/bluestore-hybrid rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} 3
pass 6449597 2021-10-18 19:10:26 2021-10-19 02:01:16 2021-10-19 02:33:58 0:32:42 0:22:34 0:10:08 smithi master rhel 8.4 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-zlib rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/default thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 6449562 2021-10-18 19:09:52 2021-10-19 01:37:52 2021-10-19 02:04:31 0:26:39 0:12:29 0:14:10 smithi master centos 8.2 rados/thrash-erasure-code-shec/{ceph clusters/{fixed-4 openstack} mon_election/classic msgr-failures/few objectstore/bluestore-comp-snappy rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=shec-k=4-m=3-c=2} 4
pass 6449525 2021-10-18 19:09:15 2021-10-19 01:07:36 2021-10-19 01:39:03 0:31:27 0:18:07 0:13:20 smithi master ubuntu 20.04 rados/cephadm/smoke/{distro/ubuntu_20.04 fixed-2 mon_election/classic start} 2
pass 6448598 2021-10-18 15:00:50 2021-10-18 15:00:58 2021-10-18 15:31:23 0:30:25 0:17:59 0:12:26 smithi master centos 8.3 rados/cephadm/smoke-roleless/{0-distro/centos_8.3_container_tools_3.0 1-start 2-services/nfs-ingress-rgw 3-final} 2
pass 6448467 2021-10-18 11:20:47 2021-10-18 11:21:13 2021-10-18 11:44:08 0:22:55 0:14:01 0:08:54 smithi master ubuntu 20.04 rgw/thrash/{clusters/fixed-2 frontend/beast install objectstore/filestore-xfs thrasher/default thrashosds-health ubuntu_latest workload/rgw_bucket_quota} 2
pass 6448289 2021-10-18 06:09:28 2021-10-18 06:42:08 2021-10-18 07:26:25 0:44:17 0:36:48 0:07:29 smithi master ubuntu 20.04 rbd/qemu/{cache/none clusters/{fixed-3 openstack} features/journaling msgr-failures/few objectstore/bluestore-hybrid pool/ec-data-pool supported-random-distro$/{ubuntu_latest} workloads/qemu_bonnie} 3
pass 6448240 2021-10-18 06:08:49 2021-10-18 06:08:50 2021-10-18 06:42:53 0:34:03 0:21:56 0:12:07 smithi master ubuntu 20.04 rbd/nbd/{base/install cluster/{fixed-3 openstack} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nbd} 3
dead 6447807 2021-10-17 16:48:24 2021-10-17 16:49:01 2021-10-17 17:07:04 0:18:03 0:06:35 0:11:28 smithi master centos 8.0 rgw:verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/https rgw_pool_type/ec sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} 2
Failure Reason:

{'Failure object was': {'smithi053.front.sepia.ceph.com': {'msg': 'non-zero return code', 'cmd': ['semodule', '-i', '/tmp/nrpe.pp'], 'stdout': '', 'stderr': 'libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).\\nsemodule: Failed on /tmp/nrpe.pp!', 'rc': 1, 'start': '2021-10-17 17:03:36.910373', 'end': '2021-10-17 17:03:41.918059', 'delta': '0:00:05.007686', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'semodule -i /tmp/nrpe.pp', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': 'None', 'chdir': 'None', 'executable': 'None', 'creates': 'None', 'removes': 'None', 'stdin': 'None'}}, 'stdout_lines': [], 'stderr_lines': ['libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).', 'semodule: Failed on /tmp/nrpe.pp!'], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_b35344e81ac507b6cad7c1cae575ce08b9c766f2/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'changed')"}

dead 6447652 2021-10-17 15:13:21 2021-10-17 15:13:21 2021-10-17 17:07:30 1:54:09 1:39:16 0:14:53 smithi master centos 8.3 rados/cephadm/with-work/{0-distro/centos_8.3_container_tools_3.0 fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} 1
Failure Reason:

SSH connection to smithi053 was lost: 'sudo yum -y install librados2'

dead 6447588 2021-10-17 15:09:53 2021-10-17 15:09:53 2021-10-17 15:29:38 0:19:45 smithi master ubuntu 20.04 fs/thrash/workloads/{begin clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{frag_enable session_timeout thrashosds-health whitelist_health whitelist_wrongly_marked_down} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/pjd}} 2
Failure Reason:

Error reimaging machines: reached maximum tries (100) after waiting for 600 seconds

fail 6447521 2021-10-17 12:41:24 2021-10-17 12:41:55 2021-10-17 13:23:58 0:42:03 0:28:49 0:13:14 smithi master centos 8.3 rbd/librbd/{cache/writearound clusters/{fixed-3 openstack} config/permit-partial-discard min-compat-client/octopus msgr-failures/few objectstore/bluestore-comp-zlib pool/replicated-data-pool supported-random-distro$/{centos_8} workloads/c_api_tests_with_defaults} 3
Failure Reason:

"2021-10-17T13:14:27.321956+0000 mon.a (mon.0) 713 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1994 objects degraded (0.050%), 1 pg degraded (PG_DEGRADED)" in cluster log

pass 6447435 2021-10-17 11:57:20 2021-10-17 11:58:01 2021-10-17 12:42:00 0:43:59 0:32:34 0:11:25 smithi master centos 8.3 rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/rgw-datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/bluestore-bitmap overrides proto/http rgw_pool_type/replicated sharding$/{default} striping$/{stripe-greater-than-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/lockdep} 2
pass 6447357 2021-10-17 08:46:09 2021-10-17 08:46:41 2021-10-17 10:21:21 1:34:40 1:23:35 0:11:05 smithi master centos 8.0 rgw/verify/{0-install centos_latest clusters/fixed-2 datacache/no_datacache frontend/beast ignore-pg-availability msgr-failures/few objectstore/filestore-xfs overrides proto/http rgw_pool_type/replicated sharding$/{single} striping$/{stripe-equals-chunk} tasks/{cls ragweed reshard s3tests-java s3tests} validater/valgrind} 2
pass 6446661 2021-10-17 03:37:03 2021-10-17 05:44:19 2021-10-17 06:25:02 0:40:43 0:31:32 0:09:11 smithi master centos 8.2 rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0 1-start 2-thrash 3-tasks/small-objects fixed-2 msgr/async root} 2
pass 6446563 2021-10-17 03:35:45 2021-10-17 05:00:57 2021-10-17 05:44:59 0:44:02 0:37:14 0:06:48 smithi master rhel 8.4 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/bluestore-comp-snappy rados recovery-overrides/{default} supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 2
pass 6446475 2021-10-17 03:34:38 2021-10-17 04:16:06 2021-10-17 05:02:18 0:46:12 0:34:58 0:11:14 smithi master centos 8.2 rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async root} 2
pass 6446344 2021-10-17 03:32:46 2021-10-17 03:32:47 2021-10-17 04:18:49 0:46:02 0:36:37 0:09:25 smithi master centos 8.2 rados/cephadm/thrash/{0-distro/centos_8.2_container_tools_3.0 1-start 2-thrash 3-tasks/rados_api_tests fixed-2 msgr/async-v2only root} 2
pass 6446295 2021-10-16 23:53:01 2021-10-16 23:53:35 2021-10-17 00:37:39 0:44:04 0:36:55 0:07:09 smithi master rhel 8.4 rbd/librbd/{cache/none clusters/{fixed-3 openstack} config/none min-compat-client/default msgr-failures/few objectstore/bluestore-comp-snappy pool/none supported-random-distro$/{rhel_8} workloads/c_api_tests} 3
pass 6446183 2021-10-16 14:33:38 2021-10-16 17:33:19 2021-10-16 17:58:23 0:25:04 0:13:10 0:11:54 smithi master centos 8.2 fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} tasks/openfiletable} 2