sjust-2023-01-28_05:12:23-crimson-rados-wip-sjust-58486-distro-default-smithi

User	Scheduled	Started	Updated	Runtime	Suite	Branch	Machine Type	Revision	Fail	Dead
sjust	2023-01-28 05:12:23	2023-01-28 05:15:11	2023-01-28 08:42:44	3:27:33	crimson-rados	wip-sjust-58486	smithi	0355c52	11	1

Average wait time: 0:10:07

See other runs of suite 'crimson-rados' on branch 'wip-sjust-58486'?

See other runs on branch 'wip-sjust-58486'?

See other runs scheduled on 2023-01-28?

See detail view?

Status	Job ID	Posted	Started	Updated	Runtime	Duration	In Waiting	Machine	Teuthology Branch	OS Type	OS Version	Description	Nodes
fail	7143216	2023-01-28 05:12:38	2023-01-28 05:15:11	2023-01-28 08:41:22	3:26:11	3:15:43	0:10:28	smithi	main	centos	8.stream	crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_api_tests}	2
Failure Reason: Command failed (workunit test rados/test.sh) on smithi090 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0355c52041532a924eab706716f67e00e8906c5f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'
fail	7143217	2023-01-28 05:12:39	2023-01-28 05:15:41	2023-01-28 08:41:33	3:25:52	3:15:15	0:10:37	smithi	main	centos	8.stream	crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_api_tests_old_format}	1
Failure Reason: Command failed (workunit test rbd/test_librbd.sh) on smithi072 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0355c52041532a924eab706716f67e00e8906c5f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd.sh'
fail	7143218	2023-01-28 05:12:40	2023-01-28 05:15:41	2023-01-28 06:10:18	0:54:37	0:43:08	0:11:29	smithi	main	centos	8.stream	crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/admin_socket_objecter_requests}	2
Failure Reason: Command failed on smithi106 with status 6: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats'
fail	7143219	2023-01-28 05:12:41	2023-01-28 05:17:12	2023-01-28 06:41:33	1:24:21	1:13:57	0:10:24	smithi	main	centos	8.stream	crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench-high-concurrency}	2
Failure Reason: reached maximum tries (500) after waiting for 3000 seconds
fail	7143220	2023-01-28 05:12:42	2023-01-28 05:17:22	2023-01-28 05:46:49	0:29:27	0:19:11	0:10:16	smithi	main	centos	8.stream	crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_cls_tests}	1
Failure Reason: SELinux denials found on ubuntu@smithi135.front.sepia.ceph.com: ['type=AVC msg=audit(1674883816.022:14039): avc: denied { map } for pid=77250 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883562.168:213): avc: denied { read } for pid=6383 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883562.168:213): avc: denied { open } for pid=6383 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883816.022:14038): avc: denied { lock } for pid=77250 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883941.532:17022): avc: denied { read } for pid=94195 comm="rpm" name="Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883815.666:14032): avc: denied { search } for pid=6270 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=525252 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674883941.532:17023): avc: denied { lock } for pid=94195 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883941.532:17024): avc: denied { map } for pid=94195 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525265 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883816.022:14037): avc: denied { open } for pid=77250 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883562.186:215): avc: denied { map } for pid=6383 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883941.532:17022): avc: denied { open } for pid=94195 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525264 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883941.173:16991): avc: denied { read } for pid=697 comm="sssd" name="resolv.conf" dev="sda1" ino=395822 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883816.022:14037): avc: denied { read } for pid=77250 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883562.173:214): avc: denied { lock } for pid=6383 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']
fail	7143221	2023-01-28 05:12:43	2023-01-28 05:18:02	2023-01-28 05:57:26	0:39:24	0:30:17	0:09:07	smithi	main	centos	8.stream	crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/rados_python}	2
Failure Reason: SELinux denials found on ubuntu@smithi035.front.sepia.ceph.com: ['type=AVC msg=audit(1674883860.170:14039): avc: denied { open } for pid=77386 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883860.170:14039): avc: denied { read } for pid=77386 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883985.642:16993): avc: denied { lock } for pid=93999 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883860.170:14041): avc: denied { map } for pid=77386 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883601.077:213): avc: denied { open } for pid=6333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883601.089:214): avc: denied { lock } for pid=6333 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883859.113:14036): avc: denied { search } for pid=6220 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674883985.642:16992): avc: denied { read } for pid=93999 comm="rpm" name="Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883985.642:16992): avc: denied { open } for pid=93999 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=525269 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883601.097:215): avc: denied { map } for pid=6333 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883601.077:213): avc: denied { read } for pid=6333 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883860.170:14040): avc: denied { lock } for pid=77386 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883985.275:16961): avc: denied { read } for pid=695 comm="sssd" name="resolv.conf" dev="sda1" ino=525260 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883985.642:16994): avc: denied { map } for pid=93999 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=525270 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']
dead	7143222	2023-01-28 05:12:45	2023-01-28 05:18:13	2023-01-28 05:23:09	0:04:56			smithi	main	centos	8.stream	crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/radosbench}	2
Failure Reason: Error reimaging machines: 'NoneType' object has no attribute '_fields'
fail	7143223	2023-01-28 05:12:46	2023-01-28 05:18:43	2023-01-28 08:41:23	3:22:40	3:14:10	0:08:30	smithi	main	centos	8.stream	crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_lock_and_fence}	1
Failure Reason: Command failed (workunit test rbd/test_lock_fence.sh) on smithi050 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0355c52041532a924eab706716f67e00e8906c5f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_lock_fence.sh'
fail	7143224	2023-01-28 05:12:47	2023-01-28 05:18:43	2023-01-28 06:12:05	0:53:22	0:43:44	0:09:38	smithi	main	centos	8.stream	crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/small-objects}	2
Failure Reason: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 0 --op snap_remove 0 --op rollback 0 --op setattr 25 --op rmattr 25 --op copy_from 0 --op write_excl 50 --pool unique_pool_0'
fail	7143225	2023-01-28 05:12:48	2023-01-28 05:18:54	2023-01-28 06:13:11	0:54:17	0:42:43	0:11:34	smithi	main	centos	8.stream	crimson-rados/basic/{centos_latest clusters/fixed-2 crimson_qa_overrides deploy/ceph tasks/readwrite}	2
Failure Reason: Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 45 --op write 22 --op delete 10 --op write_excl 22 --pool unique_pool_0'
fail	7143226	2023-01-28 05:12:49	2023-01-28 05:20:04	2023-01-28 08:42:44	3:22:40	3:13:32	0:09:08	smithi	main	centos	8.stream	crimson-rados/rbd/{centos_latest clusters/fixed-1 crimson_qa_overrides deploy/ceph tasks/rbd_python_api_tests_old_format}	1
Failure Reason: Command failed (workunit test rbd/test_librbd_python.sh) on smithi037 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0355c52041532a924eab706716f67e00e8906c5f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 CRIMSON_COMPAT=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/test_librbd_python.sh'
fail	7143227	2023-01-28 05:12:50	2023-01-28 05:20:04	2023-01-28 05:48:14	0:28:10	0:17:59	0:10:11	smithi	main	centos	8.stream	crimson-rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} centos_latest clusters/{fixed-2} crimson_qa_overrides deploy/ceph objectstore/bluestore thrashers/default thrashosds-health workloads/write_fadvise_dontneed}	2
Failure Reason: SELinux denials found on ubuntu@smithi144.front.sepia.ceph.com: ['type=AVC msg=audit(1674883767.661:215): avc: denied { lock } for pid=6380 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674884022.031:14040): avc: denied { open } for pid=77425 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883767.641:214): avc: denied { read } for pid=6380 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674884020.838:14037): avc: denied { search } for pid=6267 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=393230 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1674884022.031:14040): avc: denied { read } for pid=77425 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674884022.031:14042): avc: denied { map } for pid=77425 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883767.641:214): avc: denied { open } for pid=6380 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674884022.031:14041): avc: denied { lock } for pid=77425 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1674883767.661:216): avc: denied { map } for pid=6380 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']