Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
dead 7143854 2023-01-30 12:35:42 2023-01-30 12:42:16 2023-01-31 01:05:54 12:23:38 smithi main centos 8.stream fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/3 tasks/fsstress validater/valgrind} 2
Failure Reason:

hit max job timeout

fail 7143855 2023-01-30 12:35:43 2023-01-30 12:43:56 2023-01-30 13:14:54 0:30:58 0:19:34 0:11:24 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi040 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b9138372-a09e-11ed-9e56-001a4aab830c -e sha1=860d3ac8d012f04a6667eb411e085656deb62181 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

fail 7143856 2023-01-30 12:35:44 2023-01-30 12:43:57 2023-01-30 13:29:08 0:45:11 0:28:18 0:16:53 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

"2023-01-30T13:16:09.887594+0000 mgr.y (mgr.14103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143857 2023-01-30 12:35:45 2023-01-30 12:49:48 2023-01-30 13:34:20 0:44:32 0:33:49 0:10:43 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/pjd}} 3
Failure Reason:

Command failed (workunit test suites/pjd.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=860d3ac8d012f04a6667eb411e085656deb62181 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

fail 7143858 2023-01-30 12:35:46 2023-01-30 12:52:48 2023-01-30 13:28:26 0:35:38 0:24:26 0:11:12 smithi main rhel 8.6 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/none objectstore-ec/bluestore-ec-root overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/no prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/suites/pjd}} 2
Failure Reason:

SELinux denials found on ubuntu@smithi098.front.sepia.ceph.com: ['type=AVC msg=audit(1675083730.383:197): avc: denied { open } for pid=1470 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675083730.383:198): avc: denied { lock } for pid=1470 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084393.377:17127): avc: denied { map } for pid=78307 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524340 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084248.891:14788): avc: denied { lock } for pid=67598 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084393.377:17125): avc: denied { open } for pid=78307 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524330 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084248.891:14787): avc: denied { open } for pid=67598 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675083730.383:197): avc: denied { read } for pid=1470 comm="rpm" name="Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084248.891:14789): avc: denied { map } for pid=67598 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=262275 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084393.377:17126): avc: denied { lock } for pid=78307 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524330 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084393.377:17125): avc: denied { read } for pid=78307 comm="rpm" name="Packages" dev="sda1" ino=524330 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084248.891:14787): avc: denied { read } for pid=67598 comm="rpm" name="Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675083730.383:199): avc: denied { map } for pid=1470 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=262275 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084392.876:17102): avc: denied { read } for pid=710 comm="sssd" name="resolv.conf" dev="sda1" ino=524378 scontext=system_u:system_r:sssd_t:s0 tcontext=system_u:object_r:net_conf_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675084247.951:14770): avc: denied { search } for pid=1389 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1']

fail 7143859 2023-01-30 12:35:46 2023-01-30 12:53:29 2023-01-30 13:25:47 0:32:18 0:19:55 0:12:23 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi066 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5f1cd010-a0a0-11ed-9e56-001a4aab830c -e sha1=860d3ac8d012f04a6667eb411e085656deb62181 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

fail 7143860 2023-01-30 12:35:47 2023-01-30 12:53:29 2023-01-30 13:47:19 0:53:50 0:37:42 0:16:08 smithi main ubuntu 20.04 fs/mirror-ha/{begin/{0-install 1-ceph 2-logrotate} cephfs-mirror/three-per-cluster clients/{mirror} cluster/{1-node} objectstore/bluestore-bitmap overrides/{whitelist_health} supported-random-distro$/{ubuntu_latest} workloads/cephfs-mirror-ha-workunit} 1
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

fail 7143861 2023-01-30 12:35:48 2023-01-30 12:55:40 2023-01-30 15:26:00 2:30:20 2:15:45 0:14:35 smithi main ubuntu 20.04 fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mds 2-workunit/fs/snaps}} 2
Failure Reason:

Command failed (workunit test fs/snaps/snaptest-multiple-capsnaps.sh) on smithi111 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=860d3ac8d012f04a6667eb411e085656deb62181 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-multiple-capsnaps.sh'

fail 7143862 2023-01-30 12:35:49 2023-01-30 12:55:40 2023-01-30 13:54:35 0:58:55 0:28:56 0:29:59 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

"2023-01-30T13:40:06.842018+0000 mgr.y (mgr.14106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143863 2023-01-30 12:35:50 2023-01-30 13:13:33 2023-01-30 13:39:44 0:26:11 0:13:52 0:12:19 smithi main ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-limits} 2
Failure Reason:

Test failure: test_client_cache_size (tasks.cephfs.test_client_limits.TestClientLimits)

fail 7143864 2023-01-30 12:35:51 2023-01-30 13:13:53 2023-01-30 13:46:01 0:32:08 0:21:19 0:10:49 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

"2023-01-30T13:37:01.638861+0000 mgr.y (mgr.14102) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

pass 7143865 2023-01-30 12:35:52 2023-01-30 13:15:03 2023-01-30 14:48:47 1:33:44 1:12:10 0:21:34 smithi main ubuntu 20.04 fs/fscrypt/{begin/{0-install 1-ceph 2-logrotate} bluestore-bitmap clusters/1-mds-1-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/v1 mount overrides/{distro/testing/k-testing}} overrides/{ignorelist_health ignorelist_health_more ignorelist_wrongly_marked_down pg-warn} tasks/fscrypt-dbench} 3
fail 7143866 2023-01-30 12:35:53 2023-01-30 13:24:15 2023-01-30 13:58:13 0:33:58 0:19:59 0:13:59 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi066 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e3c029c6-a0a4-11ed-9e56-001a4aab830c -e sha1=860d3ac8d012f04a6667eb411e085656deb62181 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

fail 7143867 2023-01-30 12:35:54 2023-01-30 13:25:56 2023-01-30 14:29:24 1:03:28 0:51:26 0:12:02 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{no-subvolume} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/blogbench}} 3
Failure Reason:

error during scrub thrashing: rank damage found: {'backtrace'}

fail 7143868 2023-01-30 12:35:54 2023-01-30 13:28:36 2023-01-30 14:08:20 0:39:44 0:27:17 0:12:27 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

"2023-01-30T13:54:53.248733+0000 mgr.y (mgr.14106) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143869 2023-01-30 12:35:55 2023-01-30 13:29:17 2023-01-30 14:08:46 0:39:29 0:28:53 0:10:36 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

"2023-01-30T13:55:49.302211+0000 mgr.y (mgr.24103) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

dead 7143870 2023-01-30 12:35:56 2023-01-30 13:29:47 2023-01-31 04:21:13 14:51:26 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} 3
Failure Reason:

hit max job timeout

fail 7143871 2023-01-30 12:35:57 2023-01-30 13:31:38 2023-01-30 14:03:52 0:32:14 0:19:24 0:12:50 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi131 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 967de6b6-a0a5-11ed-9e56-001a4aab830c -e sha1=860d3ac8d012f04a6667eb411e085656deb62181 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

fail 7143872 2023-01-30 12:35:58 2023-01-30 13:32:48 2023-01-30 14:08:09 0:35:21 0:22:25 0:12:56 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

"2023-01-30T13:58:19.890951+0000 mgr.x (mgr.14099) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143873 2023-01-30 12:35:59 2023-01-30 13:34:29 2023-01-30 14:32:43 0:58:14 0:42:20 0:15:54 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsstress}} 3
Failure Reason:

SELinux denials found on ubuntu@smithi016.front.sepia.ceph.com: ['type=AVC msg=audit(1675088667.367:22627): avc: denied { search } for pid=1368 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675088221.990:21853): avc: denied { search } for pid=1368 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675087039.821:14851): avc: denied { map } for pid=67843 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=262275 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087039.110:14832): avc: denied { search } for pid=1368 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675088668.763:22639): avc: denied { open } for pid=149263 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088667.800:22628): avc: denied { open } for pid=149211 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088225.433:21854): avc: denied { open } for pid=143524 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088225.433:21855): avc: denied { lock } for pid=143524 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088667.800:22628): avc: denied { read } for pid=149211 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087039.821:14849): avc: denied { read } for pid=67843 comm="rpm" name="Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088225.433:21854): avc: denied { read } for pid=143524 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087187.140:17157): avc: denied { open } for pid=78431 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087187.140:17157): avc: denied { read } for pid=78431 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087187.140:17158): avc: denied { lock } for pid=78431 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088667.800:22629): avc: denied { lock } for pid=149211 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087039.821:14849): avc: denied { open } for pid=67843 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088667.800:22630): avc: denied { map } for pid=149211 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087187.444:17168): avc: denied { read } for pid=659 comm="sssd" name="resolv.conf" dev="sda1" ino=524383 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087186.607:17134): avc: denied { read } for pid=659 comm="sssd" name="resolv.conf" dev="sda1" ino=524381 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088668.763:22641): avc: denied { map } for pid=149263 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088668.763:22639): avc: denied { read } for pid=149263 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087039.821:14850): avc: denied { lock } for pid=67843 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087187.141:17159): avc: denied { map } for pid=78431 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088668.763:22640): avc: denied { lock } for pid=149263 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088225.433:21856): avc: denied { map } for pid=143524 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']

fail 7143874 2023-01-30 12:36:00 2023-01-30 13:39:50 2023-01-30 14:21:47 0:41:57 0:28:30 0:13:27 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

"2023-01-30T14:08:20.523580+0000 mgr.x (mgr.14098) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143875 2023-01-30 12:36:01 2023-01-30 13:42:41 2023-01-30 14:13:12 0:30:31 0:13:27 0:17:04 smithi main ubuntu 20.04 fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} 2
Failure Reason:

Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)

fail 7143876 2023-01-30 12:36:02 2023-01-30 13:46:52 2023-01-30 14:14:07 0:27:15 0:15:00 0:12:15 smithi main ubuntu 20.04 fs/libcephfs/{begin/{0-install 1-ceph 2-logrotate} clusters/1-mds-1-client-coloc conf/{client mds mon osd} distro/{ubuntu_latest} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/libcephfs/{frag test}} 2
Failure Reason:

Command failed (workunit test libcephfs/test.sh) on smithi029 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=860d3ac8d012f04a6667eb411e085656deb62181 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/libcephfs/test.sh'

fail 7143877 2023-01-30 12:36:03 2023-01-30 13:46:52 2023-01-30 15:16:23 1:29:31 1:14:41 0:14:50 smithi main rhel 8.6 fs/snaps/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-1c-client conf/{client mds mon osd} distro/{rhel_8} mount/fuse objectstore-ec/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/workunit/snaps} 2
Failure Reason:

SELinux denials found on ubuntu@smithi148.front.sepia.ceph.com: ['type=AVC msg=audit(1675087205.084:220): avc: denied { map } for pid=1497 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=262275 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090920.420:21498): avc: denied { lock } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090917.069:21492): avc: denied { lock } for pid=115647 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089667.350:20725): avc: denied { open } for pid=112064 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090916.343:21487): avc: denied { open } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090920.558:21499): avc: denied { getattr } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=524368 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089945.288:20789): avc: denied { lock } for pid=112265 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090917.069:21493): avc: denied { map } for pid=115647 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090920.419:21497): avc: denied { read write } for pid=115582 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090918.399:21495): avc: denied { lock } for pid=115650 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089945.287:20788): avc: denied { read } for pid=112265 comm="rpm" name="Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090918.399:21494): avc: denied { open } for pid=115650 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090916.343:21489): avc: denied { getattr } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=524368 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090917.069:21491): avc: denied { read } for pid=115647 comm="rpm" name="Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087549.652:14875): avc: denied { open } for pid=67939 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090920.419:21497): avc: denied { open } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090916.343:21490): avc: denied { map } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=524368 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087549.652:14877): avc: denied { map } for pid=67939 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=262275 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090917.069:21491): avc: denied { open } for pid=115647 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090918.399:21496): avc: denied { map } for pid=115650 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089667.355:20726): avc: denied { lock } for pid=112064 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090920.559:21500): avc: denied { map } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=524368 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087205.084:218): avc: denied { read } for pid=1497 comm="rpm" name="Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090918.399:21494): avc: denied { read } for pid=115650 comm="rpm" name="Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089945.287:20788): avc: denied { open } for pid=112265 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090916.343:21487): avc: denied { read write } for pid=115582 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089667.356:20727): avc: denied { map } for pid=112064 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087548.857:14846): avc: denied { search } for pid=1397 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524301 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675087549.652:14876): avc: denied { lock } for pid=67939 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089667.350:20725): avc: denied { read } for pid=112064 comm="rpm" name="Packages" dev="sda1" ino=524344 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675087549.652:14875): avc: denied { read } for pid=67939 comm="rpm" name="Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089662.889:20724): avc: denied { search } for pid=1397 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524301 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675087205.084:218): avc: denied { open } for pid=1497 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675090916.343:21488): avc: denied { lock } for pid=115582 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089945.288:20790): avc: denied { map } for pid=112265 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089941.703:20787): avc: denied { search } for pid=1397 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524301 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675087205.084:219): avc: denied { lock } for pid=1497 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']

fail 7143878 2023-01-30 12:36:03 2023-01-30 13:51:13 2023-01-30 14:32:59 0:41:46 0:29:30 0:12:16 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

"2023-01-30T14:17:47.533720+0000 mgr.y (mgr.14109) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143879 2023-01-30 12:36:04 2023-01-30 13:51:53 2023-01-30 14:25:27 0:33:34 0:19:26 0:14:08 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi179 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 91160818-a0a8-11ed-9e56-001a4aab830c -e sha1=860d3ac8d012f04a6667eb411e085656deb62181 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

fail 7143880 2023-01-30 12:36:05 2023-01-30 13:54:44 2023-01-30 14:32:29 0:37:45 0:21:52 0:15:53 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/no pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

"2023-01-30T14:22:32.435994+0000 mgr.y (mgr.14104) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143881 2023-01-30 12:36:06 2023-01-30 13:58:15 2023-01-30 20:58:45 7:00:30 6:44:24 0:16:06 smithi main ubuntu 20.04 fs/verify/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu/{latest overrides}} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug session_timeout} ranks/5 tasks/fsstress validater/valgrind} 2
Failure Reason:

Command failed (workunit test suites/fsstress.sh) on smithi131 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=860d3ac8d012f04a6667eb411e085656deb62181 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'

fail 7143882 2023-01-30 12:36:07 2023-01-30 14:03:56 2023-01-30 14:57:37 0:53:41 0:38:00 0:15:41 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

SELinux denials found on ubuntu@smithi080.front.sepia.ceph.com: ['type=AVC msg=audit(1675088591.630:14101): avc: denied { read } for pid=72679 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088591.630:14102): avc: denied { lock } for pid=72679 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088728.381:17058): avc: denied { map } for pid=89545 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524329 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088591.630:14101): avc: denied { open } for pid=72679 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088296.257:214): avc: denied { map } for pid=1583 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088728.381:17056): avc: denied { read } for pid=89545 comm="rpm" name="Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088296.227:212): avc: denied { open } for pid=1583 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088728.381:17057): avc: denied { lock } for pid=89545 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088296.249:213): avc: denied { lock } for pid=1583 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088591.630:14103): avc: denied { map } for pid=72679 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088727.968:17025): avc: denied { read } for pid=704 comm="sssd" name="resolv.conf" dev="sda1" ino=524368 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088296.227:212): avc: denied { read } for pid=1583 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088728.381:17056): avc: denied { open } for pid=89545 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088591.041:14098): avc: denied { search } for pid=1470 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1']

fail 7143883 2023-01-30 12:36:08 2023-01-30 14:08:27 2023-01-30 14:45:45 0:37:18 0:26:44 0:10:34 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/quincy}} 3
Failure Reason:

"2023-01-30T14:35:06.454679+0000 mgr.x (mgr.14100) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143884 2023-01-30 12:36:09 2023-01-30 14:08:47 2023-01-30 14:35:00 0:26:13 0:14:57 0:11:16 smithi main centos 8.stream fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota} 2
Failure Reason:

Command failed (workunit test fs/quota/quota.sh) on smithi086 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=860d3ac8d012f04a6667eb411e085656deb62181 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/quota/quota.sh'

fail 7143885 2023-01-30 12:36:10 2023-01-30 14:10:08 2023-01-30 14:59:54 0:49:46 0:39:42 0:10:04 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/pjd}} 3
Failure Reason:

Command failed (workunit test suites/pjd.sh) on smithi033 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=860d3ac8d012f04a6667eb411e085656deb62181 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

fail 7143886 2023-01-30 12:36:11 2023-01-30 14:10:38 2023-01-30 14:40:31 0:29:53 0:15:59 0:13:54 smithi main centos 8.stream fs/permission/{begin/{0-install 1-ceph 2-logrotate} clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore-ec/bluestore-comp overrides/{ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_pjd} 2
Failure Reason:

SELinux denials found on ubuntu@smithi178.front.sepia.ceph.com: ['type=AVC msg=audit(1675089027.459:17112): avc: denied { open } for pid=89804 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089027.459:17113): avc: denied { lock } for pid=89804 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089026.659:17057): avc: denied { lock } for pid=89542 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089026.659:17056): avc: denied { open } for pid=89542 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089027.459:17112): avc: denied { read } for pid=89804 comm="rpm" name="Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088594.639:212): avc: denied { open } for pid=1579 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088890.657:14100): avc: denied { lock } for pid=72621 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088890.657:14099): avc: denied { open } for pid=72621 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088890.657:14101): avc: denied { map } for pid=72621 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088594.649:214): avc: denied { map } for pid=1579 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=98535 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088890.093:14096): avc: denied { search } for pid=1466 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675088890.657:14099): avc: denied { read } for pid=72621 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088594.639:212): avc: denied { read } for pid=1579 comm="rpm" name="Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675088594.643:213): avc: denied { lock } for pid=1579 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=98534 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089027.459:17114): avc: denied { map } for pid=89804 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524329 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089026.659:17058): avc: denied { map } for pid=89542 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524329 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089026.248:17025): avc: denied { read } for pid=707 comm="sssd" name="resolv.conf" dev="sda1" ino=524368 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089026.659:17056): avc: denied { read } for pid=89542 comm="rpm" name="Packages" dev="sda1" ino=524328 scontext=system_u:system_r:setroubleshootd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']

fail 7143887 2023-01-30 12:36:11 2023-01-30 14:13:19 2023-01-30 14:55:23 0:42:04 0:30:22 0:11:42 smithi main centos 8.stream fs/upgrade/featureful_client/old_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-compat_client/no}} 3
Failure Reason:

"2023-01-30T14:41:51.495942+0000 mgr.x (mgr.14110) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log

fail 7143888 2023-01-30 12:36:12 2023-01-30 14:14:09 2023-01-30 14:46:39 0:32:30 0:19:15 0:13:15 smithi main centos 8.stream fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} 2
Failure Reason:

Command failed on smithi145 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a16a3a24-a0ab-11ed-9e56-001a4aab830c -e sha1=860d3ac8d012f04a6667eb411e085656deb62181 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr'"

fail 7143889 2023-01-30 12:36:13 2023-01-30 14:15:30 2023-01-30 16:03:46 1:48:16 1:34:59 0:13:17 smithi main rhel 8.6 fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/fs/misc}} 3
Failure Reason:

SELinux denials found on ubuntu@smithi027.front.sepia.ceph.com: ['type=AVC msg=audit(1675093294.317:26220): avc: denied { map } for pid=275751 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=524368 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.317:26217): avc: denied { open } for pid=275751 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089342.229:14849): avc: denied { open } for pid=67870 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092243.614:24935): avc: denied { map } for pid=265429 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089489.153:17161): avc: denied { map } for pid=78469 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092528.198:25743): avc: denied { open } for pid=271902 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089489.153:17159): avc: denied { open } for pid=78469 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089489.153:17160): avc: denied { lock } for pid=78469 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089489.153:17159): avc: denied { read } for pid=78469 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092528.198:25743): avc: denied { read } for pid=271902 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.317:26219): avc: denied { getattr } for pid=275751 comm="rhsmcertd-worke" path="/var/lib/rpm/__db.001" dev="sda1" ino=524368 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.746:26222): avc: denied { lock } for pid=275857 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.746:26221): avc: denied { read } for pid=275857 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092242.528:24932): avc: denied { search } for pid=1397 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675089341.298:14832): avc: denied { search } for pid=1397 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675092528.198:25745): avc: denied { map } for pid=271902 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089488.616:17134): avc: denied { read } for pid=710 comm="sssd" name="resolv.conf" dev="sda1" ino=524381 scontext=system_u:system_r:sssd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092243.596:24934): avc: denied { lock } for pid=265429 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089342.230:14850): avc: denied { lock } for pid=67870 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092528.198:25744): avc: denied { lock } for pid=271902 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.317:26217): avc: denied { read write } for pid=275751 comm="rhsmcertd-worke" name=".dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089342.229:14849): avc: denied { read } for pid=67870 comm="rpm" name="Packages" dev="sda1" ino=262274 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092243.572:24933): avc: denied { open } for pid=265429 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.317:26218): avc: denied { lock } for pid=275751 comm="rhsmcertd-worke" path="/var/lib/rpm/.dbenv.lock" dev="sda1" ino=524367 scontext=system_u:system_r:rhsmcertd_t:s0 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675089342.230:14851): avc: denied { map } for pid=67870 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=262275 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092527.760:25742): avc: denied { search } for pid=1397 comm=72733A6D61696E20513A526567 name="cephtest" dev="sda1" ino=524293 scontext=system_u:system_r:syslogd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir permissive=1', 'type=AVC msg=audit(1675093294.746:26223): avc: denied { map } for pid=275857 comm="rpm" path="/var/lib/rpm/Name" dev="sda1" ino=524345 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675092243.572:24933): avc: denied { read } for pid=265429 comm="rpm" name="Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1675093294.746:26221): avc: denied { open } for pid=275857 comm="rpm" path="/var/lib/rpm/Packages" dev="sda1" ino=524335 scontext=system_u:system_r:setroubleshootd_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1']

fail 7143890 2023-01-30 12:36:14 2023-01-30 14:15:50 2023-01-30 14:55:03 0:39:13 0:21:35 0:17:38 smithi main centos 8.stream fs/upgrade/featureful_client/upgraded_client/{bluestore-bitmap centos_latest clusters/1-mds-2-client-micro conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down multimds/yes pg-warn} tasks/{0-octopus 1-client 2-upgrade 3-client-upgrade 4-compat_client 5-client-sanity}} 3
Failure Reason:

"2023-01-30T14:44:51.191692+0000 mgr.y (mgr.14105) 1 : cluster [ERR] Failed to load ceph-mgr modules: prometheus" in cluster log