| User | Scheduled | Started | Updated | Runtime | Suite | Branch | Machine Type | Revision | Pass | Fail | Dead |
|---|---|---|---|---|---|---|---|---|---|---|---|
| yuriw | 2024-08-29 14:13:31 | 2024-08-29 14:39:45 | 2024-08-29 23:20:42 | 8:40:57 | rados | wip-yuri4-testing-2024-08-28-1359 | smithi | 1f4056d | 11 | 22 | 4 |
| Status | Job ID | Links | Posted | Started | Updated | Runtime |
Duration |
In Waiting |
Machine | Teuthology Branch | OS Type | OS Version | Description | Nodes |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| pass | 7879946 |
|
2024-08-29 14:14:44 | 2024-08-29 14:39:45 | 2024-08-29 15:02:45 | 0:23:00 | 0:15:27 | 0:07:33 | smithi | main | centos | 9.stream | rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/few rados thrashers/pggrow thrashosds-health workloads/test_rbd_api} | 3 |
| fail | 7879948 |
|
2024-08-29 14:14:45 | 2024-08-29 14:40:36 | 2024-08-29 15:17:12 | 0:36:36 | 0:29:04 | 0:07:32 | smithi | main | centos | 9.stream | rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/dashboard} | 2 |
Failure Reason:
Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) |
||||||||||||||
| pass | 7879950 |
|
2024-08-29 14:14:47 | 2024-08-29 14:43:07 | 2024-08-29 15:15:26 | 0:32:19 | 0:24:45 | 0:07:34 | smithi | main | centos | 9.stream | rados/encoder/{0-start 1-tasks supported-random-distro$/{centos_latest}} | 1 |
| fail | 7879952 |
|
2024-08-29 14:14:48 | 2024-08-29 14:46:28 | 2024-08-29 15:25:18 | 0:38:50 | 0:32:57 | 0:05:53 | smithi | main | centos | 9.stream | rados/upgrade/parallel/{0-random-distro$/{centos_9.stream_runc} 0-start 1-tasks mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
Command failed (workunit test cls/test_cls_rbd.sh) on smithi031 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' |
||||||||||||||
| fail | 7879954 |
|
2024-08-29 14:14:49 | 2024-08-29 14:49:09 | 2024-08-29 15:08:34 | 0:19:25 | 0:11:43 | 0:07:42 | smithi | main | centos | 9.stream | rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/deploy-raw} | 2 |
Failure Reason:
reached maximum tries (120) after waiting for 120 seconds |
||||||||||||||
| pass | 7879956 |
|
2024-08-29 14:14:50 | 2024-08-29 14:49:40 | 2024-08-29 15:19:11 | 0:29:31 | 0:22:52 | 0:06:39 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-4 openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} | 4 |
| fail | 7879958 |
|
2024-08-29 14:14:52 | 2024-08-29 14:49:51 | 2024-08-29 15:33:16 | 0:43:25 | 0:28:29 | 0:14:56 | smithi | main | centos | 9.stream | rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/reef backoff/peering ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/few rados thrashers/careful thrashosds-health workloads/cache-snaps} | 3 |
Failure Reason:
"2024-08-29T15:20:00.000379+0000 mon.a (mon.0) 1364 : cluster [WRN] osd.1 (root=default,host=smithi017) is down" in cluster log |
||||||||||||||
| fail | 7879960 |
|
2024-08-29 14:14:53 | 2024-08-29 14:59:33 | 2024-08-29 15:23:38 | 0:24:05 | 0:15:54 | 0:08:11 | smithi | main | centos | 9.stream | rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_monitoring_stack_basic} | 3 |
Failure Reason:
Command failed on smithi064 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f4056d968930e12926c25a8f13ee8ed6047c791 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e0048ada-6618-11ef-bcd4-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\'' |
||||||||||||||
| pass | 7879962 |
|
2024-08-29 14:14:54 | 2024-08-29 15:01:14 | 2024-08-29 15:29:45 | 0:28:31 | 0:18:21 | 0:10:10 | smithi | main | centos | 9.stream | rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/read mon_election/classic msgr-failures/fastclose msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-lz4} rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/small-objects-localized} | 4 |
| dead | 7879964 |
|
2024-08-29 14:14:56 | 2024-08-29 15:08:46 | 2024-08-29 23:20:42 | 8:11:56 | smithi | main | ubuntu | 22.04 | rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_rgw_multisite} | 3 | ||
Failure Reason:
hit max job timeout |
||||||||||||||
| pass | 7879966 |
|
2024-08-29 14:14:57 | 2024-08-29 15:10:28 | 2024-08-29 15:45:30 | 0:35:02 | 0:24:21 | 0:10:41 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-overwrites/{bluestore-bitmap ceph clusters/{fixed-4 openstack} fast/normal mon_election/connectivity msgr-failures/osd-dispatch-delay rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{ubuntu_latest} thrashers/pggrow thrashosds-health workloads/ec-small-objects-overwrites} | 4 |
| fail | 7879968 |
|
2024-08-29 14:14:58 | 2024-08-29 15:11:50 | 2024-08-29 15:28:47 | 0:16:57 | 0:10:12 | 0:06:45 | smithi | main | centos | 9.stream | rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-bitmap} tasks/e2e} | 2 |
Failure Reason:
Command failed on smithi087 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2' |
||||||||||||||
| fail | 7879969 |
|
2024-08-29 14:15:00 | 2024-08-29 15:11:50 | 2024-08-29 17:09:48 | 1:57:58 | 1:50:21 | 0:07:37 | smithi | main | centos | 9.stream | rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/squid backoff/peering_and_degraded ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/osd-delay rados thrashers/default thrashosds-health workloads/radosbench} | 3 |
Failure Reason:
"2024-08-29T15:40:00.000136+0000 mon.a (mon.0) 1450 : cluster [WRN] pg 3.0 is stuck undersized for 82s, current state active+recovery_wait+undersized+degraded+remapped, last acting [11,9]" in cluster log |
||||||||||||||
| fail | 7879971 |
|
2024-08-29 14:15:01 | 2024-08-29 15:12:52 | 2024-08-29 15:26:25 | 0:13:33 | 0:06:50 | 0:06:43 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} | 3 |
Failure Reason:
Command failed on smithi033 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 16 erasure jerasure21profile' |
||||||||||||||
| fail | 7879973 |
|
2024-08-29 14:15:02 | 2024-08-29 15:13:46 | 2024-08-29 15:29:29 | 0:15:43 | 0:08:26 | 0:07:17 | smithi | main | centos | 9.stream | rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_ca_signed_key} | 2 |
Failure Reason:
"2024-08-29T15:27:28.717343+0000 mon.a (mon.0) 341 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log |
||||||||||||||
| pass | 7879974 |
|
2024-08-29 14:15:03 | 2024-08-29 15:14:16 | 2024-08-29 15:49:05 | 0:34:49 | 0:25:18 | 0:09:31 | smithi | main | centos | 9.stream | rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados tasks/mon_recovery validater/valgrind} | 2 |
| dead | 7879975 |
|
2024-08-29 14:15:05 | 2024-08-29 15:15:27 | 2024-08-29 15:23:15 | 0:07:48 | smithi | main | ubuntu | 22.04 | rados/cephadm/osds/{0-distro/ubuntu_22.04 0-nvme-loop 1-start 2-ops/deploy-raw} | 2 | ||
Failure Reason:
SSH connection to smithi179 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' |
||||||||||||||
| fail | 7879976 |
|
2024-08-29 14:15:06 | 2024-08-29 15:15:37 | 2024-08-29 15:34:14 | 0:18:37 | 0:09:01 | 0:09:36 | smithi | main | ubuntu | 22.04 | rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_cephadm_repos} | 1 |
Failure Reason:
Command failed (workunit test cephadm/test_repos.sh) on smithi077 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f4056d968930e12926c25a8f13ee8ed6047c791 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh' |
||||||||||||||
| fail | 7879977 |
|
2024-08-29 14:15:07 | 2024-08-29 15:16:58 | 2024-08-29 15:40:41 | 0:23:43 | 0:15:55 | 0:07:48 | smithi | main | centos | 9.stream | rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} | 3 |
Failure Reason:
Command failed on smithi191 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" |
||||||||||||||
| fail | 7879978 |
|
2024-08-29 14:15:09 | 2024-08-29 15:17:28 | 2024-08-29 16:08:10 | 0:50:42 | 0:42:14 | 0:08:28 | smithi | main | centos | 9.stream | rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} | 1 |
Failure Reason:
Command failed (workunit test misc/test-ceph-helpers.sh) on smithi060 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f4056d968930e12926c25a8f13ee8ed6047c791 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' |
||||||||||||||
| fail | 7879979 |
|
2024-08-29 14:15:10 | 2024-08-29 15:19:29 | 2024-08-29 15:56:08 | 0:36:39 | 0:29:28 | 0:07:11 | smithi | main | centos | 9.stream | rados/dashboard/{0-single-container-host debug/mgr mon_election/connectivity random-objectstore$/{bluestore-comp-zstd} tasks/dashboard} | 2 |
Failure Reason:
Test failure: test_list_enabled_module (tasks.mgr.dashboard.test_mgr_module.MgrModuleTest) |
||||||||||||||
| fail | 7879980 |
|
2024-08-29 14:15:11 | 2024-08-29 15:19:29 | 2024-08-29 16:19:15 | 0:59:46 | 0:50:04 | 0:09:42 | smithi | main | ubuntu | 22.04 | rados/upgrade/parallel/{0-random-distro$/{ubuntu_22.04} 0-start 1-tasks mon_election/connectivity upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}} | 2 |
Failure Reason:
Command failed (workunit test cls/test_cls_rbd.sh) on smithi019 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=reef TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cls/test_cls_rbd.sh' |
||||||||||||||
| fail | 7879981 |
|
2024-08-29 14:15:12 | 2024-08-29 15:19:49 | 2024-08-29 17:16:42 | 1:56:53 | 1:51:07 | 0:05:46 | smithi | main | centos | 9.stream | rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zstd} rados tasks/rados_cls_all validater/valgrind} | 2 |
Failure Reason:
reached maximum tries (50) after waiting for 300 seconds |
||||||||||||||
| fail | 7879982 |
|
2024-08-29 14:15:13 | 2024-08-29 15:19:50 | 2024-08-29 15:44:05 | 0:24:15 | 0:15:53 | 0:08:22 | smithi | main | centos | 9.stream | rados/cephadm/workunits/{0-distro/centos_9.stream agent/on mon_election/connectivity task/test_monitoring_stack_basic} | 3 |
Failure Reason:
Command failed on smithi029 with status 5: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f4056d968930e12926c25a8f13ee8ed6047c791 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7b724f4-661b-11ef-bcd4-c7b262605968 -- bash -c \'set -e\nset -x\nceph orch apply node-exporter\nceph orch apply grafana\nceph orch apply alertmanager\nceph orch apply prometheus\nsleep 240\nceph orch ls\nceph orch ps\nceph orch host ls\nMON_DAEMON=$(ceph orch ps --daemon-type mon -f json | jq -r \'"\'"\'last | .daemon_name\'"\'"\')\nGRAFANA_HOST=$(ceph orch ps --daemon-type grafana -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nPROM_HOST=$(ceph orch ps --daemon-type prometheus -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nALERTM_HOST=$(ceph orch ps --daemon-type alertmanager -f json | jq -e \'"\'"\'.[]\'"\'"\' | jq -r \'"\'"\'.hostname\'"\'"\')\nGRAFANA_IP=$(ceph orch host ls -f json | jq -r --arg GRAFANA_HOST "$GRAFANA_HOST" \'"\'"\'.[] | select(.hostname==$GRAFANA_HOST) | .addr\'"\'"\')\nPROM_IP=$(ceph orch host ls -f json | jq -r --arg PROM_HOST "$PROM_HOST" \'"\'"\'.[] | select(.hostname==$PROM_HOST) | .addr\'"\'"\')\nALERTM_IP=$(ceph orch host ls -f json | jq -r --arg ALERTM_HOST "$ALERTM_HOST" \'"\'"\'.[] | select(.hostname==$ALERTM_HOST) | .addr\'"\'"\')\n# check each host node-exporter metrics endpoint is responsive\nALL_HOST_IPS=$(ceph orch host ls -f json | jq -r \'"\'"\'.[] | .addr\'"\'"\')\nfor ip in $ALL_HOST_IPS; do\n curl -s http://${ip}:9100/metric\ndone\n# check grafana endpoints are responsive and database health is okay\ncurl -k -s https://${GRAFANA_IP}:3000/api/health\ncurl -k -s https://${GRAFANA_IP}:3000/api/health | jq -e \'"\'"\'.database == "ok"\'"\'"\'\n# stop mon daemon in order to trigger an alert\nceph orch daemon stop $MON_DAEMON\nsleep 120\n# check prometheus endpoints are responsive and mon down alert is firing\ncurl -s http://${PROM_IP}:9095/api/v1/status/config\ncurl -s http://${PROM_IP}:9095/api/v1/status/config | jq -e \'"\'"\'.status == "success"\'"\'"\'\ncurl -s http://${PROM_IP}:9095/api/v1/alerts\ncurl -s http://${PROM_IP}:9095/api/v1/alerts | jq -e \'"\'"\'.data | .alerts | .[] | select(.labels | .alertname == "CephMonDown") | .state == "firing"\'"\'"\'\n# check alertmanager endpoints are responsive and mon down alert is active\ncurl -s http://${ALERTM_IP}:9093/api/v1/status\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts\ncurl -s http://${ALERTM_IP}:9093/api/v1/alerts | jq -e \'"\'"\'.data | .[] | select(.labels | .alertname == "CephMonDown") | .status | .state == "active"\'"\'"\'\n\'' |
||||||||||||||
| fail | 7879983 |
|
2024-08-29 14:15:15 | 2024-08-29 15:20:40 | 2024-08-29 15:45:40 | 0:25:00 | 0:18:42 | 0:06:18 | smithi | main | centos | 9.stream | rados/standalone/{supported-random-distro$/{centos_latest} workloads/mon} | 1 |
Failure Reason:
Command failed (workunit test mon/mon-cluster-log.sh) on smithi112 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f4056d968930e12926c25a8f13ee8ed6047c791 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-cluster-log.sh' |
||||||||||||||
| dead | 7879984 |
|
2024-08-29 14:15:16 | 2024-08-29 15:20:41 | 2024-08-29 15:32:16 | 0:11:35 | 0:01:47 | 0:09:48 | smithi | main | centos | 9.stream | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/few objectstore/{bluestore-options/write$/{write_v1} bluestore/bluestore-hybrid} rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_latest} thrashers/fastread thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} | 3 |
Failure Reason:
['ubuntu@smithi179.front.sepia.ceph.com: Permission denied (publickey).'] |
||||||||||||||
| fail | 7879985 |
|
2024-08-29 14:15:17 | 2024-08-29 15:22:31 | 2024-08-29 15:40:40 | 0:18:09 | 0:10:44 | 0:07:25 | smithi | main | centos | 9.stream | rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_rgw_multisite} | 3 |
Failure Reason:
Command failed on smithi003 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f4056d968930e12926c25a8f13ee8ed6047c791 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 148e7056-661c-11ef-bcd4-c7b262605968 -- bash -c \'set -e\nset -x\nwhile true; do TOKEN=$(ceph rgw realm tokens | jq -r \'"\'"\'.[0].token\'"\'"\'); echo $TOKEN; if [ "$TOKEN" != "master zone has no endpoint" ]; then break; fi; sleep 5; done\nTOKENS=$(ceph rgw realm tokens)\necho $TOKENS | jq --exit-status \'"\'"\'.[0].realm == "myrealm1"\'"\'"\'\necho $TOKENS | jq --exit-status \'"\'"\'.[0].token\'"\'"\'\nTOKEN_JSON=$(ceph rgw realm tokens | jq -r \'"\'"\'.[0].token\'"\'"\' | base64 --decode)\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.realm_name == "myrealm1"\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.endpoint | test("http://.+:\\\\d+")\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.realm_id | test("^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.access_key\'"\'"\'\necho $TOKEN_JSON | jq --exit-status \'"\'"\'.secret\'"\'"\'\n\'' |
||||||||||||||
| pass | 7879986 |
|
2024-08-29 14:15:18 | 2024-08-29 15:23:22 | 2024-08-29 15:55:23 | 0:32:01 | 0:22:08 | 0:09:53 | smithi | main | ubuntu | 22.04 | rados/singleton/{all/thrash_cache_writeback_proxy_none mon_election/classic msgr-failures/none msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-zlib} rados supported-random-distro$/{ubuntu_latest}} | 2 |
| pass | 7879987 |
|
2024-08-29 14:15:20 | 2024-08-29 15:23:52 | 2024-08-29 15:47:35 | 0:23:43 | 0:14:26 | 0:09:17 | smithi | main | ubuntu | 22.04 | rados/perf/{ceph mon_election/connectivity objectstore/bluestore-low-osd-mem-target openstack scheduler/wpq_default_shards settings/optimized ubuntu_latest workloads/fio_4K_rand_rw} | 1 |
| fail | 7879988 |
|
2024-08-29 14:15:21 | 2024-08-29 15:23:53 | 2024-08-29 15:44:21 | 0:20:28 | 0:11:43 | 0:08:45 | smithi | main | centos | 9.stream | rados/cephadm/osds/{0-distro/centos_9.stream_runc 0-nvme-loop 1-start 2-ops/deploy-raw} | 2 |
Failure Reason:
reached maximum tries (120) after waiting for 120 seconds |
||||||||||||||
| pass | 7879989 |
|
2024-08-29 14:15:22 | 2024-08-29 15:25:33 | 2024-08-29 17:14:42 | 1:49:09 | 1:35:59 | 0:13:10 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code/{ceph clusters/{fixed-4 openstack} fast/fast mon_election/classic msgr-failures/osd-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados recovery-overrides/{more-active-recovery} supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/ec-radosbench} | 4 |
| fail | 7879990 |
|
2024-08-29 14:15:23 | 2024-08-29 15:28:54 | 2024-08-29 15:43:54 | 0:15:00 | 0:09:07 | 0:05:53 | smithi | main | centos | 9.stream | rados/dashboard/{0-single-container-host debug/mgr mon_election/classic random-objectstore$/{bluestore-comp-zlib} tasks/e2e} | 2 |
Failure Reason:
Command failed on smithi050 with status 1: 'yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2' |
||||||||||||||
| fail | 7879991 |
|
2024-08-29 14:15:24 | 2024-08-29 15:29:04 | 2024-08-29 16:17:55 | 0:48:51 | 0:42:08 | 0:06:43 | smithi | main | centos | 9.stream | rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} | 1 |
Failure Reason:
Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi087 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f4056d968930e12926c25a8f13ee8ed6047c791 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' |
||||||||||||||
| pass | 7879992 |
|
2024-08-29 14:15:26 | 2024-08-29 15:29:05 | 2024-08-29 16:03:09 | 0:34:04 | 0:26:17 | 0:07:47 | smithi | main | centos | 9.stream | rados/thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/3-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/on mon_election/connectivity msgr-failures/fastclose rados thrashers/pggrow thrashosds-health workloads/cache-snaps} | 3 |
| fail | 7879993 |
|
2024-08-29 14:15:27 | 2024-08-29 15:29:55 | 2024-08-29 17:59:36 | 2:29:41 | 2:20:43 | 0:08:58 | smithi | main | ubuntu | 22.04 | rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} | 1 |
Failure Reason:
Command failed (workunit test scrub/osd-scrub-test.sh) on smithi193 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f4056d968930e12926c25a8f13ee8ed6047c791 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh' |
||||||||||||||
| dead | 7879994 |
|
2024-08-29 14:15:28 | 2024-08-29 15:29:56 | 2024-08-29 15:40:00 | 0:10:04 | smithi | main | ubuntu | 22.04 | rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/connectivity msgr-failures/osd-dispatch-delay objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-bitmap} rados recovery-overrides/{more-async-recovery} supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/ec-rados-plugin=jerasure-k=4-m=2} | 3 | ||
Failure Reason:
SSH connection to smithi179 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic' |
||||||||||||||
| pass | 7879995 |
|
2024-08-29 14:15:29 | 2024-08-29 15:32:26 | 2024-08-29 16:01:27 | 0:29:01 | 0:23:05 | 0:05:56 | smithi | main | centos | 9.stream | rados/singleton-nomsgr/{all/recovery-unfound-found mon_election/classic rados supported-random-distro$/{centos_latest}} | 1 |
Pulpito