Name Machine Type Up Locked Locked Since Locked By OS Type OS Version Arch Description
smithi045.front.sepia.ceph.com smithi True True 2020-08-07 16:22:35.940659 scheduled_teuthology@teuthology ubuntu 18.04 x86_64 /home/teuthworker/archive/teuthology-2020-08-07_03:30:02-rados-octopus-distro-basic-smithi/5294578
Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version Description Nodes
dead 5311705 2020-08-07 16:03:38 2020-08-07 16:03:52 2020-08-07 16:23:52 0:20:00 smithi master centos 8.1 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/fastclose msgr/async objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/small-objects-localized} 2
dead 5306554 2020-08-07 14:21:55 2020-08-07 14:33:04 2020-08-07 15:15:05 0:42:01 smithi master centos 8.1 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/fastclose msgr/async objectstore/bluestore-stupid rados supported-random-distro$/{centos_8} thrashers/mapgap thrashosds-health workloads/small-objects-localized} 2
dead 5305285 2020-08-07 13:56:22 2020-08-07 13:57:02 2020-08-07 14:13:01 0:15:59 smithi master centos 8.1 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/fastclose msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_8} thrashers/none thrashosds-health workloads/cache-pool-snaps-readproxy} 2
fail 5300898 2020-08-07 12:30:30 2020-08-07 12:31:01 2020-08-07 12:49:00 0:17:59 0:10:31 0:07:28 smithi master centos 8.1 rados:thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} d-balancer/on msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/morepggrow thrashosds-health workloads/cache-snaps-balanced} 2
Failure Reason:

Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

dead 5300553 2020-08-07 12:24:07 2020-08-07 12:24:25 2020-08-07 12:32:24 0:07:59 smithi master centos 8.1 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-hybrid rados supported-random-distro$/{centos_8} thrashers/default thrashosds-health workloads/set-chunk-promote-flush} 2
fail 5300259 2020-08-07 12:18:53 2020-08-07 12:18:59 2020-08-07 12:24:58 0:05:59 0:00:16 0:05:43 smithi master ubuntu 18.04 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v2only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/snaps-few-objects-localized} 2
Failure Reason:

psutil.NoSuchProcess process no longer exists (pid=19140)

dead 5299547 2020-08-07 12:05:01 2020-08-07 12:06:26 2020-08-07 12:16:26 0:10:00 smithi master rhel 8.1 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-partial-recovery} backoff/peering ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/rados_api_tests} 2
dead 5298392 2020-08-07 11:41:54 2020-08-07 11:53:07 2020-08-07 12:01:06 0:07:59 smithi master rhel 8.1 rados:thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} d-balancer/crush-compat msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{rhel_8} thrashers/mapgap thrashosds-health workloads/dedup_tier} 2
pass 5294986 2020-08-07 04:11:24 2020-08-07 14:11:23 2020-08-07 14:49:24 0:38:01 0:31:12 0:06:49 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-bitmap overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/client-recovery} 2
running 5294578 2020-08-07 03:31:23 2020-08-07 15:21:15 2020-08-07 21:19:24 5:59:30 smithi master ubuntu 18.04 rados/upgrade/mimic-x-singleton/{0-cluster/{openstack start} 1-install/mimic 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-octopus 8-workload/{rbd-python snaps-many-objects} bluestore-bitmap thrashosds-health ubuntu_latest} 4
pass 5294391 2020-08-07 02:18:45 2020-08-07 03:32:22 2020-08-07 07:02:28 3:30:06 0:11:59 3:18:07 smithi master ubuntu 18.04 fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/bluestore-bitmap overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/ior-shared-file} 5
dead 5293611 2020-08-06 20:19:36 2020-08-06 20:38:26 2020-08-06 20:58:25 0:19:59 smithi master ubuntu 18.04 fs/basic_functional/{begin clusters/1-mds-4-client-coloc conf/{client mds mon osd} mount/fuse objectstore/bluestore-ec-root overrides/{frag_enable no_client_pidfile whitelist_health whitelist_wrongly_marked_down} supported-random-distros$/{ubuntu_latest} tasks/cephfs-shell} 2
pass 5293211 2020-08-06 19:32:57 2020-08-07 07:35:25 2020-08-07 08:15:25 0:40:00 0:29:43 0:10:17 smithi master centos 8.1 rados/thrash-erasure-code-isa/{arch/x86_64 ceph clusters/{fixed-2 openstack} msgr-failures/few objectstore/bluestore-comp-lz4 rados recovery-overrides/{more-async-partial-recovery} supported-random-distro$/{centos_8} thrashers/careful thrashosds-health workloads/ec-rados-plugin=isa-k=2-m=1} 2
pass 5293156 2020-08-06 19:32:04 2020-08-07 06:56:59 2020-08-07 07:38:59 0:42:00 0:31:07 0:10:53 smithi master rhel 8.1 rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/osd-delay objectstore/filestore-xfs rados recovery-overrides/{more-partial-recovery} supported-random-distro$/{rhel_8} thrashers/pggrow thrashosds-health workloads/ec-rados-plugin=jerasure-k=2-m=1} 2
pass 5293026 2020-08-06 19:30:00 2020-08-06 20:20:04 2020-08-06 20:50:03 0:29:59 0:21:06 0:08:53 smithi master centos 8.0 rados/cephadm/with-work/{distro/centos_8.0 fixed-2 mode/packaged msgr/async-v1only start tasks/rados_python} 2
pass 5292956 2020-08-06 19:28:56 2020-08-06 19:44:38 2020-08-06 20:22:38 0:38:00 0:31:47 0:06:13 smithi master centos 8.0 rados/cephadm/smoke-roleless/{distro/centos_8.0 start} 2
dead 5292928 2020-08-06 18:39:55 2020-08-06 18:43:37 2020-08-07 06:48:13 12:04:36 smithi master ubuntu 18.04 upgrade:nautilus-x/stress-split-erasure-code/{0-cluster/{openstack start} 1-nautilus-install/nautilus 1.1-pg-log-overrides/normal_pg_log 2-partial-upgrade/firsthalf 3-thrash/default 3.1-objectstore/filestore-xfs 4-ec-workload/{rados-ec-workload rbd-ec-workload} 5-finish-upgrade 6-octopus 7-final-workload thrashosds-health ubuntu_18.04} 5
pass 5292821 2020-08-06 15:37:00 2020-08-06 16:37:17 2020-08-06 18:27:20 1:50:03 1:39:13 0:10:50 smithi master ubuntu 18.04 rbd/immutable-object-cache/{clusters/{fix-2 openstack} pool/ceph_and_immutable_object_cache supported-random-distro$/{ubuntu_latest} workloads/qemu_on_immutable_object_cache_and_thrash} 2
pass 5292804 2020-08-06 15:36:44 2020-08-06 16:24:14 2020-08-06 16:42:13 0:17:59 0:13:01 0:04:58 smithi master rhel 8.1 rbd/cli/{base/install clusters/{fixed-1 openstack} features/layering msgr-failures/few objectstore/filestore-xfs pool/none supported-random-distro$/{rhel_8} workloads/rbd_cli_import_export} 1
fail 5292772 2020-08-06 15:36:13 2020-08-06 15:55:26 2020-08-06 16:25:26 0:30:00 0:23:06 0:06:54 smithi master centos 8.1 rbd/cli/{base/install clusters/{fixed-1 openstack} features/defaults msgr-failures/few objectstore/bluestore-low-osd-mem-target pool/small-cache-pool supported-random-distro$/{centos_8} workloads/rbd_cli_generic} 1
Failure Reason:

"2020-08-06T16:12:20.134105+0000 osd.1 (osd.1) 3 : cluster [WRN] slow request osd_op(client.12010.0:3276 3.0 3:2c9904c3:test1::rbd_data.2ee61e930d83.0000000000000045:head [set-alloc-hint object_size 4194304 write_size 4194304,write 86016~4096 [fadvise_random] in=4096b] snapc 0=[] RETRY=24 ondisk+retry+write+known_if_redirected e115) initiated 2020-08-06T16:11:49.881436+0000 currently reached pg" in cluster log