Pulpito :: Results Dashboard

Posted	Started	Updated	Runtime	Duration	In Waiting	Machine	Teuthology Branch	OS Type	OS Version	Nodes	Status
2020-08-09 03:56:54	2020-08-09 04:10:48	2020-08-09 05:00:49	0:50:01	0:07:45	0:42:16	smithi	master	rhel	8.1	6	fail

Nodes: smithi066 smithi119 smithi065 smithi074 smithi204 smithi166

Description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2020-08-09_03:56:14-kcephfs-nautilus-distro-basic-smithi/5323461/teuthology.log

Sentry event: http://sentry.ceph.com/sepia/teuthology/?q=20e64b9394ce4c668393942dae176dc6

Failure Reason:

Command failed on smithi166 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel'

All Details...

log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2020-08-09_03:56:14-kcephfs-nautilus-distro-basic-smithi/5323461/teuthology.log
archive_path: /home/teuthworker/archive/yuriw-2020-08-09_03:56:14-kcephfs-nautilus-distro-basic-smithi/5323461
description: kcephfs/recovery/{begin clusters/1-mds-4-client conf/{client mds mon osd} kclient/{mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore-ec/bluestore-ec-root overrides/{frag_enable log-config osd-asserts whitelist_health whitelist_wrongly_marked_down} tasks/auto-repair}
duration: 0:07:45
email: ceph-qa@ceph.io
failure_reason: Command failed on smithi166 with status 1: 'sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel'
flavor: basic
job_id: 5323461
kernel:
- client:
  - sha1: distro
- kdb: True
- sha1: distro
last_in_suite: False
machine_type: smithi
name: yuriw-2020-08-09_03:56:14-kcephfs-nautilus-distro-basic-smithi
nuke_on_error: True
os_type: rhel
os_version: 8.1
overrides:
- ceph-deploy:
  - fs: xfs
  - conf:
    - client:
      - log file: /var/log/ceph/ceph-$name.$pid.log
    - mon:
      - osd default pool size: 2
    - osd:
      - mon osd full ratio: 0.9
      - mon osd backfillfull_ratio: 0.85
      - bluestore fsck on mount: True
      - mon osd nearfull ratio: 0.8
      - debug bluestore: 20
      - debug bluefs: 20
      - osd objectstore: bluestore
      - bluestore block size: 96636764160
      - debug rocksdb: 10
      - osd failsafe full ratio: 0.95
  - bluestore: True
- selinux:
  - whitelist:
    - scontext=system_u:system_r:logrotate_t:s0
- workunit:
  - sha1: ab2488773626d40fb1f8345a99f4bbd6712c0fd4
  - branch: master
- ceph:
  - log-whitelist:
    - $MDS_ALL_DOWN$
    - $MDS_UP_LESS_THAN_MAX$
    - overall HEALTH_
    - $FS_DEGRADED$
    - $MDS_FAILED$
    - $MDS_DEGRADED$
    - $FS_WITH_FAILED_MDS$
    - $MDS_DAMAGE$
    - $MDS_ALL_DOWN$
    - $MDS_UP_LESS_THAN_MAX$
    - $FS_INLINE_DATA_DEPRECATED$
    - overall HEALTH_
    - $OSD_DOWN$
    - $OSD_
    - but it is still running
    - is not responding
    - force file system read-only
    - bad backtrace
    - MDS in read-only mode
    - \(MDS_READ_ONLY$
  - sha1: aa50a1c6d3757cd0a2e1be2cd61a923ccd14c244
  - fs: xfs
  - conf:
    - global:
      - ms die on skipped message: False
    - mgr:
      - debug ms: 1
      - debug mgr: 20
    - client:
      - debug ms: 1
      - debug client: 20
      - client mount timeout: 600
    - mon:
      - debug paxos: 20
      - debug mon: 20
      - debug ms: 1
      - mon op complaint time: 120
    - mds:
      - mds bal split bits: 3
      - mds bal split size: 100
      - osd op complaint time: 180
      - debug mds: 20
      - mds bal merge size: 5
      - debug ms: 1
      - mds bal frag: True
      - mds verify scatter: True
      - mds bal fragment size max: 10000
      - mds op complaint time: 180
      - mds debug scatterstat: True
      - mds debug frag: True
    - osd:
      - mon osd full ratio: 0.9
      - debug ms: 20
      - debug journal: 20
      - debug osd: 25
      - debug bluestore: 20
      - debug bluefs: 20
      - osd objectstore: bluestore
      - mon osd backfillfull_ratio: 0.85
      - osd op complaint time: 180
      - bluestore block size: 96636764160
      - osd shutdown pgref assert: True
      - debug filestore: 20
      - debug rocksdb: 10
      - mon osd nearfull ratio: 0.8
      - osd failsafe full ratio: 0.95
      - bluestore fsck on mount: True
  - cephfs_ec_profile:
    - m=2
    - k=2
    - crush-failure-domain=osd
- install:
  - ceph:
    - sha1: aa50a1c6d3757cd0a2e1be2cd61a923ccd14c244
- admin_socket:
  - branch: nautilus
- thrashosds:
  - bdev_inject_crash_probability: 0.5
  - bdev_inject_crash: 2
owner: scheduled_yuriw@teuthology
pid:
roles:
- ['mon.a', 'mgr.y', 'mds.a', 'mds.b', 'osd.0', 'osd.1', 'osd.2', 'osd.3']
- ['mon.b', 'mon.c', 'mgr.x', 'mds.c', 'osd.4', 'osd.5', 'osd.6', 'osd.7']
- ['client.0']
- ['client.1']
- ['client.2']
- ['client.3']
sentry_event: http://sentry.ceph.com/sepia/teuthology/?q=20e64b9394ce4c668393942dae176dc6
status: fail
success: False
branch: nautilus
seed:
sha1: aa50a1c6d3757cd0a2e1be2cd61a923ccd14c244
subset:
suite:
suite_branch: master
suite_path:
suite_relpath:
suite_repo:
suite_sha1: ab2488773626d40fb1f8345a99f4bbd6712c0fd4
targets:
- smithi066.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEMBxHzyGwLwSMO4lvx+VvPY8SGtCIeyub7fm3MfuYN4zsNCin4PxQYClwedBNKq/uKsktmEktjuLRM0O3JpMvQ+AmoFxKcY5fPIk0wA4CsfMHXbV8VMo2a+DSgx44433ef8NmgjxOB5LKst33q5dxNj+UYw4O8idvBMkP+3ECdESf9B0CtAFAtDVjM+SokRjPR8eJ4kwZnDR/jGUUKp7SYGaX8mU+RUdYmRc7XknOpf7ks7AgUFZvhN4jg1CCaWgkwOuy4z4PC1TTdUU+hmBVt3efZiTVVSLRUmZAfV+JBOvov0oox0JEaVBUVNSDX0b/eo2pF0pDfxTPpDL9czANAHH9onDDT72hwxPsC/Zw4w4JAEO1lDiWz3zIZbF9tKzUcF403ZbV+eJ1s1i53/56lTH43r0JF9bx9Ds05HYUYzMs7siOLv5lQU5RokJyBOaU7eK5Pc+UC3eBn1tba09YTe8RFhsPdK4KHbVBCenxBKy8w8BkQ8mIQ7kI0W5fPd0=
- smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyEy93coLTU9ttNih5GIaVG0kLHqCDqCzGNKZTUfInqG4l5Rj+tBLJEd/59kZ+ny2+DGIVPGCb0mHI14+5LkclLZRWZRk43LNhfI7TC9IKleoxUIyaUm4KgRLzwRpvyhR7t8BQDf8fMuvFItlFEWqcffMaLZvhWjKfCaRHvMvYYXs0XJ2GDC8lm4iXHbtoHSQ4lUvOfKjRfQ/5gCphhb9MHSe3+R5q7zOLB4W79XJh16iV8L4DxxoAeml1O+uCdrMQrMPAIDmXdK3OAZ6tdHnr7NizsBwVCMdcri4gCu9Ul5iYYZTve+VwpvG/8U+aQ2eS2bkjEHt1KEdAAwqMeCOG8RJbVLJiRHsQef6QBdD8u1O2td7o6NhbP3WZc/EwcACBgb8zZ5MycIbxh5WOhHijTceMLzrxP9oqLJbdcEiTFItLk7hrEjI4z5MrsXw5NPsY1gSk+Ot0kXwI9dGNzUtMYjd5JurfrCxZmcoGpKRk/kHXfQ0EOmXoE5Imn2Yx3LM=
- smithi065.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuwFVwoR2JTIAy2bEOZjdXvL1+SmRYDU5dRCNnhYbDUsgsQKgfv9mYMnATxExob2cBE8jMN23/pEPyxOyD/9pfRXxwXEBGlihgm0X3WmuK9UafXgH4tLrmL/Wf0BUQsx29QPbUtZhLk+1q7leFi/Q1PfML3vckUdiUCjXYHUemP2v/9VXKhMti3rnlIZuh4D+SOVnsJyTZxoBWlVQnA0sdCETkl2fCmqoTXxlDMd1ggJTjUhnQWRw5Tcf0b6j/ED3aV8mbjBQiuZL0VNVOFbGc4cMJ9ibU/CZMWyytr9nWAn1vrz86c/RiOWA7l8XkJwbAkVozwTCoiLpPa0T70Ck9jGPDby8aifK5gQEXuQkSCr32Feq4lM4A3HcONzmWUZ5BwzpfwKzRQ3dZm0CXDr1Oo9T7v41jNevcIAN28A+/fDwehcMp7ukn+sF5V8f2L8b5KKUKAQ0rPWyvfkoIIr94ierWQGM+vgQButWRdpoyhpiktRZpKT1oCyNZk8/y1l8=
- smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbVe0Q++yStVK/wPg4r+yOj4tLoZfktRZ6bPfNj8xDxB8F5HGr5jG8CFA3gW8WF/ve0WsRKHaVPR0eV2GqNVCVcw+1lHwA9VI/egSGrb4PBvgmATQq3NNdYtFJkOdmtlc0kfEoYTCa9iX95T6lIgtCbZHdJJ+S9S0qAdlxIvCghTqOKtejuPFH8dWswkVJaRDouz1EMGb4gb7QKWCSiD+bKKp9ba5Gv4AA4iMM09F+DFPBXnO+CWjN+TIxAlAnHGyVj5ruBYTEJ/N/10cqPcXjFJrU1OC+S/+jTOSXII5E8uJQEjoElulCWNVBaKGMogWF+qhhNjlqZWiONVlnKVjCIfbxW6oYsT+RGrQ6cQ9Hl+Pughld2WQZnkI45vwjoPKH4RQiqlFsTKWoJeR7IkrmaIMif/udAvfAY/C1BEHze9Yig2dUeh4sSiItTPV/JiUSmjoIRDdzcjHpBBNbzHlXx4I2f0Qe8iSaDuabRO0bCfl1kHnc01i3N90VDklMDWM=
- smithi204.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoeRqXD0QgFDN67G01W3g8pZ92a34u+cH041laRWoZO1+C3tuhlnczSVl1/dkLwgN0y2q8vXAc6+HiWZY9mUOGIjgPrpXCTdrzs1E2pDOzEua7pc7pPpv+jAjINwQ65bfytP2kn6hkQeee9E/ealr04S7LHGpHR8ylQ3ZeX7+nLt+qHQOspfjVfqhCu/vv+UWWAzTOVeLTBGySYAh1atDt0sXJpmUbr1kqFHIRE+eBs6bosSI9Sp7C0koiGFdwYa193qT+TDiz0v5+mPT+GOsdLnqYOWUk+X8zAJV7sbXqne9rxbq0XHwO+A7vvNlSbAKLl1GiqN1NEkDTy5jgp2VfLATN+XbkH6LYgdtB1PBEgX42yMS3oFTa2JtgyIgUtw26sAMxfDhpYU/ZdOKNTMWQRR+HWfgHyTr6OcOQmvyVrQbmhmKGxfFCtwZQd7AobLomi6HZSznKG4umrvh0t4R283lDivXUjev3zqxQYwxCqnqvciS0z88KxAuk6deq88=
- smithi166.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCuHLhKPAoMWMj37S75rnvDmPakfadISUmTbPl+0Qx2HnOGaBYltH7B813ZsWoGm5k6i4Ch6fQYqM9tgFaAFwkb4ak8Vm+17xvcdaN7YOeOiW8Xv/VADFbh7QenwkJCZdni49HFP12DQ3heiD8xRiCjEeprKMXl5o+ouhRKdep8/ZF/oYL/WnUTx8E9Vq4cM9ZIybLevVX3Eg6jJCiKzEhdHP58NkxaZnWxGFdxMTHtWn2uDYOzCQFiGH61PR8UPFD2aOZV8Mc1lb2zxgkbXFzvYsDmtH2JxqPnlujb7gxFCF44tJ8KHXtA1y2TvY5xWykszxrdLPZiEW9MqI+m7HAhbRRdMy4KzRFlPROKnkGZZiHt4cWpAyV6MSb8wl0R/XXSuUd2KtQvWsIkufEPhABFJpedbPQhSxGLRzM42w4VvjrNlyq0FiqxORmgrCa5P90DYS9G2XI62cYYKh2pM1UrCxw2i/1I3D7Gjdzl87T91plouD/+h1zo1thRkMViTTU=
tasks:
- internal.check_packages:
- internal.buildpackages_prep:
- internal.lock_machines:
  - 6
  - smithi
- internal.save_config:
- internal.check_lock:
- internal.add_remotes:
- console_log:
- internal.connect:
- internal.push_inventory:
- internal.serialize_remote_roles:
- internal.check_conflict:
- internal.check_ceph_data:
- internal.vm_setup:
- kernel:
  - client:
    - sha1: distro
  - kdb: True
  - sha1: distro
- internal.base:
- internal.archive_upload:
- internal.archive:
- internal.coredump:
- internal.sudo:
- internal.syslog:
- internal.timer:
- pcp:
- selinux:
- ansible.cephlab:
- clock:
- install:
  - extra_system_packages:
    - deb:
      - bison
      - flex
      - libelf-dev
      - libssl-dev
      - dump
      - indent
    - rpm:
      - bison
      - flex
      - elfutils-libelf-devel
      - openssl-devel
      - libacl-devel
      - libaio-devel
      - libattr-devel
      - libtool
      - libuuid-devel
      - xfsdump
      - xfsprogs
      - xfsprogs-devel
  - extra_packages:
    - deb:
      - python3-cephfs
      - cephfs-shell
    - rpm:
      - python3-cephfs
  - sha1: aa50a1c6d3757cd0a2e1be2cd61a923ccd14c244
- ceph:
- kclient:
- cephfs_test_runner:
  - modules:
    - tasks.cephfs.test_auto_repair
teuthology_branch: master
verbose: True
pcp_grafana_url:
priority:
user:
queue:
posted: 2020-08-09 03:56:54
started: 2020-08-09 04:10:48
updated: 2020-08-09 05:00:49
status_class: danger
runtime: 0:50:01
wait_time: 0:42:16