Pulpito :: Results Dashboard

Posted	Started	Updated	Runtime	Duration	In Waiting	Machine	Teuthology Branch	OS Type	OS Version	Nodes	Status
2022-10-19 18:37:21	2022-10-19 23:48:15	2022-10-20 00:11:30	0:23:15	0:13:41	0:09:34	smithi	main	ubuntu	20.04	2	fail

Nodes: smithi033 smithi045

Description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_18:35:19-rados-wip-yuri10-testing-2022-10-19-0810-distro-default-smithi/7074754/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2d35fa9e086c4ab9a9d8be6c3c1e0298

Failure Reason:

{'smithi033.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHIYJZRAzmG893Ll6BPPO3aM2JRw6fqQti'], 'labels': [], 'masters': [], 'uuids': ['3169c7b9-07fd-4857-bac5-d594d388ce07']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.478320', 'end': '2022-10-20 00:06:39.572678', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHIYJZRAzmG893Ll6BPPO3aM2JRw6fqQti'], 'labels': [], 'masters': [], 'uuids': ['3169c7b9-07fd-4857-bac5-d594d388ce07']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-10-20 00:06:38.094358', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-l03ImE-Q6rq-5kiQ-eH5g-cjUJ-51NM-vdfIdl', 'nvme-INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN', 'nvme-nvme.8086-43564654353333303030385134303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1', 'delta': '0:00:01.066519', 'end': '2022-10-20 00:06:41.094909', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-l03ImE-Q6rq-5kiQ-eH5g-cjUJ-51NM-vdfIdl', 'nvme-INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN', 'nvme-nvme.8086-43564654353333303030385134303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-10-20 00:06:40.028390', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHv9FI1UL5pr7osacSvgfamxl9mq1LRiuL'], 'labels': [], 'masters': [], 'uuids': ['399e9bf3-cace-48a0-af4e-8697849ae37c']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4', 'delta': '0:00:01.115901', 'end': '2022-10-20 00:06:42.840513', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHv9FI1UL5pr7osacSvgfamxl9mq1LRiuL'], 'labels': [], 'masters': [], 'uuids': ['399e9bf3-cace-48a0-af4e-8697849ae37c']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-10-20 00:06:41.724612', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOH30V0ldaNhJSTZGbdD32GDbf6GDGYXFS6'], 'labels': [], 'masters': [], 'uuids': ['837ae0b0-74ef-4757-9096-4b80f1cd5058']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOH30V0ldaNhJSTZGbdD32GDbf6GDGYXFS6'], 'labels': [], 'masters': [], 'uuids': ['837ae0b0-74ef-4757-9096-4b80f1cd5058']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi033.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHfC3MzcMOeINAIODCiFYSJogeIlUuCvhZ'], 'labels': [], 'masters': [], 'uuids': ['bb714dbb-2505-4c7a-9b9e-0d6b5e661a5b']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHfC3MzcMOeINAIODCiFYSJogeIlUuCvhZ'], 'labels': [], 'masters': [], 'uuids': ['bb714dbb-2505-4c7a-9b9e-0d6b5e661a5b']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi033.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-35000c500871a8992', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'wwn-0x5000c500871a8992'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-35000c500871a8992-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'wwn-0x5000c500871a8992-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W4HQ2L', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500871a8992'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-35000c500871a8992', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'wwn-0x5000c500871a8992'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-35000c500871a8992-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'wwn-0x5000c500871a8992-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W4HQ2L', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500871a8992'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHkDF6UPQWTAdu2vDxJrfAa4hzTibsXTmM'], 'labels': [], 'masters': [], 'uuids': ['a1081113-b3ce-4784-a9e9-b4067f9e5675']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHkDF6UPQWTAdu2vDxJrfAa4hzTibsXTmM'], 'labels': [], 'masters': [], 'uuids': ['a1081113-b3ce-4784-a9e9-b4067f9e5675']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi033.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}]}}

All Details...

log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2022-10-19_18:35:19-rados-wip-yuri10-testing-2022-10-19-0810-distro-default-smithi/7074754/teuthology.log
archive_path: /home/teuthworker/archive/yuriw-2022-10-19_18:35:19-rados-wip-yuri10-testing-2022-10-19-0810-distro-default-smithi/7074754
description: rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-3} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/on mon_election/connectivity msgr-failures/osd-delay msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/redirect_promote_tests}
duration: 0:13:41
email: ceph-qa@ceph.io
failure_reason: {'smithi033.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHIYJZRAzmG893Ll6BPPO3aM2JRw6fqQti'], 'labels': [], 'masters': [], 'uuids': ['3169c7b9-07fd-4857-bac5-d594d388ce07']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.478320', 'end': '2022-10-20 00:06:39.572678', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHIYJZRAzmG893Ll6BPPO3aM2JRw6fqQti'], 'labels': [], 'masters': [], 'uuids': ['3169c7b9-07fd-4857-bac5-d594d388ce07']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-10-20 00:06:38.094358', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-l03ImE-Q6rq-5kiQ-eH5g-cjUJ-51NM-vdfIdl', 'nvme-INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN', 'nvme-nvme.8086-43564654353333303030385134303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1', 'delta': '0:00:01.066519', 'end': '2022-10-20 00:06:41.094909', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-l03ImE-Q6rq-5kiQ-eH5g-cjUJ-51NM-vdfIdl', 'nvme-INTEL_SSDPEDMD400G4_CVFT5330008Q400BGN', 'nvme-nvme.8086-43564654353333303030385134303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-10-20 00:06:40.028390', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHv9FI1UL5pr7osacSvgfamxl9mq1LRiuL'], 'labels': [], 'masters': [], 'uuids': ['399e9bf3-cace-48a0-af4e-8697849ae37c']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4', 'delta': '0:00:01.115901', 'end': '2022-10-20 00:06:42.840513', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHv9FI1UL5pr7osacSvgfamxl9mq1LRiuL'], 'labels': [], 'masters': [], 'uuids': ['399e9bf3-cace-48a0-af4e-8697849ae37c']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-10-20 00:06:41.724612', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOH30V0ldaNhJSTZGbdD32GDbf6GDGYXFS6'], 'labels': [], 'masters': [], 'uuids': ['837ae0b0-74ef-4757-9096-4b80f1cd5058']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOH30V0ldaNhJSTZGbdD32GDbf6GDGYXFS6'], 'labels': [], 'masters': [], 'uuids': ['837ae0b0-74ef-4757-9096-4b80f1cd5058']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi033.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHfC3MzcMOeINAIODCiFYSJogeIlUuCvhZ'], 'labels': [], 'masters': [], 'uuids': ['bb714dbb-2505-4c7a-9b9e-0d6b5e661a5b']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHfC3MzcMOeINAIODCiFYSJogeIlUuCvhZ'], 'labels': [], 'masters': [], 'uuids': ['bb714dbb-2505-4c7a-9b9e-0d6b5e661a5b']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi033.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-35000c500871a8992', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'wwn-0x5000c500871a8992'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-35000c500871a8992-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'wwn-0x5000c500871a8992-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W4HQ2L', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500871a8992'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L', 'scsi-35000c500871a8992', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L', 'wwn-0x5000c500871a8992'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W4HQ2L-part1', 'scsi-35000c500871a8992-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W4HQ2L-part1', 'wwn-0x5000c500871a8992-part1'], 'labels': [], 'masters': [], 'uuids': ['78886b72-f581-4b1d-8578-0d8161fd4ca1']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '78886b72-f581-4b1d-8578-0d8161fd4ca1'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W4HQ2L', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500871a8992'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHkDF6UPQWTAdu2vDxJrfAa4hzTibsXTmM'], 'labels': [], 'masters': [], 'uuids': ['a1081113-b3ce-4784-a9e9-b4067f9e5675']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-YlN7Djx6ZToQOg6KOe0bOoeiPdIcEoOHkDF6UPQWTAdu2vDxJrfAa4hzTibsXTmM'], 'labels': [], 'masters': [], 'uuids': ['a1081113-b3ce-4784-a9e9-b4067f9e5675']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi033.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}]}}
flavor:
job_id: 7074754
kernel:
- kdb: True
- sha1: distro
last_in_suite: False
machine_type: smithi
name: yuriw-2022-10-19_18:35:19-rados-wip-yuri10-testing-2022-10-19-0810-distro-default-smithi
nuke_on_error: True
os_type: ubuntu
os_version: 20.04
overrides:
- admin_socket:
  - branch: wip-yuri10-testing-2022-10-19-0810
- ceph:
  - conf:
    - global:
      - mon client directed command retry: 5
      - mon election default strategy: 3
      - ms bind msgr1: False
      - ms bind msgr2: True
      - ms inject delay max: 1
      - ms inject delay probability: 0.005
      - ms inject delay type: osd
      - ms inject internal delays: 0.002
      - ms inject socket failures: 2500
      - ms type: async
      - osd_max_pg_log_entries: 2
      - osd_min_pg_log_entries: 1
      - osd_object_clean_region_max_num_intervals: 1000
      - osd_pg_log_trim_min: 0
      - osd_pool_default_min_size: 2
      - osd_pool_default_size: 3
    - mgr:
      - debug mgr: 20
      - debug ms: 1
    - mon:
      - debug mon: 20
      - debug ms: 1
      - debug paxos: 20
      - mon min osdmap epochs: 50
      - mon osdmap full prune interval: 2
      - mon osdmap full prune min: 15
      - mon osdmap full prune txsize: 2
      - mon scrub interval: 300
      - paxos service trim min: 10
    - osd:
      - bluestore block size: 96636764160
      - bluestore compression algorithm: zstd
      - bluestore compression mode: aggressive
      - bluestore fsck on mount: True
      - bluestore zero block detection: True
      - debug bluefs: 20
      - debug bluestore: 20
      - debug ms: 1
      - debug osd: 20
      - debug rocksdb: 10
      - mon osd backfillfull_ratio: 0.85
      - mon osd full ratio: 0.9
      - mon osd nearfull ratio: 0.8
      - osd backoff on peering: True
      - osd blocked scrub grace period: 3600
      - osd debug reject backfill probability: 0.3
      - osd debug verify cached snaps: True
      - osd debug verify missing on start: True
      - osd delete sleep: 1
      - osd failsafe full ratio: 0.95
      - osd max backfills: 3
      - osd max markdown count: 1000
      - osd max scrubs: 3
      - osd objectstore: bluestore
      - osd op queue: debug_random
      - osd op queue cut off: debug_random
      - osd scrub max interval: 120
      - osd scrub min interval: 60
      - osd shutdown pgref assert: True
      - osd snap trim sleep: 2
  - flavor: default
  - fs: xfs
  - log-ignorelist:
    - $MDS_ALL_DOWN$
    - $MDS_UP_LESS_THAN_MAX$
    - $OSD_SLOW_PING_TIME
    - but it is still running
    - objects unfound and apparently lost
    - overall HEALTH_
    - \(OSDMAP_FLAGS$
    - $OSD_
    - \(PG_
    - \(POOL_
    - \(CACHE_POOL_
    - \(SMALLER_PGP_NUM$
    - $OBJECT_
    - \(SLOW_OPS$
    - $REQUEST_SLOW$
    - $TOO_FEW_PGS$
    - slow request
  - sha1: b40a3bed7bdeff6224dd522c5b540bbe0d11c858
- ceph-deploy:
  - conf:
    - client:
      - log file: /var/log/ceph/ceph-$name.$pid.log
    - mon:
      - osd default pool size: 2
- install:
  - ceph:
    - flavor: default
    - sha1: b40a3bed7bdeff6224dd522c5b540bbe0d11c858
- thrashosds:
  - bdev_inject_crash: 2
  - bdev_inject_crash_probability: 0.5
  - min_in: 4
- workunit:
  - branch: wip-yuri10-testing-2022-10-19-0810
  - sha1: b40a3bed7bdeff6224dd522c5b540bbe0d11c858
owner: scheduled_yuriw@teuthology
pid:
roles:
- ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
- ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2d35fa9e086c4ab9a9d8be6c3c1e0298
status: fail
success: False
branch: wip-yuri10-testing-2022-10-19-0810
seed:
sha1: b40a3bed7bdeff6224dd522c5b540bbe0d11c858
subset:
suite:
suite_branch: wip-yuri10-testing-2022-10-19-0810
suite_path:
suite_relpath:
suite_repo:
suite_sha1: b40a3bed7bdeff6224dd522c5b540bbe0d11c858
targets:
- smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBG2nq3TtCrWXx8sc+3uRdSXZPkg3PYfd0N5Q4tnugu5Q8878TCKeChhSze+QsT6knBvD1Mn+YuxbIqAK8S/dDow=
- smithi045.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZWPjnkXDN4ljKlKGY7ZBbvgL1X3petdjh8iyS5NPvZsMA61I8uuBr345zkmVIoADItyFR/iCOkrkYAO5NpKrNaatK1CbL8vzPJMiuW6dNIw/wE7a7vKuJMxY9E9oi2MXtzMl1b/hWzXLJpq8g0aOjKRylZd7vWC/OUXL6BuwIK4smfMnEWcmha6FCpjyt/8/2VWLPuhkz4HIYpdktDJ33g+VaA/JDCQdHOL+jZXKM/szLYRPJnmajhLKIpb5/i4H1cmPMfk8aJPaCiYmPNYFclrLr81ch3AgytqE2v4Nt10SmxpvHNzfc9NWZ+rhvvmfBmWv6nFRPTABGts1hx0jXxzbzZDuwoK+KqNub5HA9kJTyE7qRdiIlk6QjgjrjaQQzSGri9feUzbq8pewXmbZ7XRPk5HCyxxBya35JUuRgYr32UV8gQFnjtIhC8eMUwJbKKhfPcS9Evlyt4ZkJsxlpzs2l5WTthJZ1oYhOvEyerfgT15ruhTvL8TVU1KWqnjU=
tasks:
- internal.check_packages:
- internal.buildpackages_prep:
- internal.save_config:
- internal.check_lock:
- internal.add_remotes:
- console_log:
- internal.connect:
- internal.push_inventory:
- internal.serialize_remote_roles:
- internal.check_conflict:
- internal.check_ceph_data:
- internal.vm_setup:
- kernel:
  - kdb: True
  - sha1: distro
- internal.base:
- internal.archive_upload:
- internal.archive:
- internal.coredump:
- internal.sudo:
- internal.syslog:
- internal.timer:
- pcp:
- selinux:
- ansible.cephlab:
- clock:
- install:
- ceph:
  - conf:
    - osd:
      - debug monc: 20
- thrashosds:
  - bluestore_new_sharding: random
  - chance_bluestore_reshard: 1
  - chance_pgnum_grow: 1
  - chance_pgnum_shrink: 1
  - chance_pgpnum_fix: 1
  - timeout: 1200
- exec:
  - client.0:
    - sudo ceph osd pool create low_tier 4
- rados:
  - clients:
    - client.0
  - low_tier_pool: low_tier
  - objects: 500
  - op_weights:
    - read: 50
    - set_redirect: 100
    - tier_promote: 30
  - ops: 4000
  - set_redirect: True
teuthology_branch: main
verbose: True
pcp_grafana_url:
priority:
user:
queue:
posted: 2022-10-19 18:37:21
started: 2022-10-19 23:48:15
updated: 2022-10-20 00:11:30
status_class: danger
runtime: 0:23:15
wait_time: 0:09:34