Pulpito :: Results Dashboard

Posted	Started	Updated	Runtime	Duration	In Waiting	Machine	Teuthology Branch	OS Type	OS Version	Nodes	Status
2022-08-15 17:59:41	2022-08-15 23:50:53	2022-08-16 00:06:13	0:15:20	0:09:07	0:06:13	smithi	main	rhel	8.4	2	fail

Nodes: smithi078 smithi165

Description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/repair_test}

Log: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973866/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0b53083909094a69ad38bc45f99d581b

Failure Reason:

{'smithi165.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT30FttZ5s0AiFYSATx6Y0KjYGuAcvEMWh4'], 'labels': [], 'masters': [], 'uuids': ['ead70463-fb70-40d1-81e1-32f67eef278a']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.290956', 'end': '2022-08-15 23:59:59.250440', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT30FttZ5s0AiFYSATx6Y0KjYGuAcvEMWh4'], 'labels': [], 'masters': [], 'uuids': ['ead70463-fb70-40d1-81e1-32f67eef278a']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-08-15 23:59:57.959484', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-O1eT7A-94aA-KB2Q-NwlE-Lq6O-L8F2-03YTlQ', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-O1eT7A-94aA-KB2Q-NwlE-Lq6O-L8F2-03YTlQ', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi165.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3zufmnabJ4JFLDCceT3Y7Duro9PmCcpe2'], 'labels': [], 'masters': [], 'uuids': ['76232f36-8efc-49d9-a6be-4f8ea51514ca']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3zufmnabJ4JFLDCceT3Y7Duro9PmCcpe2'], 'labels': [], 'masters': [], 'uuids': ['76232f36-8efc-49d9-a6be-4f8ea51514ca']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi165.front.sepia.ceph.com,172.21.15.165' (ECDSA) to the list of known hosts.\r\nubuntu@smithi165.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3sgPUTsU0FrsUXoHZGftiBMVoRcBi96WX'], 'labels': [], 'masters': [], 'uuids': ['9bc1e61d-ff63-4a65-bcb7-b5b0e1ecfbf8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3sgPUTsU0FrsUXoHZGftiBMVoRcBi96WX'], 'labels': [], 'masters': [], 'uuids': ['9bc1e61d-ff63-4a65-bcb7-b5b0e1ecfbf8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi165.front.sepia.ceph.com,172.21.15.165' (ECDSA) to the list of known hosts.\r\nubuntu@smithi165.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3JAYMFpJCdT1Pueh21ZjNoJeO5Zf61HWK'], 'labels': [], 'masters': [], 'uuids': ['447f2ee2-0335-47f9-a15e-21554486719e']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0', 'delta': '0:00:00.027928', 'end': '2022-08-16 00:05:08.287600', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3JAYMFpJCdT1Pueh21ZjNoJeO5Zf61HWK'], 'labels': [], 'masters': [], 'uuids': ['447f2ee2-0335-47f9-a15e-21554486719e']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'non-zero return code', 'rc': 2, 'start': '2022-08-16 00:05:08.259672', 'stderr': "Problem opening /dev/dm-0 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!\nProblem opening /dev/dm-0 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!", 'stderr_lines': ['Problem opening /dev/dm-0 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!', 'Problem opening /dev/dm-0 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!'], 'stdout': '', 'stdout_lines': []}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-35000c500929239ac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-35000c500929239ac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'labels': [], 'masters': [], 'uuids': ['23773305-6e80-4400-b974-fa5bbfb65a07']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '23773305-6e80-4400-b974-fa5bbfb65a07'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5XG09', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500929239ac'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-35000c500929239ac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-35000c500929239ac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'labels': [], 'masters': [], 'uuids': ['23773305-6e80-4400-b974-fa5bbfb65a07']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '23773305-6e80-4400-b974-fa5bbfb65a07'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5XG09', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500929239ac'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3Rh73cH26433Dd35PtMqDSpIgdL7ABFet'], 'labels': [], 'masters': [], 'uuids': ['b257ac89-bbe6-40a0-ab99-bc9ba7d999a8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3', 'delta': '0:00:00.004924', 'end': '2022-08-16 00:05:08.886799', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3Rh73cH26433Dd35PtMqDSpIgdL7ABFet'], 'labels': [], 'masters': [], 'uuids': ['b257ac89-bbe6-40a0-ab99-bc9ba7d999a8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'non-zero return code', 'rc': 2, 'start': '2022-08-16 00:05:08.881875', 'stderr': "Problem opening /dev/dm-3 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!\nProblem opening /dev/dm-3 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!", 'stderr_lines': ['Problem opening /dev/dm-3 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!', 'Problem opening /dev/dm-3 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!'], 'stdout': '', 'stdout_lines': []}]}}

All Details...

log_href: http://qa-proxy.ceph.com/teuthology/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973866/teuthology.log
archive_path: /home/teuthworker/archive/yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi/6973866
description: rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zlib rados supported-random-distro$/{rhel_8} tasks/repair_test}
duration: 0:09:07
email: ceph-qa@ceph.io
failure_reason: {'smithi165.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT30FttZ5s0AiFYSATx6Y0KjYGuAcvEMWh4'], 'labels': [], 'masters': [], 'uuids': ['ead70463-fb70-40d1-81e1-32f67eef278a']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.290956', 'end': '2022-08-15 23:59:59.250440', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT30FttZ5s0AiFYSATx6Y0KjYGuAcvEMWh4'], 'labels': [], 'masters': [], 'uuids': ['ead70463-fb70-40d1-81e1-32f67eef278a']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-08-15 23:59:57.959484', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-O1eT7A-94aA-KB2Q-NwlE-Lq6O-L8F2-03YTlQ', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-O1eT7A-94aA-KB2Q-NwlE-Lq6O-L8F2-03YTlQ', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi165.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3zufmnabJ4JFLDCceT3Y7Duro9PmCcpe2'], 'labels': [], 'masters': [], 'uuids': ['76232f36-8efc-49d9-a6be-4f8ea51514ca']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3zufmnabJ4JFLDCceT3Y7Duro9PmCcpe2'], 'labels': [], 'masters': [], 'uuids': ['76232f36-8efc-49d9-a6be-4f8ea51514ca']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi165.front.sepia.ceph.com,172.21.15.165' (ECDSA) to the list of known hosts.\r\nubuntu@smithi165.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3sgPUTsU0FrsUXoHZGftiBMVoRcBi96WX'], 'labels': [], 'masters': [], 'uuids': ['9bc1e61d-ff63-4a65-bcb7-b5b0e1ecfbf8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3sgPUTsU0FrsUXoHZGftiBMVoRcBi96WX'], 'labels': [], 'masters': [], 'uuids': ['9bc1e61d-ff63-4a65-bcb7-b5b0e1ecfbf8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi165.front.sepia.ceph.com,172.21.15.165' (ECDSA) to the list of known hosts.\r\nubuntu@smithi165.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3JAYMFpJCdT1Pueh21ZjNoJeO5Zf61HWK'], 'labels': [], 'masters': [], 'uuids': ['447f2ee2-0335-47f9-a15e-21554486719e']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0', 'delta': '0:00:00.027928', 'end': '2022-08-16 00:05:08.287600', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3JAYMFpJCdT1Pueh21ZjNoJeO5Zf61HWK'], 'labels': [], 'masters': [], 'uuids': ['447f2ee2-0335-47f9-a15e-21554486719e']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'non-zero return code', 'rc': 2, 'start': '2022-08-16 00:05:08.259672', 'stderr': "Problem opening /dev/dm-0 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!\nProblem opening /dev/dm-0 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!", 'stderr_lines': ['Problem opening /dev/dm-0 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!', 'Problem opening /dev/dm-0 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!'], 'stdout': '', 'stdout_lines': []}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-35000c500929239ac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-35000c500929239ac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'labels': [], 'masters': [], 'uuids': ['23773305-6e80-4400-b974-fa5bbfb65a07']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '23773305-6e80-4400-b974-fa5bbfb65a07'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5XG09', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500929239ac'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-35000c500929239ac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-35000c500929239ac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'labels': [], 'masters': [], 'uuids': ['23773305-6e80-4400-b974-fa5bbfb65a07']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '23773305-6e80-4400-b974-fa5bbfb65a07'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5XG09', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500929239ac'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3Rh73cH26433Dd35PtMqDSpIgdL7ABFet'], 'labels': [], 'masters': [], 'uuids': ['b257ac89-bbe6-40a0-ab99-bc9ba7d999a8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3', 'delta': '0:00:00.004924', 'end': '2022-08-16 00:05:08.886799', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3Rh73cH26433Dd35PtMqDSpIgdL7ABFet'], 'labels': [], 'masters': [], 'uuids': ['b257ac89-bbe6-40a0-ab99-bc9ba7d999a8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'non-zero return code', 'rc': 2, 'start': '2022-08-16 00:05:08.881875', 'stderr': "Problem opening /dev/dm-3 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!\nProblem opening /dev/dm-3 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!", 'stderr_lines': ['Problem opening /dev/dm-3 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!', 'Problem opening /dev/dm-3 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!'], 'stdout': '', 'stdout_lines': []}]}}
flavor:
job_id: 6973866
kernel:
- kdb: True
- sha1: distro
last_in_suite: False
machine_type: smithi
name: yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi
nuke_on_error: True
os_type: rhel
os_version: 8.4
overrides:
- admin_socket:
  - branch: wip-yuri2-testing-2022-08-15-0848-quincy
- ceph:
  - conf:
    - global:
      - mon client directed command retry: 5
      - mon election default strategy: 1
      - ms bind msgr2: False
      - ms inject socket failures: 5000
      - ms type: async
    - mgr:
      - debug mgr: 20
      - debug ms: 1
    - mon:
      - debug mon: 20
      - debug ms: 1
      - debug paxos: 20
      - mon min osdmap epochs: 50
      - mon osdmap full prune interval: 2
      - mon osdmap full prune min: 15
      - mon osdmap full prune txsize: 2
      - mon scrub interval: 300
      - paxos service trim min: 10
    - osd:
      - bluestore block size: 96636764160
      - bluestore compression algorithm: zlib
      - bluestore compression mode: aggressive
      - bluestore debug inject read err: True
      - bluestore fsck on mount: True
      - bluestore zero block detection: True
      - debug bluefs: 20
      - debug bluestore: 20
      - debug ms: 1
      - debug osd: 20
      - debug rocksdb: 10
      - filestore debug inject read err: True
      - mon osd backfillfull_ratio: 0.85
      - mon osd full ratio: 0.9
      - mon osd nearfull ratio: 0.8
      - osd debug verify cached snaps: True
      - osd debug verify missing on start: True
      - osd failsafe full ratio: 0.95
      - osd objectstore: bluestore
      - osd op queue: debug_random
      - osd op queue cut off: debug_random
      - osd shutdown pgref assert: True
  - flavor: default
  - fs: xfs
  - log-ignorelist:
    - $MDS_ALL_DOWN$
    - $MDS_UP_LESS_THAN_MAX$
    - $OSD_SLOW_PING_TIME
    - candidate had a stat error
    - candidate had a read error
    - deep-scrub 0 missing, 1 inconsistent objects
    - deep-scrub 0 missing, 4 inconsistent objects
    - deep-scrub [0-9]+ errors
    - != omap_digest
    - != data_digest
    - repair 0 missing, 1 inconsistent objects
    - repair 0 missing, 4 inconsistent objects
    - repair [0-9]+ errors, [0-9]+ fixed
    - scrub 0 missing, 1 inconsistent objects
    - scrub [0-9]+ errors
    - size 1 != size
    - attr name mismatch
    - Regular scrub request, deep-scrub details will be lost
    - candidate size [0-9]+ info size [0-9]+ mismatch
    - overall HEALTH_
    - \(OSDMAP_FLAGS$
    - \(OSD_
    - \(PG_
  - mon_bind_msgr2: False
  - sha1: 7451e3a8fd516a9ef77cc4fae440433cb71033fb
  - wait-for-scrub: False
- ceph-deploy:
  - conf:
    - client:
      - log file: /var/log/ceph/ceph-$name.$pid.log
    - mon:
      - osd default pool size: 2
- install:
  - ceph:
    - flavor: default
    - sha1: 7451e3a8fd516a9ef77cc4fae440433cb71033fb
- selinux:
  - whitelist:
    - scontext=system_u:system_r:logrotate_t:s0
- thrashosds:
  - bdev_inject_crash: 2
  - bdev_inject_crash_probability: 0.5
- workunit:
  - branch: wip-yuri2-testing-2022-08-15-0848-quincy
  - sha1: 7451e3a8fd516a9ef77cc4fae440433cb71033fb
owner: scheduled_yuriw@teuthology
pid:
roles:
- ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a']
- ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0b53083909094a69ad38bc45f99d581b
status: fail
success: False
branch: wip-yuri2-testing-2022-08-15-0848-quincy
seed:
sha1: 7451e3a8fd516a9ef77cc4fae440433cb71033fb
subset:
suite:
suite_branch: wip-yuri2-testing-2022-08-15-0848-quincy
suite_path:
suite_relpath:
suite_repo:
suite_sha1: 7451e3a8fd516a9ef77cc4fae440433cb71033fb
targets:
- smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtf2N7borydbWTlYThS0Ogb9Dg8l6BVORqneRO0jEEHKS144yrD86m9w6jbqQto35x2wgNNP8XhJJeAnboSA6Oept2JiJOsMccJeZ84ZzKu7AR875Ze5CCkXi0yPhq0Sj9jOiPnQ2OTGrORr1WTekKOl6wjeWB+zsNaUfJ4ehh86IRXPD+W6W6KZ9qUu1rbY9l4Tu1/geJiMGJRNfrv8ar09sZogMb3pfZOpAa16DomnfuYb/76IMWdAqiMGAm3GUYqznE+eB5+R9p07YbPvmZ/kFA8E+HFvLrdZu19ANWIxQJDW5XTSyiVR5L6SCbSsHztAbgPDzhphKUsMWdTXteXaxNbbVS57kAgtrSDMlluAHjvr6bLOMjqDGjroyejXJ5q9nBRe10bZcaxiKxUmKJ9+V2+7hlfe9yLR+5skNr7Mg7FDXoSx+46rE17ubFdtdanuWlWvsJ745IILEqC+EkhZWn3HwNbqFnpqtyt1E53NYiILlKuwxYS/H72df6mN0=
- smithi165.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj+TEAnwI/N26xmmrLQ1tVznxRnJY80QgPtVfI1aFVG9ZS7qbvyBx/eP2QEm9W7alBYtlg2jGDBSAW99CCkl5ic/GfPFADf7N8Yb4X2e5lUYvOgLv7Vj/FXH/JFReK+jg58e9LXWWZZuAfCyfMnHoQzDe5Mp0bCDevisVHNq2h50wti1GvFlMNFPYDh7qwo2YG37o4xbe5f/6w8vtGOBNZnTXlHXPYp+Ucr0uBx0EYZvIJhYJkY67eF3VcRz8tYTw4fxlV6xdu5iAe4zx2dJwykU/emT51czdWB36dVxH+CXQoKYwtOYS2GssUbfg/e1FXioPi9apal0bYPojkz5zwW28xbzmu8kABfctIax46Y1oeEegS75AZNoaSzzCCGAh+GLgH7+Hvuojo2ieF2bDzxzB0SRSzpN16L4SCExDec9UxrfWXIYgG1kGFueE/OWbVEV7SBdVSUsgz91WhiKTq7rzHBtOltXrIehQMtuXNZQJPTI/7iTfZ8JNEDworc3E=
tasks:
- internal.check_packages:
- internal.buildpackages_prep:
- internal.save_config:
- internal.check_lock:
- internal.add_remotes:
- console_log:
- internal.connect:
- internal.push_inventory:
- internal.serialize_remote_roles:
- internal.check_conflict:
- internal.check_ceph_data:
- internal.vm_setup:
- kernel:
  - kdb: True
  - sha1: distro
- internal.base:
- internal.archive_upload:
- internal.archive:
- internal.coredump:
- internal.sudo:
- internal.syslog:
- internal.timer:
- pcp:
- selinux:
- ansible.cephlab:
- clock:
- install:
  - extra_system_packages:
    - deb:
      - sqlite3
    - rpm:
      - sqlite-devel
- ceph:
- repair_test:
teuthology_branch: main
verbose: True
pcp_grafana_url:
priority:
user:
queue:
posted: 2022-08-15 17:59:41
started: 2022-08-15 23:50:53
updated: 2022-08-16 00:06:13
status_class: danger
runtime: 0:15:20
wait_time: 0:06:13