yuriw-2022-08-15_17:54:08-rados-wip-yuri2-testing-2022-08-15-0848-quincy-distro-default-smithi

'wait for operator' reached maximum tries (90) after waiting for 900 seconds

expected valgrind issues and found none

Command failed on smithi039 with status 22: 'kubectl -n rook-ceph exec rook-ceph-tools-79bc54b8d8-zpjbq -- ceph orch apply osd --all-available-devices'

hit max job timeout

Test failure: test_export_create_with_non_existing_fsname (tasks.cephfs.test_nfs.TestNFS)

'wait for operator' reached maximum tries (90) after waiting for 900 seconds

{'smithi165.front.sepia.ceph.com': {'censored': "the output has been hidden due to the fact that 'no_log: true' was specified for this result", 'changed': True}}

{'smithi165.front.sepia.ceph.com': {'changed': True, 'msg': 'All items completed', 'results': [{'_ansible_item_label': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT30FttZ5s0AiFYSATx6Y0KjYGuAcvEMWh4'], 'labels': [], 'masters': [], 'uuids': ['ead70463-fb70-40d1-81e1-32f67eef278a']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', 'delta': '0:00:01.290956', 'end': '2022-08-15 23:59:59.250440', 'failed': False, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-1', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT30FttZ5s0AiFYSATx6Y0KjYGuAcvEMWh4'], 'labels': [], 'masters': [], 'uuids': ['ead70463-fb70-40d1-81e1-32f67eef278a']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': '', 'rc': 0, 'start': '2022-08-15 23:59:57.959484', 'stderr': '', 'stderr_lines': [], 'stdout': 'Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.', 'stdout_lines': ['Creating new GPT entries.', 'Warning: The kernel is still using the old partition table.', 'The new table will be used at the next reboot or after you', 'run partprobe(8) or kpartx(8)', 'GPT data structures destroyed! You may now partition the disk using fdisk or', 'other utilities.']}, {'_ansible_item_label': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-O1eT7A-94aA-KB2Q-NwlE-Lq6O-L8F2-03YTlQ', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'nvme0n1', 'value': {'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'links': {'ids': ['lvm-pv-uuid-O1eT7A-94aA-KB2Q-NwlE-Lq6O-L8F2-03YTlQ', 'nvme-INTEL_SSDPEDMD400G4_CVFT623300CE400BGN', 'nvme-nvme.8086-43564654363233333030434534303042474e-494e54454c205353445045444d443430304734-00000001'], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'uuids': []}, 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi165.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3zufmnabJ4JFLDCceT3Y7Duro9PmCcpe2'], 'labels': [], 'masters': [], 'uuids': ['76232f36-8efc-49d9-a6be-4f8ea51514ca']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-4', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3zufmnabJ4JFLDCceT3Y7Duro9PmCcpe2'], 'labels': [], 'masters': [], 'uuids': ['76232f36-8efc-49d9-a6be-4f8ea51514ca']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi165.front.sepia.ceph.com,172.21.15.165' (ECDSA) to the list of known hosts.\r\nubuntu@smithi165.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3sgPUTsU0FrsUXoHZGftiBMVoRcBi96WX'], 'labels': [], 'masters': [], 'uuids': ['9bc1e61d-ff63-4a65-bcb7-b5b0e1ecfbf8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'ansible_loop_var': 'item', 'item': {'key': 'dm-2', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3sgPUTsU0FrsUXoHZGftiBMVoRcBi96WX'], 'labels': [], 'masters': [], 'uuids': ['9bc1e61d-ff63-4a65-bcb7-b5b0e1ecfbf8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi165.front.sepia.ceph.com,172.21.15.165' (ECDSA) to the list of known hosts.\r\nubuntu@smithi165.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}, {'_ansible_item_label': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3JAYMFpJCdT1Pueh21ZjNoJeO5Zf61HWK'], 'labels': [], 'masters': [], 'uuids': ['447f2ee2-0335-47f9-a15e-21554486719e']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0', 'delta': '0:00:00.027928', 'end': '2022-08-16 00:05:08.287600', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-0', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3JAYMFpJCdT1Pueh21ZjNoJeO5Zf61HWK'], 'labels': [], 'masters': [], 'uuids': ['447f2ee2-0335-47f9-a15e-21554486719e']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'non-zero return code', 'rc': 2, 'start': '2022-08-16 00:05:08.259672', 'stderr': "Problem opening /dev/dm-0 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!\nProblem opening /dev/dm-0 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!", 'stderr_lines': ['Problem opening /dev/dm-0 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!', 'Problem opening /dev/dm-0 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!'], 'stdout': '', 'stdout_lines': []}, {'_ansible_item_label': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-35000c500929239ac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-35000c500929239ac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'labels': [], 'masters': [], 'uuids': ['23773305-6e80-4400-b974-fa5bbfb65a07']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '23773305-6e80-4400-b974-fa5bbfb65a07'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5XG09', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500929239ac'}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': False, 'item': {'key': 'sda', 'value': {'holders': [], 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09', 'scsi-35000c500929239ac', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09', 'wwn-0x5000c500929239ac'], 'labels': [], 'masters': [], 'uuids': []}, 'model': 'ST1000NM0033-9ZM', 'partitions': {'sda1': {'holders': [], 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5XG09-part1', 'scsi-35000c500929239ac-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5XG09-part1', 'wwn-0x5000c500929239ac-part1'], 'labels': [], 'masters': [], 'uuids': ['23773305-6e80-4400-b974-fa5bbfb65a07']}, 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'start': '2048', 'uuid': '23773305-6e80-4400-b974-fa5bbfb65a07'}}, 'removable': '0', 'rotational': '1', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'serial': 'Z1W5XG09', 'size': '931.51 GB', 'support_discard': '0', 'vendor': 'ATA', 'virtual': 1, 'wwn': '0x5000c500929239ac'}}, 'skip_reason': 'Conditional result was False', 'skipped': True}, {'_ansible_item_label': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3Rh73cH26433Dd35PtMqDSpIgdL7ABFet'], 'labels': [], 'masters': [], 'uuids': ['b257ac89-bbe6-40a0-ab99-bc9ba7d999a8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, '_ansible_no_log': False, 'ansible_loop_var': 'item', 'changed': True, 'cmd': 'sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3', 'delta': '0:00:00.004924', 'end': '2022-08-16 00:05:08.886799', 'failed': True, 'invocation': {'module_args': {'_raw_params': 'sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3', '_uses_shell': True, 'argv': None, 'chdir': None, 'creates': None, 'executable': None, 'removes': None, 'stdin': None, 'stdin_add_newline': True, 'strip_empty_ends': True, 'warn': True}}, 'item': {'key': 'dm-3', 'value': {'holders': [], 'host': '', 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dF9sFmWbcumfcbBilW3yUe4vKKz67tT3Rh73cH26433Dd35PtMqDSpIgdL7ABFet'], 'labels': [], 'masters': [], 'uuids': ['b257ac89-bbe6-40a0-ab99-bc9ba7d999a8']}, 'model': None, 'partitions': {}, 'removable': '0', 'rotational': '0', 'sas_address': None, 'sas_device_handle': None, 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'support_discard': '512', 'vendor': None, 'virtual': 1}}, 'msg': 'non-zero return code', 'rc': 2, 'start': '2022-08-16 00:05:08.881875', 'stderr': "Problem opening /dev/dm-3 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!\nProblem opening /dev/dm-3 for reading! Error is 2.\nThe specified file does not exist!\nProblem opening '' for writing! Program will now terminate.\nWarning! MBR not overwritten! Error is 2!", 'stderr_lines': ['Problem opening /dev/dm-3 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!', 'Problem opening /dev/dm-3 for reading! Error is 2.', 'The specified file does not exist!', "Problem opening '' for writing! Program will now terminate.", 'Warning! MBR not overwritten! Error is 2!'], 'stdout': '', 'stdout_lines': []}]}}

Error reimaging machines: 'ssh_keyscan smithi165.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

expected valgrind issues and found none

Command failed (workunit test rados/test.sh) on smithi071 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7451e3a8fd516a9ef77cc4fae440433cb71033fb TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh'

Command failed on smithi145 with status 22: 'kubectl -n rook-ceph exec rook-ceph-tools-79bc54b8d8-ktxjk -- ceph orch apply osd --all-available-devices'