teuthology-2021-04-19_03:30:03-rados-nautilus-distro-basic-gibba

6057141

DEAD

nautilus

nautilus

master

gibba

ubuntu 18.04

gibba021

rados/singleton/{all/mon-config msgr-failures/many msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{ubuntu_latest}}

Error reimaging machines: Failed to power on gibba021

6057154

DEAD

nautilus

nautilus

master

gibba

rhel 7.9

gibba006 gibba021

rados/thrash-erasure-code/{ceph clusters/{fixed-2 openstack} fast/normal msgr-failures/fastclose objectstore/bluestore-comp-snappy rados recovery-overrides/{more-active-recovery} supported-random-distro$/{rhel_7} thrashers/default thrashosds-health workloads/ec-rados-plugin=jerasure-k=3-m=1}

Error reimaging machines: Failed to power on gibba021

6057175

DEAD

nautilus

nautilus

master

gibba

ubuntu 18.04

gibba009 gibba017

rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/normal ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/upmap msgr-failures/osd-delay msgr/async-v1only objectstore/filestore-xfs rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/radosbench}

Error reimaging machines: 'NoneType' object has no attribute 'get'

6057196

DEAD

nautilus

nautilus

master

gibba

ubuntu 16.04

gibba009 gibba045

rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{ubuntu_16.04} tasks/rados_workunit_loadgen_mix}

Error reimaging machines: 'gibba009.front.sepia.ceph.com'

6057246

DEAD

nautilus

nautilus

master

gibba

ubuntu 18.04

gibba039

rados/singleton-nomsgr/{all/full-tiering rados supported-random-distro$/{ubuntu_latest}}

Error reimaging machines: 'gibba039.front.sepia.ceph.com'

6057301

DEAD

nautilus

nautilus

master

gibba

centos 7.8

gibba005 gibba042

rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/cache-agent-small}

Error reimaging machines: 'gibba005.front.sepia.ceph.com'

6057759

DEAD

nautilus

nautilus

master

gibba

ubuntu 16.04

gibba017 gibba030

rados/monthrash/{ceph clusters/9-mons msgr-failures/mon-delay msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_16.04} thrashers/many workloads/pool-create-delete}

SSH connection to gibba030 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

6057765

DEAD

nautilus

nautilus

master

gibba

centos 7.8

gibba007 gibba030

rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/off msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/careful thrashosds-health workloads/small-objects}

6057767

DEAD

nautilus

nautilus

master

gibba

centos 7.8

gibba017 gibba030 gibba041

rados/multimon/{clusters/9 msgr-failures/many msgr/async-v1only no_pools objectstore/bluestore-hybrid rados supported-random-distro$/{centos_latest} tasks/mon_clock_no_skews}

SSH connection to gibba030 was lost: 'rpm -q kernel --last | head -n 1'

6057807

DEAD

nautilus

nautilus

master

gibba

centos 7.8

gibba016 gibba017

rados/basic/{ceph clusters/{fixed-2 openstack} msgr-failures/many msgr/simple objectstore/filestore-xfs rados supported-random-distro$/{centos_latest} tasks/rados_python}

Error reimaging machines: sequence item 0: expected str instance, NoneType found

6057821

DEAD

nautilus

nautilus

master

gibba

ubuntu 16.04

gibba010 gibba040

rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_16.04} tasks/workunits}

SSH connection to gibba010 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

6057829

DEAD

nautilus

nautilus

master

gibba

ubuntu 18.04

gibba011 gibba043

rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/fastclose msgr/async-v1only objectstore/bluestore-hybrid rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap thrashosds-health workloads/write_fadvise_dontneed}

SSH connection to gibba011 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

6057835

DEAD

nautilus

nautilus

master

gibba

centos 7.8

gibba035 gibba037

rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-active-recovery} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/crush-compat msgr-failures/fastclose msgr/random objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-agent-small}

SSH connection to gibba035 was lost: 'rpm -q kernel --last | head -n 1'

6057846

DEAD

nautilus

nautilus

master

gibba

ubuntu 18.04

gibba035 gibba037

rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} backoff/peering_and_degraded ceph clusters/{fixed-2 openstack} crc-failures/default d-balancer/off msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache}

SSH connection to gibba035 was lost: 'rpm -q kernel --last | head -n 1'

6057860

DEAD

nautilus

nautilus

master

gibba

ubuntu 16.04

gibba010

rados/objectstore/{backends/alloc-hint supported-random-distro$/{ubuntu_16.04}}

SSH connection to gibba010 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

6057870

DEAD

nautilus

nautilus

master

gibba

centos 7.8

gibba010 gibba030 gibba037 gibba041

rados/upgrade/mimic-x-singleton/{0-cluster/{openstack start} 1-install/mimic 2-partial-upgrade/firsthalf 3-thrash/default 4-workload/{rbd-cls rbd-import-export readwrite snaps-few-objects} 5-workload/{radosbench rbd_api} 6-finish-upgrade 7-nautilus 8-workload/{rbd-python rgw-swift snaps-many-objects} supported-random-distro$/{centos_latest} thrashosds-health}

SSH connection to gibba010 was lost: 'rpm -q kernel --last | head -n 1'

6057888

DEAD

nautilus

nautilus

master

gibba

ubuntu 16.04

gibba010 gibba030 gibba037

rados/multimon/{clusters/21 msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_16.04} tasks/mon_clock_with_skews}

Error reimaging machines: 'ssh_keyscan gibba010.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

6057906

DEAD

nautilus

nautilus

master

gibba

ubuntu 16.04

gibba002 gibba007 gibba010 gibba027

rados/thrash-old-clients/{0-size-min-size-overrides/2-size-2-min-size 1-install/jewel backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/off distro$/{ubuntu_16.04} msgr-failures/fastclose msgr/random rados thrashers/mapgap thrashosds-health workloads/test_rbd_api}

SSH connection to gibba010 was lost: 'sudo apt-get clean'

6057918

DEAD

nautilus

nautilus

master

gibba

rhel 7.9

gibba002 gibba010

rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{default} backoff/peering ceph clusters/{fixed-2 openstack} crc-failures/bad_map_crc_failure d-balancer/crush-compat msgr-failures/fastclose msgr/simple objectstore/bluestore-comp-zstd rados supported-random-distro$/{rhel_7} thrashers/mapgap thrashosds-health workloads/cache-snaps}

Failure object was: {'gibba010.front.sepia.ceph.com': {'results': [{'msg': 'line added', 'diff': [{'after': '', 'before_header': '/scratch_devs (content)', 'after_header': '/scratch_devs (content)', 'before': ''}, {'before_header': '/scratch_devs (file attributes)', 'after_header': '/scratch_devs (file attributes)'}], 'changed': True, 'backup': '', 'invocation': {'module_args': {'directory_mode': None, 'force': None, 'remote_src': None, 'backrefs': False, 'insertafter': None, 'path': '/scratch_devs', 'owner': 'root', 'follow': False, 'line': '/dev/vg_nvme/lv_1', 'group': 'root', 'insertbefore': None, 'unsafe_writes': None, 'create': True, 'state': 'present', 'content': None, 'serole': None, 'setype': None, 'dest': '/scratch_devs', 'selevel': None, 'regexp': None, 'validate': None, 'src': None, 'seuser': None, 'delimiter': None, 'mode': 420, 'firstmatch': False, 'attributes': None, 'backup': False}}, '_ansible_no_log': False, 'failed': False, 'item': {'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}, 'ansible_loop_var': 'item', '_ansible_item_label': {'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}}, {'unreachable': True, 'msg': 'Data could not be sent to remote host "gibba010.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host gibba010.front.sepia.ceph.com port 22: No route to host\r\n', 'item': {'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}, 'ansible_loop_var': 'item', '_ansible_item_label': {'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}}, {'unreachable': True, 'msg': 'Data could not be sent to remote host "gibba010.front.sepia.ceph.com". Make sure this host can be reached over ssh: Warning: Permanently added \'gibba010.front.sepia.ceph.com,172.21.2.110\' (ECDSA) to the list of known hosts.\r\nubuntu@gibba010.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).\r\n', 'item': {'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}, 'ansible_loop_var': 'item', '_ansible_item_label': {'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}}, {'unreachable': True, 'msg': 'Data could not be sent to remote host "gibba010.front.sepia.ceph.com". Make sure this host can be reached over ssh: Warning: Permanently added \'gibba010.front.sepia.ceph.com,172.21.2.110\' (ECDSA) to the list of known hosts.\r\nubuntu@gibba010.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).\r\n', 'item': {'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}, 'ansible_loop_var': 'item', '_ansible_item_label': {'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '84G', 'scratch_dev': True}}}, {'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', '_ansible_no_log': False, 'item': {'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}, 'ansible_loop_var': 'item', '_ansible_item_label': {'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '13G'}}}], 'changed': True, 'msg': 'All items completed'}}Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence('tag:yaml.org,2002:seq', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping('tag:yaml.org,2002:map', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)yaml.representer.RepresenterError: ('cannot represent an object', 'key')

6057925

DEAD

nautilus

nautilus

master

gibba

rhel 7.9

gibba010 gibba017

rados/mgr/{clusters/{2-node-mgr} debug/mgr objectstore/bluestore-comp-lz4 supported-random-distro$/{rhel_7} tasks/crash}

{'Failure object was': {'gibba010.front.sepia.ceph.com': {'changed': True, 'end': '2021-05-04 20:55:02.798071', 'stdout': 'This system has no repositories available through subscriptions.', 'cmd': ['subscription-manager', 'repos', '--enable=rhel-7-server-rpms', '--enable=rhel-7-server-optional-rpms', '--enable=rhel-7-server-extras-rpms', '--enable=rhel-ha-for-rhel-7-server-rpms'], 'delta': '0:00:00.124122', 'stderr': '', 'rc': 1, 'invocation': {'module_args': {'creates': 'None', 'executable': 'None', '_uses_shell': False, 'strip_empty_ends': True, '_raw_params': 'subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-optional-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-ha-for-rhel-7-server-rpms ', 'removes': 'None', 'argv': 'None', 'warn': True, 'chdir': 'None', 'stdin_add_newline': True, 'stdin': 'None'}}, 'start': '2021-05-04 20:55:02.673949', 'msg': 'non-zero return code', 'stdout_lines': ['This system has no repositories available through subscriptions.'], 'stderr_lines': [], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 117, in represent_mapping node_key = self.represent_data(item_key) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_2713a3cd31b17738a50039eaa9d859b5dc39fb8a/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'changed')"}

6057938