teuthology-2020-05-10_07:01:02-rados-master-distro-basic-smithi

psutil.NoSuchProcess process no longer exists (pid=6503)

SSH connection to smithi110 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

Command failed on smithi164 with status 1: 'sudo yum install -y kernel'

SSH connection to smithi138 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

Failed to fetch package version from https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=nautilus

psutil.NoSuchProcess process no longer exists (pid=31599)

psutil.NoSuchProcess process no longer exists (pid=29421)

[Errno None] Unable to connect to port 22 on 172.21.15.100

[Errno None] Unable to connect to port 22 on 172.21.15.104

Host key for server 'smithi079.front.sepia.ceph.com' does not match: got 'AAAAB3NzaC1yc2EAAAADAQABAAABAQDU0NrDdfKotc1CnXt7lStrKXX397jK/h55Ik/WzMomJtT8nJZM1fuiyVihOEJHEOMmebZ8aOcVPOYPH3Gv1lTUIza2OSNmoBsW2g1SuNTx426vj3YVMBeGfitHuuaZDYNWKOYKnK1jLsbKyaYCBajVFrX0B4hXkZbOPvyLWkKlyGCSem/YHAgs6fx/6x2uS7vXP08qILsUTwc6YlklrSHty2mfwJKDOvLc+sXh+/cFgXEOFXByQTe3A43JJizgLOnAoYlYtLHK0gM4eeBJhd7TyPMnZ5dteAIqjHRiGyUT4yVRtkSm0daq3n8V5K7ZGuH7AmA5SqxNDSvAlPl/ZnZJ', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDmO9VPa9dhRLAbWxp6vwMeF6QpiB6MSO4e4PnbQdyIiIg8eHE1xl1+aNBHzwXLJriuCHJiA5Lrz/S8u62QZZe4Nadsxmka62oInWB4a8oobgJwo8sBT1LAZMxx74eBKDY9x58KRRXwrdgVh7/4BGND7oAli45VaqnmfayB2jWRJzSqNfJ2FhxO/RD69dBoSSnRWtFL+kMC4OTu2CC7NxLKqO1knG84aKWDR3w+KiGDSRc8GBkv7HiveGceIm5Whi1uibpJGp56dns5QXj1Uilonml5duZ8ElKgBXNieGtONdQUuR3H3Gdm2OLWz4GKPQp5LTz628Th1YRJrfH0hW8vq7BC3tY21n6TKJqnuWos99VYcR2RGUlsV4L8o+Md8mBSEg30IkF41saFO3JovLZR+oH4bMDmXp5qDkI2PsWEsYL4IvmqaZbdydx/nYp9zzYEywW/9OB+CNlJtwwknwC5zANX5Wp7OuQyI72MNf9Z8gCJ37HzEh1o2u8V3uve3UE='

SSH connection to smithi139 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

SSH connection to smithi201 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

psutil.NoSuchProcess process no longer exists (pid=6785)

SSH connection to smithi138 was lost: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 -p unique_pool_2 bench 90 write'"

Test failure: setUpClass (tasks.mgr.dashboard.test_crush_rule.CrushRuleTest)

Command failed on smithi175 with status 4: "sudo bash -c 'nmcli connection up ceph-brx'"

SSH connection to smithi164 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

psutil.NoSuchProcess process no longer exists (pid=24274)

psutil.NoSuchProcess process no longer exists (pid=26136)

psutil.NoSuchProcess process no longer exists (pid=31397)

[Errno None] Unable to connect to port 22 on 172.21.15.32

SSH connection to smithi043 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

Host key for server 'smithi064.front.sepia.ceph.com' does not match: got 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDj+ORm2KE1h6U0UDFsEJuZjxfLKy8sDeE3iKTGQ67EOedDYBsYuThLgo0ZQVng4JZV6lm+VaU+BuCybVK/m1rz4+OwkRiqd2BCEMfEXMRYbUKY+rELxG7xb5ZdbG0IXeY3awzjVzXOlXh7IW+hXysNtoAY4jswKSeJSEcunuPk1Bt/x6KRahhEXs7ULS8LJ5i4Jmer15LcrlGmZirD11MY/Z7weogMi6cnmhVfmyjCJkJypIv9AoAZccwEM0f4uQ+v/M/UZxpN0nXVJGXTPes4gY4OsZZPxXn1DSanA+zG0AhK7zxqGAHTEFhyXfUsxrvTrvGWGX6aNPn3BJ6aYgPXgI+g+ZZVdj8nPYA6SgsDTdXeh1PSCUWMQtZumi9P/pcrvzmY0La/A8LVJIX0zHHeHSB9ogpQPITgKEfaP7/zAXhoS4LgFjZLUtqpx9mWeLtazul/7LrqsEgWIeao42kgwWOOR4gGoupRlytNEkuY0NExMPlJBaGpAbPSXBWCymE=', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQC9mpve62Ih+AyGLR/imLBF/fIYX/76X+rzA7lHeK4O+I3BVnEiG8gaZN4e1tk/DXUrezAxZzZdS0VRIWvGD0CECYfy/12YLcMw7WEa+jrxr+DQ/rZKb+P+6c8URo8YC/BoXOdn2vm+fwF9h8t2lhXXs9E7Pp60M+AaRDp/6DxmPhOqByh3/PGqG33N7JD2Cc2+eMlVfq6TDx9oQFLXZTrD+/92PChEmLaAIzbelhwGILyx6uC1i2AYl9JYxt9uOxS4KZM0b0dMbB8DpjDPUGW1d1sbvOO9ioMmBdF6+Ilv4dQXcquiY38YLfc5qxJvWgPnPX3T67hsAYKLETzurBfv87lo9+y+h3cRnkVBOtmppRrO+wA9N/xOEb1MVEyI5IhdONEoQUWHiDOvp3kzgBGTAPsX92t0OTUnup0PQ7NQ9N1YznXUuyQJonnvGSCVhEfdpRvnwAYg2HJgT+BfPlGzeF5CkRuLYWCyXcT0he95IHBa1lzsiLE2TMYIhZ3aitE='

psutil.NoSuchProcess process no longer exists (pid=15979)

SSH connection to smithi205 was lost: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 -p unique_pool_2 bench 90 write'"

psutil.NoSuchProcess process no longer exists (pid=25412)

[Errno None] Unable to connect to port 22 on 172.21.15.129

psutil.NoSuchProcess process no longer exists (pid=29747)

SSH connection to smithi185 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

psutil.NoSuchProcess process no longer exists (pid=31787)

Test failure: test_create_can_handle_boolean_values (tasks.mgr.dashboard.test_cluster_configuration.ClusterConfigurationTest)

SSH connection to smithi138 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

psutil.NoSuchProcess process no longer exists (pid=1262)

SSH connection to smithi045 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

Command failed (workunit test cephadm/test_repos.sh) on smithi169 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ba805463ec24e1cab34f27f70162ebbddf5d68c7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_repos.sh'

SSH connection to smithi139 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 10000 --objects 6600 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 1200 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op write_excl 50 --pool base'

SSH connection to smithi202 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --balance-reads --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base'

'ssh_keyscan smithi089.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

[Errno None] Unable to connect to port 22 on 172.21.15.32

Stale jobs detected, aborting.

Command failed on smithi068 with status 1: 'sudo yum install -y kernel'

'ssh_keyscan smithi146.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

{'smithi159.front.sepia.ceph.com': {'changed': False, 'msg': "Failed to connect to the host via ssh: Warning: Permanently added 'smithi159.front.sepia.ceph.com,172.21.15.159' (ECDSA) to the list of known hosts.\r\nubuntu@smithi159.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).", 'unreachable': True}}

Host key for server 'smithi039.front.sepia.ceph.com' does not match: got 'AAAAB3NzaC1yc2EAAAADAQABAAABAQC1tmCDUHxc5O0pnjl/Kv8RO+RS8O6lAatYgIvwWf3WLSwONWCXCZ6fXNW+VmPQ0JHONuBYJMjJQq5t+kBwztzxww68MlA+iGKyxEguiVNRV5BWoGuaOrZypJdG/2mvEaXhLMkDZOQ8O8iPftRhkilhWSp7mNQj5Wg5+4j7VoyMvXEDz7tMZ33Hl71nMNQhZCOBNEwDZFs3Jqfi/YUmyyAfg53YUn3SUGHtX9doURRx1wcYKjM0Oi3f+Z+7FO2HTKstgzmYBx1ITFnTJnemwAHyIYwpiG2GcrP8Wrx0MxRZoGqo2SqS4btEQ8kLeZSTbfAJAu3t5vbfgjx/339vW2Dr', expected 'AAAAB3NzaC1yc2EAAAADAQABAAABgQDMu3RPQJ361lAotHrx3OCw/fas1Z7ARAnJ+SBQhaRGHyLiSpjdWjPHbq86aaITFV3BIwiuXlvuLqf4/HwWyZ6Aku7Yxr2K3yTyPi+TrUAUS7oTI6SFv0UAX8sZetGeWqGrWlyvT6ejROJDhPcT9k5+ZElNX70bd5Fsl0m9KPxTmi4PP1+9b7CI34Y0ZXSBXiN7lq4ZQC0OzZudMvGIaLKsZkdE1UqfFdanyWjRpm+R/zHRrmU7t5/Xtp7MXkS3Bf9PcweDYwVGw8vP0tIPTORhH6HFeBdASlrQuZ6mIRG4gAulA9tgp31aOXFNgaiPH/s+ZDGcHk1NmAgD34g64xSUjiNU3mhrvThp1L69zO1fSO+jcK5my2q1QJXSkyO28iZ40WO7YjCOclzXdFxI0lWMho57RbGAiP7fc2eBMTT5LEmKGifymSWKFkgk2eQvaJ+ITdq+bXv2AhwzIERXm/8BPW80kxhE2kX3uoE/9eSu43RkYp9YnfknUABOurTGCTc='

'ssh_keyscan smithi089.front.sepia.ceph.com' reached maximum tries (5) after waiting for 5 seconds

SSH connection to smithi096 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --localize-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

Stale jobs detected, aborting.

{'Failure object was': {'smithi089.front.sepia.ceph.com': {'msg': 'non-zero return code', 'cmd': ['semodule', '-i', '/tmp/nrpe.pp'], 'stdout': '', 'stderr': 'libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).\\nsemodule: Failed on /tmp/nrpe.pp!', 'rc': 1, 'start': '2020-05-11 01:45:44.681198', 'end': '2020-05-11 01:45:49.688817', 'delta': '0:00:05.007619', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'semodule -i /tmp/nrpe.pp', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': 'None', 'chdir': 'None', 'executable': 'None', 'creates': 'None', 'removes': 'None', 'stdin': 'None'}}, 'stdout_lines': [], 'stderr_lines': ['libsemanage.semanage_get_lock: Could not get direct transaction lock at /var/lib/selinux/targeted/semanage.trans.LOCK. (Resource temporarily unavailable).', 'semodule: Failed on /tmp/nrpe.pp!'], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence(\'tag:yaml.org,2002:seq\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'semodule')"}

{'smithi089.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi089.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi089.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}, 'smithi125.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi125.front.sepia.ceph.com". Make sure this host can be reached over ssh: ssh: connect to host smithi125.front.sepia.ceph.com port 22: No route to host\r\n', 'unreachable': True}}

[Errno None] Unable to connect to port 22 on 172.21.15.83

Stale jobs detected, aborting.

psutil.NoSuchProcess process no longer exists (pid=15336)

{'smithi159.front.sepia.ceph.com': {'changed': False, 'msg': 'Data could not be sent to remote host "smithi159.front.sepia.ceph.com". Make sure this host can be reached over ssh: Warning: Permanently added \'smithi159.front.sepia.ceph.com,172.21.15.159\' (ECDSA) to the list of known hosts.\r\nubuntu@smithi159.front.sepia.ceph.com: Permission denied (publickey,password,keyboard-interactive).\r\n', 'unreachable': True}}

[Errno None] Unable to connect to port 22 on 172.21.15.39

SSH connection to smithi089 was lost: 'sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic'

SSH connection to smithi045 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --pool-snaps --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

SSH connection to smithi125 was lost: 'sudo yum -y install ceph-radosgw'

Command failed on smithi089 with status 100: 'sudo apt-get clean'

{'Failure object was': {'smithi125.front.sepia.ceph.com': {'msg': 'Failed to manage policy for boolean nagios_run_sudo: [Errno 11] Resource temporarily unavailable', 'invocation': {'module_args': {'name': 'nagios_run_sudo', 'state': True, 'persistent': True, 'ignore_selinux_state': False}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'nagios_run_sudo')"}

SSH connection to smithi148 was lost: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 -p unique_pool_2 bench 90 write'"

[Errno None] Unable to connect to port 22 on 172.21.15.159

SSH connection to smithi187 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 50 --op write_excl 0 --op append_excl 50 --pool unique_pool_0'

{'Failure object was': {'smithi089.front.sepia.ceph.com': {'msg': 'non-zero return code', 'cmd': ['rpm', '--rebuilddb'], 'stdout': '', 'stderr': "error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)", 'rc': 255, 'start': '2020-05-11 02:08:58.987411', 'end': '2020-05-11 02:08:58.995802', 'delta': '0:00:00.008391', 'changed': True, 'invocation': {'module_args': {'_raw_params': 'rpm --rebuilddb', 'warn': True, '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': 'None', 'chdir': 'None', 'executable': 'None', 'creates': 'None', 'removes': 'None', 'stdin': 'None'}}, 'warnings': ["Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."], 'stdout_lines': [], 'stderr_lines': ["error: can't create transaction lock on /var/lib/rpm/.rpm.lock (Resource temporarily unavailable)"], '_ansible_no_log': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 199, in represent_list return self.represent_sequence(\'tag:yaml.org,2002:seq\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 92, in represent_sequence node_item = self.represent_data(item) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'rpm')"}

{'Failure object was': {'smithi159.front.sepia.ceph.com': {'msg': 'Failed to manage policy for boolean nagios_run_sudo: [Errno 11] Resource temporarily unavailable', 'invocation': {'module_args': {'name': 'nagios_run_sudo', 'state': True, 'persistent': True, 'ignore_selinux_state': False}}, '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'nagios_run_sudo')"}

psutil.NoSuchProcess process no longer exists (pid=20209)

psutil.NoSuchProcess process no longer exists (pid=27729)

SSH connection to smithi179 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 400000 --objects 1024 --max-in-flight 64 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 600 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op write_excl 50 --pool unique_pool_0'

{'smithi159.front.sepia.ceph.com': {'changed': False, 'msg': 'Failed to connect to the host via ssh: ssh: connect to host smithi159.front.sepia.ceph.com port 22: No route to host', 'unreachable': True}}

Stale jobs detected, aborting.

{'Failure object was': {'smithi159.front.sepia.ceph.com': {'msg': 'Failed to lock apt for exclusive operation', 'invocation': {'module_args': {'dpkg_options': 'force-confdef,force-confold', 'autoremove': False, 'force': False, 'force_apt_get': False, 'policy_rc_d': 'None', 'package': 'None', 'autoclean': False, 'install_recommends': 'None', 'purge': False, 'allow_unauthenticated': False, 'state': 'present', 'upgrade': 'None', 'update_cache': True, 'default_release': 'None', 'only_upgrade': False, 'deb': 'None', 'cache_valid_time': 0}}, 'warnings': ['Could not find aptitude. Using apt-get instead'], '_ansible_no_log': False, 'changed': False}}, 'Traceback (most recent call last)': 'File "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/callback_plugins/failure_log.py", line 44, in log_failure log.error(yaml.safe_dump(failure)) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 306, in safe_dump return dump_all([data], stream, Dumper=SafeDumper, **kwds) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 207, in represent_dict return self.represent_mapping(\'tag:yaml.org,2002:map\', data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 118, in represent_mapping node_value = self.represent_data(item_value) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 58, in represent_data node = self.yaml_representers[None](self, data) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/lib/python3.6/site-packages/yaml/representer.py", line 231, in represent_undefined raise RepresenterError("cannot represent an object", data)', 'yaml.representer.RepresenterError': "('cannot represent an object', 'force-confdef,force-confold')"}

empty file

Test failure: setUpClass (tasks.mgr.dashboard.test_crush_rule.CrushRuleTest)

Stale jobs detected, aborting.

SSH connection to smithi089 was lost: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 -p unique_pool_0 bench 90 write'"

[Errno None] Unable to connect to port 22 on 172.21.15.13

Command failed on smithi060 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early quorum_status'

psutil.NoSuchProcess process no longer exists (pid=10773)

psutil.NoSuchProcess process no longer exists (pid=3146)

[Errno None] Unable to connect to port 22 on 172.21.15.42

SSH connection to smithi100 was lost: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --no-omap --ec-pool --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 0 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op setattr 25 --op rmattr 25 --op copy_from 50 --op append 100 --pool unique_pool_0'

psutil.NoSuchProcess process no longer exists (pid=10005)

SSH connection to smithi065 was lost: "/bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 8192 -O 65536 -p unique_pool_7 bench 90 write'"