Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version
fail 3308120 2018-12-05 05:05:21 2018-12-14 01:22:15 2018-12-14 07:52:20 6:30:05 0:08:31 6:21:34 ovh master rhel 7.5
Failure Reason:

Command failed on ovh029 with status 2: 'cd ~/ceph-ansible ; virtualenv --system-site-packages venv ; source venv/bin/activate ; pip install --upgrade pip ; pip install setuptools>=11.3 notario>=0.0.13 netaddr ansible==2.5 ; ANSIBLE_STDOUT_CALLBACK=debug ansible-playbook -vv -i inven.yml site.yml'

pass 3308121 2018-12-05 05:05:22 2018-12-14 01:22:15 2018-12-14 01:54:15 0:32:00 0:21:45 0:10:15 ovh master ubuntu
fail 3308122 2018-12-05 05:05:23 2018-12-14 01:22:15 2018-12-14 06:54:20 5:32:05 5:13:29 0:18:36 ovh master centos
Failure Reason:

rgw multisite test failures

fail 3308123 2018-12-05 05:05:24 2018-12-14 01:30:10 2018-12-14 01:54:10 0:24:00 0:09:36 0:14:24 ovh master ubuntu 18.04
Failure Reason:

{'ovh064.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 01:48:53.180374', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 01:48:53.176344', 'delta': '0:00:00.004030', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh073.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 01:48:58.371022', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 01:48:58.367767', 'delta': '0:00:00.003255', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

fail 3308124 2018-12-05 05:05:24 2018-12-14 01:30:10 2018-12-14 01:48:10 0:18:00 0:13:38 0:04:22 ovh master centos 7.4
Failure Reason:

Command failed on ovh080 with status 1: 'source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox==2.3.1'

pass 3308125 2018-12-05 05:05:25 2018-12-14 01:34:21 2018-12-14 02:30:21 0:56:00 0:41:48 0:14:12 ovh master ubuntu
pass 3308126 2018-12-05 05:05:26 2018-12-14 01:35:50 2018-12-14 02:39:50 1:04:00 0:51:05 0:12:55 ovh master centos
pass 3308127 2018-12-05 05:05:26 2018-12-14 01:38:18 2018-12-14 02:12:17 0:33:59 0:19:22 0:14:37 ovh master ubuntu
fail 3308128 2018-12-05 05:05:27 2018-12-14 01:40:02 2018-12-14 04:54:04 3:14:02 3:04:24 0:09:38 ovh master centos
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 3308129 2018-12-05 05:05:28 2018-12-14 01:42:05 2018-12-14 02:18:05 0:36:00 0:14:18 0:21:42 ovh master
pass 3308130 2018-12-05 05:05:29 2018-12-14 01:48:22 2018-12-14 02:46:22 0:58:00 0:48:28 0:09:32 ovh master centos
pass 3308131 2018-12-05 05:05:30 2018-12-14 01:54:11 2018-12-14 02:26:11 0:32:00 0:18:30 0:13:30 ovh master
pass 3308132 2018-12-05 05:05:30 2018-12-14 01:54:16 2018-12-14 05:06:18 3:12:02 2:56:44 0:15:18 ovh master centos
pass 3308133 2018-12-05 05:05:31 2018-12-14 01:57:53 2018-12-14 02:41:52 0:43:59 0:22:52 0:21:07 ovh master ubuntu
pass 3308134 2018-12-05 05:05:32 2018-12-14 02:00:21 2018-12-14 02:38:21 0:38:00 0:14:09 0:23:51 ovh master
pass 3308135 2018-12-05 05:05:33 2018-12-14 02:00:44 2018-12-14 03:06:44 1:06:00 0:46:06 0:19:54 ovh master centos
pass 3308136 2018-12-05 05:05:33 2018-12-14 02:02:16 2018-12-14 03:16:16 1:14:00 0:54:05 0:19:55 ovh master
pass 3308137 2018-12-05 05:05:34 2018-12-14 02:02:16 2018-12-14 05:18:18 3:16:02 2:38:55 0:37:07 ovh master centos
pass 3308138 2018-12-05 05:05:35 2018-12-14 02:05:23 2018-12-14 02:43:23 0:38:00 0:16:43 0:21:17 ovh master
pass 3308139 2018-12-05 05:05:36 2018-12-14 02:12:19 2018-12-14 03:08:19 0:56:00 0:46:32 0:09:28 ovh master centos
pass 3308140 2018-12-05 05:05:36 2018-12-14 02:12:20 2018-12-14 03:08:20 0:56:00 0:21:41 0:34:19 ovh master
pass 3308141 2018-12-05 05:05:37 2018-12-14 02:14:12 2018-12-14 02:58:11 0:43:59 0:16:21 0:27:38 ovh master ubuntu
pass 3308142 2018-12-05 05:05:38 2018-12-14 02:18:18 2018-12-14 05:38:20 3:20:02 3:01:08 0:18:54 ovh master centos
pass 3308143 2018-12-05 05:05:39 2018-12-14 03:20:29 2018-12-14 04:06:29 0:46:00 0:23:58 0:22:02 ovh master ubuntu
fail 3308144 2018-12-05 05:05:39 2018-12-14 03:36:25 2018-12-14 03:58:24 0:21:59 0:09:20 0:12:39 ovh master ubuntu 18.04
Failure Reason:

{'ovh090.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 03:51:46.134134', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 03:51:46.131703', 'delta': '0:00:00.002431', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh067.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 03:51:30.002104', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 03:51:29.998976', 'delta': '0:00:00.003128', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

pass 3308145 2018-12-05 05:05:40 2018-12-14 03:36:28 2018-12-14 05:04:29 1:28:01 0:45:28 0:42:33 ovh master centos
pass 3308146 2018-12-05 05:05:41 2018-12-14 03:42:27 2018-12-14 04:14:27 0:32:00 0:19:58 0:12:02 ovh master ubuntu
dead 3308147 2018-12-05 05:05:42 2018-12-14 03:54:18 2018-12-14 15:56:30 12:02:12 ovh master centos
pass 3308148 2018-12-05 05:05:43 2018-12-14 03:58:27 2018-12-14 04:28:27 0:30:00 0:15:05 0:14:55 ovh master
fail 3308149 2018-12-05 05:05:43 2018-12-14 04:02:00 2018-12-14 05:12:00 1:10:00 0:44:06 0:25:54 ovh master centos
Failure Reason:

Command failed on ovh053 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"

pass 3308150 2018-12-05 05:05:44 2018-12-14 04:04:45 2018-12-14 04:36:44 0:31:59 0:15:08 0:16:51 ovh master
pass 3308151 2018-12-05 05:05:45 2018-12-14 04:06:31 2018-12-14 04:48:30 0:41:59 0:18:01 0:23:58 ovh master
pass 3308152 2018-12-05 05:05:46 2018-12-14 04:14:28 2018-12-14 07:08:30 2:54:02 2:41:33 0:12:29 ovh master centos
pass 3308153 2018-12-05 05:05:47 2018-12-14 04:18:29 2018-12-14 04:44:28 0:25:59 0:14:16 0:11:43 ovh master
pass 3308154 2018-12-05 05:05:47 2018-12-14 04:22:12 2018-12-14 06:14:13 1:52:01 1:42:33 0:09:28 ovh master centos
pass 3308155 2018-12-05 05:05:48 2018-12-14 04:24:18 2018-12-14 05:40:18 1:16:00 0:55:17 0:20:43 ovh master
fail 3308156 2018-12-05 05:05:49 2018-12-14 04:28:37 2018-12-14 06:52:38 2:24:01 2:04:30 0:19:31 ovh master centos
Failure Reason:

Command failed on ovh048 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"

pass 3308157 2018-12-05 05:05:49 2018-12-14 04:36:46 2018-12-14 06:00:47 1:24:01 1:09:25 0:14:36 ovh master
pass 3308158 2018-12-05 05:05:50 2018-12-14 04:44:30 2018-12-14 05:12:29 0:27:59 0:16:48 0:11:11 ovh master
pass 3308159 2018-12-05 05:05:51 2018-12-14 04:46:34 2018-12-14 06:48:35 2:02:01 1:34:50 0:27:11 ovh master centos
dead 3308160 2018-12-05 05:05:52 2018-12-14 04:47:28 2018-12-14 16:49:41 12:02:13 ovh master ubuntu
pass 3308161 2018-12-05 05:05:52 2018-12-14 04:48:32 2018-12-14 07:10:34 2:22:02 2:05:45 0:16:17 ovh master centos
pass 3308162 2018-12-05 05:05:53 2018-12-14 04:48:48 2018-12-14 05:22:47 0:33:59 0:22:10 0:11:49 ovh master ubuntu
pass 3308163 2018-12-05 05:05:54 2018-12-14 04:54:16 2018-12-14 05:20:16 0:26:00 0:14:19 0:11:41 ovh master rhel 7.5
pass 3308164 2018-12-05 05:05:55 2018-12-14 04:54:16 2018-12-14 08:28:19 3:34:03 1:39:34 1:54:29 ovh master centos
fail 3308165 2018-12-05 05:05:55 2018-12-14 04:55:53 2018-12-14 05:19:53 0:24:00 0:14:32 0:09:28 ovh master ubuntu
Failure Reason:

Command failed on ovh083 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 802ee2380274f15db187ddd1219533c6b233da6a'

pass 3308166 2018-12-05 05:05:56 2018-12-14 08:46:55 2018-12-14 11:12:57 2:26:02 2:10:05 0:15:57 ovh master centos
pass 3308167 2018-12-05 05:05:57 2018-12-14 08:48:42 2018-12-14 09:34:42 0:46:00 0:24:53 0:21:07 ovh master
pass 3308168 2018-12-05 05:05:58 2018-12-14 08:48:49 2018-12-14 10:08:50 1:20:01 0:13:17 1:06:44 ovh master
pass 3308169 2018-12-05 05:05:58 2018-12-14 08:56:54 2018-12-14 11:30:55 2:34:01 1:40:23 0:53:38 ovh master centos
pass 3308170 2018-12-05 05:05:59 2018-12-14 08:58:30 2018-12-14 09:56:30 0:58:00 0:23:15 0:34:45 ovh master
pass 3308171 2018-12-05 05:06:00 2018-12-14 09:10:41 2018-12-14 11:36:42 2:26:01 1:59:45 0:26:16 ovh master centos
pass 3308172 2018-12-05 05:06:01 2018-12-14 09:12:45 2018-12-14 09:36:45 0:24:00 0:13:07 0:10:53 ovh master
pass 3308173 2018-12-05 05:06:01 2018-12-14 09:14:42 2018-12-14 10:58:43 1:44:01 1:36:23 0:07:38 ovh master centos
pass 3308174 2018-12-05 05:06:02 2018-12-14 09:14:42 2018-12-14 09:46:42 0:32:00 0:19:55 0:12:05 ovh master ubuntu
pass 3308175 2018-12-05 05:06:03 2018-12-14 09:16:47 2018-12-14 10:50:47 1:34:00 0:53:53 0:40:07 ovh master
pass 3308176 2018-12-05 05:06:04 2018-12-14 09:24:41 2018-12-14 11:48:42 2:24:01 1:57:28 0:26:33 ovh master centos
fail 3308177 2018-12-05 05:06:04 2018-12-14 09:24:44 2018-12-14 12:20:46 2:56:02 0:35:15 2:20:47 ovh master
Failure Reason:

"2018-12-14 11:58:57.345498 mon.a mon.0 158.69.67.103:6789/0 25 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN)" in cluster log

pass 3308178 2018-12-05 05:06:05 2018-12-14 09:24:48 2018-12-14 11:34:49 2:10:01 1:47:10 0:22:51 ovh master centos
pass 3308179 2018-12-05 05:06:06 2018-12-14 09:30:26 2018-12-14 10:04:25 0:33:59 0:17:04 0:16:55 ovh master ubuntu
pass 3308180 2018-12-05 05:06:06 2018-12-14 09:30:33 2018-12-14 11:32:34 2:02:01 1:52:50 0:09:11 ovh master centos
pass 3308181 2018-12-05 05:06:07 2018-12-14 09:31:57 2018-12-14 10:13:57 0:42:00 0:26:13 0:15:47 ovh master ubuntu
fail 3308182 2018-12-05 05:06:08 2018-12-14 09:32:33 2018-12-14 15:02:37 5:30:04 4:49:49 0:40:15 ovh master centos
Failure Reason:

rgw multisite test failures

pass 3308183 2018-12-05 05:06:09 2018-12-14 09:34:44 2018-12-14 10:08:43 0:33:59 0:16:12 0:17:47 ovh master ubuntu 16.04
pass 3308184 2018-12-05 05:06:09 2018-12-14 09:36:57 2018-12-14 10:48:57 1:12:00 0:47:52 0:24:08 ovh master ubuntu
pass 3308185 2018-12-05 05:06:10 2018-12-14 09:40:31 2018-12-14 11:10:33 1:30:02 1:20:48 0:09:14 ovh master centos
pass 3308186 2018-12-05 05:06:11 2018-12-14 09:40:32 2018-12-14 10:14:31 0:33:59 0:19:45 0:14:14 ovh master ubuntu
pass 3308187 2018-12-05 05:06:11 2018-12-14 09:40:33 2018-12-14 12:40:35 3:00:02 2:31:39 0:28:23 ovh master centos
pass 3308188 2018-12-05 05:06:12 2018-12-14 09:40:43 2018-12-14 10:20:42 0:39:59 0:17:55 0:22:04 ovh master
pass 3308189 2018-12-05 05:06:13 2018-12-14 09:46:55 2018-12-14 12:14:56 2:28:01 1:37:33 0:50:28 ovh master centos
pass 3308190 2018-12-05 05:06:14 2018-12-14 09:55:08 2018-12-14 10:41:08 0:46:00 0:18:24 0:27:36 ovh master
fail 3308191 2018-12-05 05:06:14 2018-12-14 09:56:43 2018-12-14 14:12:46 4:16:03 2:25:17 1:50:46 ovh master centos
Failure Reason:

'/home/ubuntu/cephtest/archive/syslog/kern.log:2018-12-14T11:59:06.323455+00:00 ovh009 kernel: NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [urlgrabber-ext-:28084] ' in syslog

pass 3308192 2018-12-05 05:06:15 2018-12-14 10:04:38 2018-12-14 11:30:38 1:26:00 0:44:12 0:41:48 ovh master ubuntu
pass 3308193 2018-12-05 05:06:16 2018-12-14 10:06:55 2018-12-14 11:28:55 1:22:00 0:28:49 0:53:11 ovh master
fail 3308194 2018-12-05 05:06:17 2018-12-14 10:08:56 2018-12-14 12:36:57 2:28:01 1:47:37 0:40:24 ovh master centos
Failure Reason:

"2018-12-14 11:59:08.884067 mon.a mon.0 158.69.66.121:6789/0 241 : cluster [WRN] Health check failed: 6 osds down (OSD_DOWN)" in cluster log

pass 3308195 2018-12-05 05:06:17 2018-12-14 10:08:56 2018-12-14 11:16:56 1:08:00 0:49:51 0:18:09 ovh master
fail 3308196 2018-12-05 05:06:18 2018-12-14 10:10:54 2018-12-14 12:22:55 2:12:01 2:03:20 0:08:41 ovh master centos
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 3308197 2018-12-05 05:06:19 2018-12-14 10:14:00 2018-12-14 11:04:00 0:50:00 0:28:36 0:21:24 ovh master
pass 3308198 2018-12-05 05:06:19 2018-12-14 10:14:43 2018-12-14 12:10:44 1:56:01 1:39:16 0:16:45 ovh master centos
pass 3308199 2018-12-05 05:06:20 2018-12-14 10:14:43 2018-12-14 10:48:43 0:34:00 0:21:12 0:12:48 ovh master
fail 3308200 2018-12-05 05:06:21 2018-12-14 10:14:59 2018-12-14 10:54:58 0:39:59 0:31:36 0:08:23 ovh master ubuntu
Failure Reason:

"2018-12-14 10:45:04.992190 mon.a mon.0 158.69.66.6:6789/0 247 : cluster [WRN] Health check failed: 7 slow ops, oldest one blocked for 34 sec, daemons [mon.a,mon.c] have slow ops. (SLOW_OPS)" in cluster log

pass 3308201 2018-12-05 05:06:22 2018-12-14 10:16:52 2018-12-14 12:44:54 2:28:02 2:11:01 0:17:01 ovh master centos
fail 3308202 2018-12-05 05:06:22 2018-12-14 10:18:04 2018-12-14 11:30:05 1:12:01 0:38:50 0:33:11 ovh master ubuntu
Failure Reason:

"2018-12-14 11:02:43.255250 mon.b mon.0 158.69.69.251:6789/0 145 : cluster [WRN] Health check failed: 35 slow ops, oldest one blocked for 50 sec, daemons [mon.a,mon.b,mon.c] have slow ops. (SLOW_OPS)" in cluster log

pass 3308203 2018-12-05 05:06:23 2018-12-14 10:18:34 2018-12-14 10:44:33 0:25:59 0:14:28 0:11:31 ovh master rhel 7.5
pass 3308204 2018-12-05 05:06:24 2018-12-14 10:20:55 2018-12-14 12:12:55 1:52:00 1:15:41 0:36:19 ovh master centos
pass 3308205 2018-12-05 05:06:25 2018-12-14 10:26:41 2018-12-14 11:06:41 0:40:00 0:19:21 0:20:39 ovh master ubuntu
pass 3308206 2018-12-05 05:06:25 2018-12-14 10:28:42 2018-12-14 13:16:43 2:48:01 2:08:05 0:39:56 ovh master centos
pass 3308207 2018-12-05 05:06:26 2018-12-14 10:41:11 2018-12-14 11:13:10 0:31:59 0:13:55 0:18:04 ovh master
pass 3308208 2018-12-05 05:06:27 2018-12-14 10:42:59 2018-12-14 12:14:59 1:32:00 1:16:15 0:15:45 ovh master centos
pass 3308209 2018-12-05 05:06:28 2018-12-14 10:44:46 2018-12-14 12:44:47 2:00:01 0:40:18 1:19:43 ovh master
pass 3308210 2018-12-05 05:06:28 2018-12-14 10:44:46 2018-12-14 12:22:46 1:38:00 0:39:20 0:58:40 ovh master
pass 3308211 2018-12-05 05:06:29 2018-12-14 10:48:56 2018-12-14 12:54:57 2:06:01 1:47:24 0:18:37 ovh master centos
pass 3308212 2018-12-05 05:06:30 2018-12-14 10:48:59 2018-12-14 12:02:59 1:14:00 0:12:48 1:01:12 ovh master
pass 3308213 2018-12-05 05:06:31 2018-12-14 10:51:00 2018-12-14 13:39:02 2:48:02 2:24:28 0:23:34 ovh master centos
pass 3308214 2018-12-05 05:06:31 2018-12-14 10:55:11 2018-12-14 12:11:11 1:16:00 0:50:29 0:25:31 ovh master
fail 3308215 2018-12-05 05:06:32 2018-12-14 10:58:56 2018-12-14 11:40:55 0:41:59 0:18:43 0:23:16 ovh master centos
Failure Reason:

{'ovh036.front.sepia.ceph.com': {'msg': "The conditional check 'polo.rc != 0' failed. The error was: error while evaluating conditional (polo.rc != 0): 'dict object' has no attribute 'rc'\n\nThe error appears to have been in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/set_python_path.yml': line 15, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n ignore_errors: true\n - name: Set ansible_python_interpreter=/usr/bin/python3\n ^ here\n"}}

pass 3308216 2018-12-05 05:06:33 2018-12-14 11:00:28 2018-12-14 12:56:29 1:56:01 1:15:00 0:41:01 ovh master
fail 3308217 2018-12-05 05:06:33 2018-12-14 11:04:02 2018-12-14 11:36:02 0:32:00 0:21:10 0:10:50 ovh master
Failure Reason:

{'ovh041.front.sepia.ceph.com': {'msg': "The conditional check 'polo.rc != 0' failed. The error was: error while evaluating conditional (polo.rc != 0): 'dict object' has no attribute 'rc'\n\nThe error appears to have been in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/set_python_path.yml': line 15, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n ignore_errors: true\n - name: Set ansible_python_interpreter=/usr/bin/python3\n ^ here\n"}}

fail 3308218 2018-12-05 05:06:34 2018-12-14 11:06:53 2018-12-14 14:14:55 3:08:02 2:32:03 0:35:59 ovh master centos
Failure Reason:

"2018-12-14 12:16:25.260706 mon.b mon.0 158.69.70.210:6789/0 10 : cluster [WRN] Health check failed: 1/3 mons down, quorum b,a (MON_DOWN)" in cluster log

pass 3308219 2018-12-05 05:06:35 2018-12-14 11:08:39 2018-12-14 12:38:40 1:30:01 0:42:20 0:47:41 ovh master ubuntu
pass 3308220 2018-12-05 05:06:36 2018-12-14 11:10:45 2018-12-14 12:52:46 1:42:01 1:06:13 0:35:48 ovh master centos
pass 3308221 2018-12-05 05:06:37 2018-12-14 11:13:07 2018-12-14 11:43:07 0:30:00 0:22:45 0:07:15 ovh master ubuntu
pass 3308222 2018-12-05 05:06:37 2018-12-14 11:13:11 2018-12-14 11:49:11 0:36:00 0:13:54 0:22:06 ovh master centos 7.4
pass 3308223 2018-12-05 05:06:38 2018-12-14 11:14:50 2018-12-14 13:44:51 2:30:01 2:09:38 0:20:23 ovh master centos
pass 3308224 2018-12-05 05:06:39 2018-12-14 11:16:58 2018-12-14 12:44:58 1:28:00 0:34:15 0:53:45 ovh master ubuntu
fail 3308225 2018-12-05 05:06:40 2018-12-14 11:24:22 2018-12-14 12:14:22 0:50:00 0:27:12 0:22:48 ovh master centos
Failure Reason:

{'ovh026.front.sepia.ceph.com': {'msg': "The conditional check 'polo.rc != 0' failed. The error was: error while evaluating conditional (polo.rc != 0): 'dict object' has no attribute 'rc'\n\nThe error appears to have been in '/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/set_python_path.yml': line 15, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n ignore_errors: true\n - name: Set ansible_python_interpreter=/usr/bin/python3\n ^ here\n"}}

pass 3308226 2018-12-05 05:06:40 2018-12-14 11:29:07 2018-12-14 12:31:12 1:02:05 0:38:54 0:23:11 ovh master
pass 3308227 2018-12-05 05:06:41 2018-12-14 11:30:06 2018-12-14 12:26:06 0:56:00 0:34:10 0:21:50 ovh master
fail 3308228 2018-12-05 05:06:42 2018-12-14 11:30:50 2018-12-14 14:20:52 2:50:02 2:20:47 0:29:15 ovh master centos
Failure Reason:

Command failed on ovh015 with status 1: 'sudo tar cz -f /tmp/tmpl379q1 -C /var/lib/ceph/mon/ceph-c -- .'

pass 3308229 2018-12-05 05:06:43 2018-12-14 11:30:56 2018-12-14 12:46:57 1:16:01 0:27:47 0:48:14 ovh master
pass 3308230 2018-12-05 05:06:44 2018-12-14 11:30:58 2018-12-14 13:21:04 1:50:06 1:22:42 0:27:24 ovh master centos
pass 3308231 2018-12-05 05:06:44 2018-12-14 11:32:46 2018-12-14 12:14:46 0:42:00 0:12:48 0:29:12 ovh master
pass 3308232 2018-12-05 05:06:45 2018-12-14 11:35:00 2018-12-14 14:03:02 2:28:02 2:10:44 0:17:18 ovh master centos
pass 3308233 2018-12-05 05:06:46 2018-12-14 11:36:14 2018-12-14 12:36:14 1:00:00 0:20:52 0:39:08 ovh master ubuntu
pass 3308234 2018-12-05 05:06:46 2018-12-14 11:36:43 2018-12-14 15:36:46 4:00:03 1:00:00 3:00:03 ovh master
pass 3308235 2018-12-05 05:06:47 2018-12-14 11:40:57 2018-12-14 14:14:59 2:34:02 1:22:15 1:11:47 ovh master centos
pass 3308236 2018-12-05 05:06:48 2018-12-14 11:43:21 2018-12-14 12:09:20 0:25:59 0:14:51 0:11:08 ovh master
pass 3308237 2018-12-05 05:06:49 2018-12-14 11:48:54 2018-12-14 14:34:56 2:46:02 2:10:59 0:35:03 ovh master centos
pass 3308238 2018-12-05 05:06:49 2018-12-14 11:49:12 2018-12-14 13:33:13 1:44:01 0:31:13 1:12:48 ovh master ubuntu
pass 3308239 2018-12-05 05:06:50 2018-12-14 11:53:08 2018-12-14 13:31:09 1:38:01 1:09:38 0:28:23 ovh master centos
fail 3308240 2018-12-05 05:06:51 2018-12-14 12:03:01 2018-12-14 13:19:01 1:16:00 0:15:25 1:00:35 ovh master centos 7.4
Failure Reason:

Command failed on ovh096 with status 2: 'cd ~/ceph-ansible ; virtualenv --system-site-packages venv ; source venv/bin/activate ; pip install --upgrade pip ; pip install setuptools>=11.3 notario>=0.0.13 netaddr ansible==2.5 ; ANSIBLE_STDOUT_CALLBACK=debug ansible-playbook -vv -i inven.yml site.yml'

pass 3308241 2018-12-05 05:06:51 2018-12-14 12:09:22 2018-12-14 12:37:22 0:28:00 0:21:00 0:07:00 ovh master ubuntu
fail 3308242 2018-12-05 05:06:52 2018-12-14 12:10:46 2018-12-14 21:10:54 9:00:08 8:14:15 0:45:53 ovh master centos
Failure Reason:

rgw multisite test failures

pass 3308243 2018-12-05 05:06:53 2018-12-14 12:11:12 2018-12-14 13:23:12 1:12:00 0:18:25 0:53:35 ovh master centos 7.4
fail 3308244 2018-12-05 05:06:54 2018-12-14 12:13:08 2018-12-14 13:15:08 1:02:00 0:10:29 0:51:31 ovh master centos 7.4
Failure Reason:

Command failed on ovh028 with status 1: 'source /home/ubuntu/cephtest/tox-venv/bin/activate && pip install tox==2.3.1'

pass 3308245 2018-12-05 05:06:54 2018-12-14 12:14:24 2018-12-14 12:56:24 0:42:00 0:25:52 0:16:08 ovh master ubuntu
pass 3308246 2018-12-05 05:06:55 2018-12-14 12:14:47 2018-12-14 14:22:48 2:08:01 1:28:27 0:39:34 ovh master centos
pass 3308247 2018-12-05 05:06:56 2018-12-14 12:14:57 2018-12-14 13:42:58 1:28:01 0:34:39 0:53:22 ovh master ubuntu
fail 3308248 2018-12-05 05:06:57 2018-12-14 12:15:01 2018-12-14 15:37:03 3:22:02 2:33:07 0:48:55 ovh master centos
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 3308249 2018-12-05 05:06:57 2018-12-14 12:20:49 2018-12-14 12:52:48 0:31:59 0:12:28 0:19:31 ovh master
pass 3308250 2018-12-05 05:06:58 2018-12-14 12:23:01 2018-12-14 14:51:02 2:28:01 1:29:40 0:58:21 ovh master centos
pass 3308251 2018-12-05 05:06:59 2018-12-14 12:23:01 2018-12-14 14:35:02 2:12:01 0:24:53 1:47:08 ovh master
pass 3308252 2018-12-05 05:07:00 2018-12-14 12:26:08 2018-12-14 14:54:09 2:28:01 2:05:22 0:22:39 ovh master centos
pass 3308253 2018-12-05 05:07:00 2018-12-14 12:31:14 2018-12-14 13:43:15 1:12:01 0:22:41 0:49:20 ovh master ubuntu
pass 3308254 2018-12-05 05:07:01 2018-12-14 12:36:16 2018-12-14 15:26:18 2:50:02 0:15:32 2:34:30 ovh master
pass 3308255 2018-12-05 05:07:02 2018-12-14 12:36:58 2018-12-14 14:14:59 1:38:01 1:18:05 0:19:56 ovh master centos
pass 3308256 2018-12-05 05:07:03 2018-12-14 12:37:23 2018-12-14 14:07:24 1:30:01 1:04:23 0:25:38 ovh master
pass 3308257 2018-12-05 05:07:04 2018-12-14 12:38:52 2018-12-14 15:10:54 2:32:02 2:15:30 0:16:32 ovh master centos
pass 3308258 2018-12-05 05:07:04 2018-12-14 12:40:48 2018-12-14 13:32:47 0:51:59 0:32:10 0:19:49 ovh master
pass 3308259 2018-12-05 05:07:05 2018-12-14 12:44:59 2018-12-14 14:39:00 1:54:01 1:11:15 0:42:46 ovh master centos
pass 3308260 2018-12-05 05:07:06 2018-12-14 12:44:59 2018-12-14 13:42:59 0:58:00 0:35:39 0:22:21 ovh master
pass 3308261 2018-12-05 05:07:07 2018-12-14 12:44:59 2018-12-14 13:34:59 0:50:00 0:16:47 0:33:13 ovh master ubuntu
pass 3308262 2018-12-05 05:07:08 2018-12-14 12:46:58 2018-12-14 16:47:01 4:00:03 2:45:13 1:14:50 ovh master centos
pass 3308263 2018-12-05 05:07:08 2018-12-14 12:47:09 2018-12-14 13:27:08 0:39:59 0:23:16 0:16:43 ovh master ubuntu
fail 3308264 2018-12-05 05:07:09 2018-12-14 12:53:00 2018-12-14 13:12:59 0:19:59 0:09:38 0:10:21 ovh master ubuntu 18.04
Failure Reason:

{'ovh073.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 13:10:52.556932', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 13:10:52.553375', 'delta': '0:00:00.003557', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh020.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 13:11:22.297278', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 13:11:22.293437', 'delta': '0:00:00.003841', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

fail 3308265 2018-12-05 05:07:10 2018-12-14 12:53:00 2018-12-14 14:41:01 1:48:01 1:09:51 0:38:10 ovh master centos
Failure Reason:

Command failed on ovh046 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"

pass 3308266 2018-12-05 05:07:11 2018-12-14 12:55:10 2018-12-14 14:21:12 1:26:02 0:31:14 0:54:48 ovh master ubuntu
pass 3308267 2018-12-05 05:07:11 2018-12-14 12:56:37 2018-12-14 15:56:38 3:00:01 2:09:36 0:50:25 ovh master centos
pass 3308268 2018-12-05 05:07:12 2018-12-14 12:56:37 2018-12-14 13:48:36 0:51:59 0:12:35 0:39:24 ovh master
pass 3308269 2018-12-05 05:07:13 2018-12-14 13:13:12 2018-12-14 15:13:13 2:00:01 1:13:34 0:46:27 ovh master centos
pass 3308270 2018-12-05 05:07:14 2018-12-14 13:15:14 2018-12-14 14:09:14 0:54:00 0:15:09 0:38:51 ovh master
pass 3308271 2018-12-05 05:07:14 2018-12-14 13:16:56 2018-12-14 14:30:57 1:14:01 0:18:15 0:55:46 ovh master
pass 3308272 2018-12-05 05:07:15 2018-12-14 13:19:14 2018-12-14 16:41:16 3:22:02 2:47:26 0:34:36 ovh master centos
pass 3308273 2018-12-05 05:07:16 2018-12-14 13:21:17 2018-12-14 14:03:17 0:42:00 0:13:09 0:28:51 ovh master
fail 3308274 2018-12-05 05:07:17 2018-12-14 13:23:14 2018-12-14 17:15:17 3:52:03 2:28:19 1:23:44 ovh master centos
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 3308275 2018-12-05 05:07:17 2018-12-14 13:27:20 2018-12-14 14:39:21 1:12:01 0:52:39 0:19:22 ovh master
pass 3308276 2018-12-05 05:07:18 2018-12-14 13:31:10 2018-12-14 14:55:11 1:24:01 1:14:45 0:09:16 ovh master centos
pass 3308277 2018-12-05 05:07:19 2018-12-14 13:32:59 2018-12-14 15:25:00 1:52:01 0:57:03 0:54:58 ovh master
pass 3308278 2018-12-05 05:07:19 2018-12-14 13:32:59 2018-12-14 14:12:59 0:40:00 0:21:09 0:18:51 ovh master
pass 3308279 2018-12-05 05:07:20 2018-12-14 13:33:14 2018-12-14 16:27:16 2:54:02 1:57:51 0:56:11 ovh master centos
pass 3308280 2018-12-05 05:07:21 2018-12-14 13:35:09 2018-12-14 14:03:09 0:28:00 0:16:31 0:11:29 ovh master ubuntu
pass 3308281 2018-12-05 05:07:22 2018-12-14 13:39:14 2018-12-14 15:25:15 1:46:01 1:26:51 0:19:10 ovh master centos
pass 3308282 2018-12-05 05:07:22 2018-12-14 13:43:10 2018-12-14 14:45:10 1:02:00 0:21:04 0:40:56 ovh master ubuntu
pass 3308283 2018-12-05 05:07:23 2018-12-14 13:43:10 2018-12-14 14:43:10 1:00:00 0:13:25 0:46:35 ovh master rhel 7.5
pass 3308284 2018-12-05 05:07:24 2018-12-14 13:43:16 2018-12-14 16:55:18 3:12:02 2:14:48 0:57:14 ovh master centos
pass 3308285 2018-12-05 05:07:25 2018-12-14 13:45:04 2018-12-14 14:47:04 1:02:00 0:28:12 0:33:48 ovh master ubuntu
pass 3308286 2018-12-05 05:07:25 2018-12-14 13:48:49 2018-12-14 16:22:50 2:34:01 1:31:53 1:02:08 ovh master centos
pass 3308287 2018-12-05 05:07:26 2018-12-14 14:03:15 2018-12-14 14:31:14 0:27:59 0:17:02 0:10:57 ovh master
pass 3308288 2018-12-05 05:07:27 2018-12-14 14:03:15 2018-12-14 15:07:15 1:04:00 0:21:01 0:42:59 ovh master
pass 3308289 2018-12-05 05:07:28 2018-12-14 14:03:18 2018-12-14 16:31:19 2:28:01 2:07:42 0:20:19 ovh master centos
pass 3308290 2018-12-05 05:07:28 2018-12-14 14:07:26 2018-12-14 14:55:26 0:48:00 0:24:35 0:23:25 ovh master
pass 3308291 2018-12-05 05:07:29 2018-12-14 14:09:20 2018-12-14 16:57:21 2:48:01 1:15:52 1:32:09 ovh master centos
pass 3308292 2018-12-05 05:07:30 2018-12-14 14:12:58 2018-12-14 15:08:58 0:56:00 0:12:20 0:43:40 ovh master
pass 3308293 2018-12-05 05:07:31 2018-12-14 14:13:00 2018-12-14 16:51:02 2:38:02 2:07:16 0:30:46 ovh master centos
pass 3308294 2018-12-05 05:07:32 2018-12-14 14:15:09 2018-12-14 15:59:10 1:44:01 0:21:18 1:22:43 ovh master ubuntu
pass 3308295 2018-12-05 05:07:32 2018-12-14 14:15:09 2018-12-14 15:57:10 1:42:01 0:53:52 0:48:09 ovh master
pass 3308296 2018-12-05 05:07:33 2018-12-14 14:15:09 2018-12-14 16:19:10 2:04:01 1:44:35 0:19:26 ovh master centos
pass 3308297 2018-12-05 05:07:34 2018-12-14 14:21:04 2018-12-14 14:53:04 0:32:00 0:22:43 0:09:17 ovh master
pass 3308298 2018-12-05 05:07:35 2018-12-14 14:21:13 2018-12-14 17:09:14 2:48:01 2:16:45 0:31:16 ovh master centos
pass 3308299 2018-12-05 05:07:36 2018-12-14 14:23:00 2018-12-14 14:59:00 0:36:00 0:17:02 0:18:58 ovh master ubuntu
pass 3308300 2018-12-05 05:07:37 2018-12-14 14:31:10 2018-12-14 16:49:11 2:18:01 1:17:02 1:00:59 ovh master centos
pass 3308301 2018-12-05 05:07:37 2018-12-14 14:31:15 2018-12-14 16:07:16 1:36:01 0:22:08 1:13:53 ovh master ubuntu
fail 3308302 2018-12-05 05:07:38 2018-12-14 14:35:08 2018-12-14 23:17:16 8:42:08 4:53:44 3:48:24 ovh master centos
Failure Reason:

rgw multisite test failures

pass 3308303 2018-12-05 05:07:39 2018-12-14 14:35:08 2018-12-14 16:53:09 2:18:01 0:14:10 2:03:51 ovh master ubuntu 16.04
pass 3308304 2018-12-05 05:07:40 2018-12-14 14:39:12 2018-12-14 15:19:12 0:40:00 0:27:41 0:12:19 ovh master ubuntu
fail 3308305 2018-12-05 05:07:40 2018-12-14 14:39:22 2018-12-14 16:39:23 2:00:01 1:32:14 0:27:47 ovh master centos
Failure Reason:

Command failed on ovh046 with status 2: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data --cluster ceph'

pass 3308306 2018-12-05 05:07:41 2018-12-14 14:41:13 2018-12-14 16:41:14 2:00:01 0:46:51 1:13:10 ovh master ubuntu
pass 3308307 2018-12-05 05:07:42 2018-12-14 14:43:23 2018-12-14 17:17:24 2:34:01 1:44:44 0:49:17 ovh master centos
pass 3308308 2018-12-05 05:07:43 2018-12-14 14:45:23 2018-12-14 15:43:23 0:58:00 0:13:37 0:44:23 ovh master
pass 3308309 2018-12-05 05:07:44 2018-12-14 14:47:16 2018-12-14 17:11:17 2:24:01 2:01:00 0:23:01 ovh master centos
pass 3308310 2018-12-05 05:07:45 2018-12-14 14:51:14 2018-12-14 15:41:14 0:50:00 0:18:10 0:31:50 ovh master
fail 3308311 2018-12-05 05:07:45 2018-12-14 14:53:16 2018-12-14 18:05:18 3:12:02 1:27:41 1:44:21 ovh master centos
Failure Reason:

Command failed (s3 tests against rgw) on ovh039 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616'"

pass 3308312 2018-12-05 05:07:46 2018-12-14 14:54:11 2018-12-14 15:32:10 0:37:59 0:23:03 0:14:56 ovh master ubuntu
pass 3308313 2018-12-05 05:07:47 2018-12-14 14:55:23 2018-12-14 16:29:23 1:34:00 0:29:53 1:04:07 ovh master
pass 3308314 2018-12-05 05:07:48 2018-12-14 14:55:27 2018-12-14 18:11:29 3:16:02 2:15:03 1:00:59 ovh master centos
pass 3308315 2018-12-05 05:07:48 2018-12-14 14:59:13 2018-12-14 17:07:15 2:08:02 1:17:42 0:50:20 ovh master
pass 3308316 2018-12-05 05:07:49 2018-12-14 15:02:50 2018-12-14 16:52:50 1:50:00 1:32:09 0:17:51 ovh master centos
pass 3308317 2018-12-05 05:07:50 2018-12-14 15:07:25 2018-12-14 16:05:25 0:58:00 0:14:40 0:43:20 ovh master
fail 3308318 2018-12-05 05:07:51 2018-12-14 15:09:11 2018-12-14 17:49:12 2:40:01 1:28:52 1:11:09 ovh master centos
Failure Reason:

Command failed (s3 tests against rgw) on ovh067 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616'"

pass 3308319 2018-12-05 05:07:51 2018-12-14 15:11:07 2018-12-14 16:51:07 1:40:00 0:43:38 0:56:22 ovh master
pass 3308320 2018-12-05 05:07:52 2018-12-14 15:13:15 2018-12-14 16:39:15 1:26:00 0:16:15 1:09:45 ovh master ubuntu
fail 3308321 2018-12-05 05:07:53 2018-12-14 15:19:24 2018-12-14 17:51:25 2:32:01 1:42:59 0:49:02 ovh master centos
Failure Reason:

Command failed (s3 tests against rgw) on ovh082 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616'"

pass 3308322 2018-12-05 05:07:54 2018-12-14 15:25:12 2018-12-14 18:13:14 2:48:02 0:31:14 2:16:48 ovh master ubuntu
pass 3308323 2018-12-05 05:07:55 2018-12-14 15:25:16 2018-12-14 16:41:17 1:16:01 0:33:38 0:42:23 ovh master ubuntu 16.04
pass 3308324 2018-12-05 05:07:55 2018-12-14 15:26:31 2018-12-14 18:30:33 3:04:02 1:58:03 1:05:59 ovh master centos
pass 3308325 2018-12-05 05:07:56 2018-12-14 15:32:12 2018-12-14 16:56:13 1:24:01 0:20:04 1:03:57 ovh master ubuntu
pass 3308326 2018-12-05 05:07:57 2018-12-14 15:36:59 2018-12-14 18:19:00 2:42:01 2:09:43 0:32:18 ovh master centos
pass 3308327 2018-12-05 05:07:58 2018-12-14 15:37:04 2018-12-14 16:49:04 1:12:00 0:13:27 0:58:33 ovh master
pass 3308328 2018-12-05 05:07:59 2018-12-14 15:41:27 2018-12-14 18:23:28 2:42:01 2:19:29 0:22:32 ovh master centos
pass 3308329 2018-12-05 05:07:59 2018-12-14 15:43:25 2018-12-14 16:49:25 1:06:00 0:36:26 0:29:34 ovh master
pass 3308330 2018-12-05 05:08:00 2018-12-14 15:49:11 2018-12-14 17:21:12 1:32:01 0:22:48 1:09:13 ovh master
pass 3308331 2018-12-05 05:08:01 2018-12-14 15:56:43 2018-12-14 19:38:45 3:42:02 1:28:36 2:13:26 ovh master centos
pass 3308332 2018-12-05 05:08:02 2018-12-14 15:56:43 2018-12-14 17:08:43 1:12:00 0:13:01 0:58:59 ovh master
pass 3308333 2018-12-05 05:08:03 2018-12-14 15:57:11 2018-12-14 19:27:13 3:30:02 3:00:07 0:29:55 ovh master centos
pass 3308334 2018-12-05 05:08:03 2018-12-14 15:59:22 2018-12-14 17:33:23 1:34:01 0:47:31 0:46:30 ovh master
pass 3308335 2018-12-05 05:08:04 2018-12-14 16:05:38 2018-12-14 18:09:39 2:04:01 0:49:12 1:14:49 ovh master centos
fail 3308336 2018-12-05 05:08:05 2018-12-14 16:07:28 2018-12-14 17:43:29 1:36:01 0:57:11 0:38:50 ovh master
Failure Reason:

Socket is closed

pass 3308337 2018-12-05 05:08:06 2018-12-14 16:19:23 2018-12-14 18:43:24 2:24:01 0:21:25 2:02:36 ovh master
pass 3308338 2018-12-05 05:08:07 2018-12-14 16:22:52 2018-12-14 20:25:01 4:02:09 2:56:13 1:05:56 ovh master centos
pass 3308339 2018-12-05 05:08:07 2018-12-14 16:27:28 2018-12-14 17:21:28 0:54:00 0:18:30 0:35:30 ovh master ubuntu
fail 3308340 2018-12-05 05:08:08 2018-12-14 16:29:36 2018-12-14 17:51:36 1:22:00 0:40:28 0:41:32 ovh master centos
Failure Reason:

Command failed (s3 tests against rgw) on ovh027 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw,!lifecycle_expiration,!fails_strict_rfc2616'"

pass 3308341 2018-12-05 05:08:09 2018-12-14 16:31:21 2018-12-14 17:23:21 0:52:00 0:27:52 0:24:08 ovh master ubuntu
fail 3308342 2018-12-05 05:08:10 2018-12-14 16:39:24 2018-12-14 17:03:23 0:23:59 0:09:48 0:14:11 ovh master ubuntu 18.04
Failure Reason:

{'ovh046.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 16:56:38.031396', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 16:56:38.027931', 'delta': '0:00:00.003465', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}, 'ovh092.front.sepia.ceph.com': {'_ansible_parsed': True, 'stderr_lines': ['/bin/sh: 1: ifdown: not found'], 'cmd': 'ifdown ens3 && ifup ens3', 'end': '2018-12-14 16:56:42.782909', '_ansible_no_log': False, 'stdout': '', 'changed': True, 'invocation': {'module_args': {'creates': None, 'executable': None, 'chdir': None, '_raw_params': 'ifdown ens3 && ifup ens3', 'removes': None, 'warn': True, '_uses_shell': True, 'stdin': None}}, 'start': '2018-12-14 16:56:42.779057', 'delta': '0:00:00.003852', 'stderr': '/bin/sh: 1: ifdown: not found', 'rc': 127, 'msg': 'non-zero return code', 'stdout_lines': []}}

pass 3308343 2018-12-05 05:08:11 2018-12-14 16:39:24 2018-12-14 19:45:26 3:06:02 2:52:33 0:13:29 ovh master centos
pass 3308344 2018-12-05 05:08:11 2018-12-14 16:41:27 2018-12-14 17:31:27 0:50:00 0:26:50 0:23:10 ovh master ubuntu
fail 3308345 2018-12-05 05:08:12 2018-12-14 16:41:27 2018-12-14 17:51:27 1:10:00 0:51:12 0:18:48 ovh master centos
Failure Reason:

"2018-12-14 17:25:57.819665 mon.b mon.0 158.69.65.129:6789/0 186 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 3308346 2018-12-05 05:08:13 2018-12-14 16:41:27 2018-12-14 17:19:27 0:38:00 0:17:44 0:20:16 ovh master
pass 3308347 2018-12-05 05:08:14 2018-12-14 16:47:13 2018-12-14 18:37:14 1:50:01 0:21:17 1:28:44 ovh master
fail 3308348 2018-12-05 05:08:14 2018-12-14 16:49:17 2018-12-14 20:07:19 3:18:02 3:00:07 0:17:55 ovh master centos
Failure Reason:

reached maximum tries (50) after waiting for 300 seconds

pass 3308349 2018-12-05 05:08:15 2018-12-14 16:49:17 2018-12-14 18:03:17 1:14:00 0:24:56 0:49:04 ovh master
fail 3308350 2018-12-05 05:08:16 2018-12-14 16:49:26 2018-12-14 17:51:26 1:02:00 0:48:17 0:13:43 ovh master centos
Failure Reason:

"2018-12-14 17:25:16.926892 mon.b mon.0 158.69.65.223:6789/0 204 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)" in cluster log

pass 3308351 2018-12-05 05:08:17 2018-12-14 16:49:42 2018-12-14 18:39:43 1:50:01 0:13:43 1:36:18 ovh master
pass 3308352 2018-12-05 05:08:17 2018-12-14 16:51:07 2018-12-14 19:55:09 3:04:02 2:50:19 0:13:43 ovh master centos
pass 3308353 2018-12-05 05:08:18 2018-12-14 17:08:57 2018-12-14 17:56:57 0:48:00 0:25:46 0:22:14 ovh master ubuntu
pass 3308354 2018-12-05 05:08:19 2018-12-14 17:09:16 2018-12-14 18:19:16 1:10:00 0:55:09 0:14:51 ovh master
pass 3308355 2018-12-05 05:08:20 2018-12-14 17:11:29 2018-12-14 18:33:30 1:22:01 0:47:03 0:34:58 ovh master centos
pass 3308356 2018-12-05 05:08:20 2018-12-14 17:15:29 2018-12-14 17:43:28 0:27:59 0:14:21 0:13:38 ovh master
pass 3308357 2018-12-05 05:08:21 2018-12-14 17:17:36 2018-12-14 20:57:38 3:40:02 2:57:59 0:42:03 ovh master centos
pass 3308358 2018-12-05 05:08:22 2018-12-14 17:19:35 2018-12-14 17:53:35 0:34:00 0:21:39 0:12:21 ovh master ubuntu
pass 3308359 2018-12-05 05:08:22 2018-12-14 17:21:23 2018-12-14 18:21:23 1:00:00 0:50:28 0:09:32 ovh master centos