Status Job ID Links Posted Started Updated
Runtime
Duration
In Waiting
Machine Teuthology Branch OS Type OS Version
pass 3448543 2019-01-11 17:29:39 2019-01-11 17:30:11 2019-01-11 18:12:11 0:42:00 0:24:03 0:17:57 smithi master
pass 3448544 2019-01-11 17:29:40 2019-01-11 17:33:48 2019-01-11 19:29:49 1:56:01 1:43:35 0:12:26 smithi master
pass 3448545 2019-01-11 17:29:41 2019-01-11 17:36:33 2019-01-11 18:20:33 0:44:00 0:19:26 0:24:34 smithi master
pass 3448546 2019-01-11 17:29:42 2019-01-11 17:36:33 2019-01-11 18:04:33 0:28:00 0:10:17 0:17:43 smithi master
pass 3448547 2019-01-11 17:29:43 2019-01-11 17:39:06 2019-01-11 18:25:06 0:46:00 0:24:04 0:21:56 smithi master
pass 3448548 2019-01-11 17:29:43 2019-01-11 17:42:55 2019-01-11 18:26:55 0:44:00 0:28:34 0:15:26 smithi master
pass 3448549 2019-01-11 17:29:44 2019-01-11 17:42:55 2019-01-11 18:18:55 0:36:00 0:17:58 0:18:02 smithi master
pass 3448550 2019-01-11 17:29:45 2019-01-11 17:45:09 2019-01-11 18:21:08 0:35:59 0:19:05 0:16:54 smithi master
pass 3448551 2019-01-11 17:29:46 2019-01-11 17:47:35 2019-01-11 18:27:35 0:40:00 0:23:20 0:16:40 smithi master
pass 3448552 2019-01-11 17:29:46 2019-01-11 17:50:53 2019-01-11 18:12:53 0:22:00 0:10:58 0:11:02 smithi master
pass 3448553 2019-01-11 17:29:47 2019-01-11 17:50:54 2019-01-11 18:22:53 0:31:59 0:19:07 0:12:52 smithi master
dead 3448554 2019-01-11 17:29:48 2019-01-11 17:52:38 2019-01-12 05:55:04 12:02:26 smithi master
pass 3448555 2019-01-11 17:29:49 2019-01-11 17:52:57 2019-01-11 18:36:56 0:43:59 0:28:27 0:15:32 smithi master ubuntu 14.04
pass 3448556 2019-01-11 17:29:49 2019-01-11 17:53:09 2019-01-11 18:23:09 0:30:00 0:10:19 0:19:41 smithi master
pass 3448557 2019-01-11 17:29:50 2019-01-11 17:53:38 2019-01-11 18:15:37 0:21:59 0:10:58 0:11:01 smithi master
pass 3448558 2019-01-11 17:29:51 2019-01-11 17:55:09 2019-01-11 18:33:09 0:38:00 0:22:45 0:15:15 smithi master
pass 3448559 2019-01-11 17:29:52 2019-01-11 17:56:49 2019-01-11 18:26:48 0:29:59 0:17:21 0:12:38 smithi master
pass 3448560 2019-01-11 17:29:53 2019-01-11 17:56:56 2019-01-11 18:34:56 0:38:00 0:27:07 0:10:53 smithi master
pass 3448561 2019-01-11 17:29:53 2019-01-11 18:00:37 2019-01-11 18:36:37 0:36:00 0:10:36 0:25:24 smithi master
pass 3448562 2019-01-11 17:29:54 2019-01-11 18:00:37 2019-01-11 18:34:37 0:34:00 0:09:21 0:24:39 smithi master
fail 3448563 2019-01-11 17:29:55 2019-01-11 18:00:46 2019-01-11 18:32:45 0:31:59 0:21:05 0:10:54 smithi master
Failure Reason:

"2019-01-11 18:21:08.075496 osd.1 osd.1 172.21.15.197:6801/13303 283 : cluster [WRN] slow request 30.332886 seconds old, received at 2019-01-11 18:20:37.742584: osd_op(client.5585.0:3 180.6 180:602f83fe:::foo:head [writefull 0~8] snapc 0=[] ondisk+write+known_if_redirected e358) currently waiting for blocked object" in cluster log

pass 3448564 2019-01-11 17:29:56 2019-01-11 18:01:00 2019-01-11 18:33:00 0:32:00 0:19:32 0:12:28 smithi master
dead 3448565 2019-01-11 17:29:56 2019-01-11 18:01:18 2019-01-11 18:33:18 0:32:00 smithi master
pass 3448566 2019-01-11 17:29:57 2019-01-11 18:01:20 2019-01-11 18:33:19 0:31:59 0:20:07 0:11:52 smithi master
pass 3448567 2019-01-11 17:29:58 2019-01-11 18:04:45 2019-01-11 18:40:45 0:36:00 0:19:30 0:16:30 smithi master
pass 3448568 2019-01-11 17:29:59 2019-01-11 18:12:27 2019-01-11 18:50:27 0:38:00 0:24:02 0:13:58 smithi master
pass 3448569 2019-01-11 17:29:59 2019-01-11 18:12:54 2019-01-11 18:38:54 0:26:00 0:09:21 0:16:39 smithi master
pass 3448570 2019-01-11 17:30:00 2019-01-11 18:13:10 2019-01-11 18:45:09 0:31:59 0:24:09 0:07:50 smithi master
pass 3448571 2019-01-11 17:30:01 2019-01-11 18:15:49 2019-01-11 18:39:48 0:23:59 0:12:46 0:11:13 smithi master
pass 3448572 2019-01-11 17:30:02 2019-01-11 18:17:08 2019-01-11 18:41:07 0:23:59 0:11:05 0:12:54 smithi master
pass 3448573 2019-01-11 17:30:03 2019-01-11 18:17:10 2019-01-11 18:43:09 0:25:59 0:13:16 0:12:43 smithi master
pass 3448574 2019-01-11 17:30:03 2019-01-11 18:19:07 2019-01-11 18:51:06 0:31:59 0:20:42 0:11:17 smithi master
pass 3448575 2019-01-11 17:30:04 2019-01-11 18:19:07 2019-01-11 18:37:06 0:17:59 0:07:21 0:10:38 smithi master
pass 3448576 2019-01-11 17:30:05 2019-01-11 18:39:42 2019-01-11 19:15:41 0:35:59 0:12:44 0:23:15 smithi master
pass 3448577 2019-01-11 17:30:06 2019-01-11 18:39:43 2019-01-11 19:37:42 0:57:59 0:19:12 0:38:47 smithi master
pass 3448578 2019-01-11 17:30:06 2019-01-11 18:39:43 2019-01-11 19:11:41 0:31:58 0:12:28 0:19:30 smithi master
pass 3448579 2019-01-11 17:30:07 2019-01-11 18:39:44 2019-01-11 19:25:42 0:45:58 0:20:15 0:25:43 smithi master
pass 3448580 2019-01-11 17:30:08 2019-01-11 18:39:43 2019-01-11 19:21:41 0:41:58 0:21:29 0:20:29 smithi master
pass 3448581 2019-01-11 17:30:09 2019-01-11 18:39:42 2019-01-11 19:09:41 0:29:59 0:09:18 0:20:41 smithi master
pass 3448582 2019-01-11 17:30:10 2019-01-11 18:39:42 2019-01-11 19:15:41 0:35:59 0:24:24 0:11:35 smithi master
pass 3448583 2019-01-11 17:30:10 2019-01-11 18:39:44 2019-01-11 19:33:42 0:53:58 0:12:15 0:41:43 smithi master
pass 3448584 2019-01-11 17:30:11 2019-01-11 18:39:44 2019-01-11 19:07:42 0:27:58 0:06:14 0:21:44 smithi master
pass 3448585 2019-01-11 17:30:12 2019-01-11 18:39:43 2019-01-11 19:35:42 0:55:59 0:19:03 0:36:56 smithi master
pass 3448586 2019-01-11 17:30:13 2019-01-11 18:39:41 2019-01-11 18:55:40 0:15:59 0:06:37 0:09:22 smithi master
pass 3448587 2019-01-11 17:30:13 2019-01-11 18:39:42 2019-01-11 19:37:41 0:57:59 0:42:36 0:15:23 smithi master
pass 3448588 2019-01-11 17:30:14 2019-01-11 18:39:43 2019-01-11 19:19:41 0:39:58 0:20:09 0:19:49 smithi master
pass 3448589 2019-01-11 17:30:15 2019-01-11 18:39:44 2019-01-11 19:43:43 1:03:59 0:20:56 0:43:03 smithi master
pass 3448590 2019-01-11 17:30:16 2019-01-11 18:39:43 2019-01-11 19:35:42 0:55:59 0:14:00 0:41:59 smithi master
pass 3448591 2019-01-11 17:30:16 2019-01-11 18:39:43 2019-01-11 19:21:41 0:41:58 0:10:08 0:31:50 smithi master
pass 3448592 2019-01-11 17:30:17 2019-01-11 18:39:44 2019-01-11 19:11:42 0:31:58 0:09:14 0:22:44 smithi master
pass 3448593 2019-01-11 17:30:18 2019-01-11 18:39:45 2019-01-11 20:03:45 1:24:00 0:14:07 1:09:53 smithi master
pass 3448594 2019-01-11 17:30:19 2019-01-11 18:39:45 2019-01-12 02:31:52 7:52:07 7:27:24 0:24:43 smithi master
pass 3448595 2019-01-11 17:30:19 2019-01-11 18:39:45 2019-01-11 19:19:45 0:40:00 0:11:31 0:28:29 smithi master
pass 3448596 2019-01-11 17:30:20 2019-01-11 18:39:45 2019-01-11 19:57:45 1:18:00 0:38:27 0:39:33 smithi master
pass 3448597 2019-01-11 17:30:21 2019-01-11 18:39:49 2019-01-11 19:17:49 0:38:00 0:27:14 0:10:46 smithi master ubuntu 16.04
pass 3448598 2019-01-11 17:30:22 2019-01-11 18:40:57 2019-01-11 19:02:56 0:21:59 0:11:09 0:10:50 smithi master
fail 3448599 2019-01-11 17:30:22 2019-01-11 18:41:08 2019-01-11 19:37:08 0:56:00 0:40:34 0:15:26 smithi master centos
Failure Reason:

"2019-01-11 19:20:43.153127 osd.7 osd.7 172.21.15.62:6805/34054 127 : cluster [WRN] slow request 30.332664 seconds old, received at 2019-01-11 19:20:12.806922: osd_op(client.4407.0:1096 46.1 46.d592ddb1 (undecoded) ondisk+write+known_if_redirected e297) currently queued_for_pg" in cluster log

pass 3448600 2019-01-11 17:30:23 2019-01-11 18:43:21 2019-01-11 19:27:21 0:44:00 0:27:12 0:16:48 smithi master
pass 3448601 2019-01-11 17:30:24 2019-01-11 18:45:21 2019-01-11 19:05:20 0:19:59 0:10:28 0:09:31 smithi master
pass 3448602 2019-01-11 17:30:25 2019-01-11 18:46:15 2019-01-11 19:46:15 1:00:00 0:16:16 0:43:44 smithi master
pass 3448603 2019-01-11 17:30:25 2019-01-11 18:50:39 2019-01-11 19:24:38 0:33:59 0:23:28 0:10:31 smithi master
fail 3448604 2019-01-11 17:30:26 2019-01-11 18:50:59 2019-01-11 19:36:58 0:45:59 0:25:30 0:20:29 smithi master
Failure Reason:

"2019-01-11 19:18:47.790833 osd.4 osd.4 172.21.15.166:6801/12891 138 : cluster [WRN] slow request 32.444895 seconds old, received at 2019-01-11 19:18:15.345879: osd_op(client.4222.0:2524 2.28s1 2.3a04b068 (undecoded) ondisk+retry+read+rwordered+known_if_redirected e236) currently waiting for peered" in cluster log

pass 3448605 2019-01-11 17:30:27 2019-01-11 18:51:08 2019-01-11 19:17:07 0:25:59 0:07:34 0:18:25 smithi master
pass 3448606 2019-01-11 17:30:28 2019-01-11 18:51:19 2019-01-11 19:23:18 0:31:59 0:19:53 0:12:06 smithi master
fail 3448607 2019-01-11 17:30:28 2019-01-11 18:54:16 2019-01-11 19:30:16 0:36:00 0:10:13 0:25:47 smithi master
Failure Reason:

"2019-01-11 19:24:44.202073 osd.1 osd.1 172.21.15.129:6800/13040 64 : cluster [WRN] slow request 31.600418 seconds old, received at 2019-01-11 19:24:12.601584: osd_op(client.4262.0:83 3.0 3:17df6b18:::smithi12915966-21 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [setxattr _header (357),truncate 2336140] snapc 0=[] ondisk+write+known_if_redirected e28) currently waiting for blocked object" in cluster log

pass 3448608 2019-01-11 17:30:29 2019-01-11 18:55:52 2019-01-11 19:33:52 0:38:00 0:24:16 0:13:44 smithi master
pass 3448609 2019-01-11 17:30:30 2019-01-11 18:57:09 2019-01-11 19:19:09 0:22:00 0:09:35 0:12:25 smithi master
pass 3448610 2019-01-11 17:30:30 2019-01-11 19:00:48 2019-01-11 19:24:47 0:23:59 0:09:53 0:14:06 smithi master
pass 3448611 2019-01-11 17:30:31 2019-01-11 19:02:30 2019-01-11 19:32:30 0:30:00 0:13:48 0:16:12 smithi master
pass 3448612 2019-01-11 17:30:32 2019-01-11 19:02:57 2019-01-11 19:38:57 0:36:00 0:17:56 0:18:04 smithi master
pass 3448613 2019-01-11 17:30:33 2019-01-11 19:02:57 2019-01-11 19:56:57 0:54:00 0:37:42 0:16:18 smithi master
pass 3448614 2019-01-11 17:30:33 2019-01-11 19:03:52 2019-01-11 19:19:52 0:16:00 0:07:13 0:08:47 smithi master
pass 3448615 2019-01-11 17:30:34 2019-01-11 19:05:31 2019-01-11 20:13:31 1:08:00 0:40:53 0:27:07 smithi master
pass 3448616 2019-01-11 17:30:35 2019-01-11 19:07:54 2019-01-11 19:25:53 0:17:59 0:08:35 0:09:24 smithi master
fail 3448617 2019-01-11 17:30:36 2019-01-11 19:09:54 2019-01-11 19:41:53 0:31:59 0:02:46 0:29:13 smithi master
Failure Reason:

500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=22899e551b699eb624418e679c3da589b8a8fb5b

pass 3448618 2019-01-11 17:30:36 2019-01-11 19:09:58 2019-01-11 19:47:58 0:38:00 0:22:13 0:15:47 smithi master
pass 3448619 2019-01-11 17:30:37 2019-01-11 19:11:53 2019-01-11 19:39:52 0:27:59 0:14:37 0:13:22 smithi master
pass 3448620 2019-01-11 17:30:38 2019-01-11 19:11:53 2019-01-11 19:35:53 0:24:00 0:12:45 0:11:15 smithi master
fail 3448621 2019-01-11 17:30:38 2019-01-11 19:11:53 2019-01-11 19:43:53 0:32:00 0:18:57 0:13:03 smithi master
Failure Reason:

"2019-01-11 19:33:31.824601 osd.2 osd.2 172.21.15.194:6813/13231 189 : cluster [WRN] slow request 30.554817 seconds old, received at 2019-01-11 19:33:01.269751: osd_op(client.4265.0:39054 3.3 3:f8d4c127:::smithi19415645-1443 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [read 0~2235694,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e165) currently waiting for blocked object" in cluster log

pass 3448622 2019-01-11 17:30:39 2019-01-11 19:13:07 2019-01-11 19:49:07 0:36:00 0:24:36 0:11:24 smithi master
pass 3448623 2019-01-11 17:30:40 2019-01-11 19:15:10 2019-01-11 19:41:09 0:25:59 0:11:51 0:14:08 smithi master
pass 3448624 2019-01-11 17:30:41 2019-01-11 19:15:42 2019-01-11 19:55:42 0:40:00 0:24:09 0:15:51 smithi master
fail 3448625 2019-01-11 17:30:41 2019-01-11 19:15:42 2019-01-11 19:49:42 0:34:00 0:21:57 0:12:03 smithi master
Failure Reason:

"2019-01-11 19:39:59.838337 osd.0 osd.0 172.21.15.110:6805/36508 50 : cluster [WRN] slow request 30.295919 seconds old, received at 2019-01-11 19:39:29.542340: osd_op(client.4266.0:7731 3.3 3:f12780ab:::smithi11015377-453 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [copy-from ver 2220] snapc 0=[] ondisk+write+known_if_redirected e452) currently reached_pg" in cluster log

pass 3448626 2019-01-11 17:30:42 2019-01-11 19:17:13 2019-01-11 19:49:13 0:32:00 0:17:11 0:14:49 smithi master
pass 3448627 2019-01-11 17:30:43 2019-01-11 19:17:13 2019-01-12 02:45:20 7:28:07 7:19:22 0:08:45 smithi master
pass 3448628 2019-01-11 17:30:44 2019-01-11 19:17:50 2019-01-11 19:41:50 0:24:00 0:10:26 0:13:34 smithi master
pass 3448629 2019-01-11 17:30:45 2019-01-11 19:19:21 2019-01-11 19:57:21 0:38:00 0:25:09 0:12:51 smithi master
pass 3448630 2019-01-11 17:30:45 2019-01-11 19:19:42 2019-01-11 19:33:42 0:14:00 0:05:09 0:08:51 smithi master
pass 3448631 2019-01-11 17:30:46 2019-01-11 19:19:46 2019-01-11 19:57:45 0:37:59 0:26:28 0:11:31 smithi master
pass 3448632 2019-01-11 17:30:47 2019-01-11 19:19:53 2019-01-11 19:41:52 0:21:59 0:08:23 0:13:36 smithi master
pass 3448633 2019-01-11 17:30:47 2019-01-11 19:21:54 2019-01-11 19:51:53 0:29:59 0:18:31 0:11:28 smithi master
pass 3448634 2019-01-11 17:30:48 2019-01-11 19:21:54 2019-01-11 19:37:53 0:15:59 0:07:10 0:08:49 smithi master
fail 3448635 2019-01-11 17:30:49 2019-01-11 19:23:31 2019-01-11 19:39:30 0:15:59 0:03:52 0:12:07 smithi master
Failure Reason:

500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=22899e551b699eb624418e679c3da589b8a8fb5b

pass 3448636 2019-01-11 17:30:50 2019-01-11 19:24:20 2019-01-11 19:50:20 0:26:00 0:16:54 0:09:06 smithi master
fail 3448637 2019-01-11 17:30:50 2019-01-11 19:24:51 2019-01-11 19:38:50 0:13:59 0:02:45 0:11:14 smithi master
Failure Reason:

500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=22899e551b699eb624418e679c3da589b8a8fb5b

pass 3448638 2019-01-11 17:30:51 2019-01-11 19:24:51 2019-01-11 19:58:51 0:34:00 0:25:19 0:08:41 smithi master
fail 3448639 2019-01-11 17:30:52 2019-01-11 19:25:44 2019-01-11 19:39:43 0:13:59 0:02:36 0:11:23 smithi master
Failure Reason:

500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=22899e551b699eb624418e679c3da589b8a8fb5b

pass 3448640 2019-01-11 17:30:53 2019-01-11 19:26:06 2019-01-11 20:00:05 0:33:59 0:16:08 0:17:51 smithi master
pass 3448641 2019-01-11 17:30:53 2019-01-11 19:27:32 2019-01-11 19:57:32 0:30:00 0:15:08 0:14:52 smithi master
fail 3448642 2019-01-11 17:30:54 2019-01-11 19:29:09 2019-01-11 20:09:09 0:40:00 0:26:55 0:13:05 smithi master
Failure Reason:

"2019-01-11 19:58:29.572605 osd.3 osd.3 172.21.15.12:6801/13162 467 : cluster [WRN] slow request 30.880354 seconds old, received at 2019-01-11 19:57:58.692220: osd_op(client.4220.0:7472 2.7s0 2.6ecaec87 (undecoded) ondisk+retry+write+known_if_redirected e682) currently waiting for peered" in cluster log

pass 3448643 2019-01-11 17:30:55 2019-01-11 19:29:50 2019-01-11 20:33:50 1:04:00 0:25:24 0:38:36 smithi master
pass 3448644 2019-01-11 17:30:55 2019-01-11 19:30:28 2019-01-11 20:16:28 0:46:00 0:29:36 0:16:24 smithi master centos 7.4
fail 3448645 2019-01-11 17:30:56 2019-01-11 19:32:42 2019-01-11 19:46:41 0:13:59 0:02:41 0:11:18 smithi master
Failure Reason:

500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=22899e551b699eb624418e679c3da589b8a8fb5b

fail 3448646 2019-01-11 17:30:57 2019-01-11 19:33:55 2019-01-11 20:01:54 0:27:59 0:16:41 0:11:18 smithi master
Failure Reason:

"2019-01-11 19:52:28.164578 osd.7 osd.7 172.21.15.134:6804/12977 109 : cluster [WRN] slow request 32.453523 seconds old, received at 2019-01-11 19:51:55.711024: osd_op(osd.6.13:5933 2.1s0 2:a385c949:::smithi14815885-3053 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [copy-get max 8388608] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e107) currently waiting for blocked object" in cluster log

pass 3448647 2019-01-11 17:30:58 2019-01-11 19:33:58 2019-01-11 19:55:57 0:21:59 0:12:28 0:09:31 smithi master
pass 3448648 2019-01-11 17:30:59 2019-01-11 19:33:57 2019-01-11 19:53:57 0:20:00 0:09:19 0:10:41 smithi master
pass 3448649 2019-01-11 17:30:59 2019-01-11 19:35:53 2019-01-11 19:55:53 0:20:00 0:08:41 0:11:19 smithi master
pass 3448650 2019-01-11 17:31:00 2019-01-11 19:35:53 2019-01-11 20:07:53 0:32:00 0:21:37 0:10:23 smithi master
pass 3448651 2019-01-11 17:31:01 2019-01-11 19:35:54 2019-01-11 19:55:53 0:19:59 0:10:47 0:09:12 smithi master
pass 3448652 2019-01-11 17:31:02 2019-01-11 19:37:10 2019-01-11 20:09:09 0:31:59 0:20:05 0:11:54 smithi master
fail 3448653 2019-01-11 17:31:02 2019-01-11 19:37:10 2019-01-11 20:13:09 0:35:59 0:25:24 0:10:35 smithi master
Failure Reason:

"2019-01-11 20:06:18.331746 osd.6 osd.6 172.21.15.159:6808/15031 610 : cluster [WRN] slow request 30.741623 seconds old, received at 2019-01-11 20:05:47.590083: osd_op(client.4265.0:9539 3.1 3:bb41159e:::smithi05816173-400:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e708) currently reached_pg" in cluster log

pass 3448654 2019-01-11 17:31:03 2019-01-11 19:37:42 2019-01-11 20:13:42 0:36:00 0:27:12 0:08:48 smithi master
pass 3448655 2019-01-11 17:31:04 2019-01-11 19:37:43 2019-01-11 20:09:43 0:32:00 0:18:30 0:13:30 smithi master
pass 3448656 2019-01-11 17:31:05 2019-01-11 19:37:54 2019-01-11 19:59:54 0:22:00 0:11:29 0:10:31 smithi master
pass 3448657 2019-01-11 17:31:05 2019-01-11 19:39:02 2019-01-11 20:03:01 0:23:59 0:08:47 0:15:12 smithi master
pass 3448658 2019-01-11 17:31:06 2019-01-11 19:39:02 2019-01-11 20:05:01 0:25:59 0:09:49 0:16:10 smithi master
pass 3448659 2019-01-11 17:31:07 2019-01-11 19:39:31 2019-01-11 20:13:31 0:34:00 0:21:35 0:12:25 smithi master
pass 3448660 2019-01-11 17:31:08 2019-01-11 19:39:44 2019-01-11 20:25:44 0:46:00 0:27:38 0:18:22 smithi master
pass 3448661 2019-01-11 17:31:08 2019-01-11 19:39:54 2019-01-11 20:09:53 0:29:59 0:19:24 0:10:35 smithi master
fail 3448662 2019-01-11 17:31:09 2019-01-11 19:41:21 2019-01-11 19:57:20 0:15:59 0:06:35 0:09:24 smithi master
Failure Reason:

Command failed (workunit test rados/test_librados_build.sh) on smithi073 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=22899e551b699eb624418e679c3da589b8a8fb5b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_librados_build.sh'

pass 3448663 2019-01-11 17:31:10 2019-01-11 19:41:51 2019-01-11 20:11:50 0:29:59 0:21:17 0:08:42 smithi master
pass 3448664 2019-01-11 17:31:11 2019-01-11 19:41:53 2019-01-11 19:55:53 0:14:00 0:06:14 0:07:46 smithi master
pass 3448665 2019-01-11 17:31:11 2019-01-11 19:41:54 2019-01-11 20:17:54 0:36:00 0:20:59 0:15:01 smithi master
pass 3448666 2019-01-11 17:31:12 2019-01-11 19:43:55 2019-01-11 20:23:54 0:39:59 0:24:56 0:15:03 smithi master
pass 3448667 2019-01-11 17:31:13 2019-01-11 19:43:55 2019-01-11 20:03:54 0:19:59 0:05:20 0:14:39 smithi master
pass 3448668 2019-01-11 17:31:14 2019-01-11 19:46:27 2019-01-11 20:10:27 0:24:00 0:11:42 0:12:18 smithi master
fail 3448669 2019-01-11 17:31:14 2019-01-11 19:46:42 2019-01-11 23:12:45 3:26:03 3:07:08 0:18:55 smithi master
Failure Reason:

Command failed (workunit test mgr/test_localpool.sh) on smithi173 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=22899e551b699eb624418e679c3da589b8a8fb5b TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/mgr/test_localpool.sh'

pass 3448670 2019-01-11 17:31:15 2019-01-11 19:46:48 2019-01-11 20:24:48 0:38:00 0:09:22 0:28:38 smithi master
pass 3448671 2019-01-11 17:31:16 2019-01-11 19:48:10 2019-01-11 20:12:09 0:23:59 0:10:01 0:13:58 smithi master
fail 3448672 2019-01-11 17:31:17 2019-01-11 19:49:08 2019-01-11 20:03:08 0:14:00 0:02:49 0:11:11 smithi master
Failure Reason:

500 Server Error: Internal Server Error for url: https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F16.04%2Fx86_64&sha1=22899e551b699eb624418e679c3da589b8a8fb5b

pass 3448673 2019-01-11 17:31:17 2019-01-11 19:49:24 2019-01-11 20:17:24 0:28:00 0:14:52 0:13:08 smithi master
pass 3448674 2019-01-11 17:31:18 2019-01-11 19:49:43 2019-01-11 20:17:42 0:27:59 0:11:29 0:16:30 smithi master
fail 3448675 2019-01-11 17:31:19 2019-01-11 19:50:21 2019-01-11 20:30:21 0:40:00 0:24:18 0:15:42 smithi master
Failure Reason:

"2019-01-11 20:16:49.791863 osd.5 osd.5 172.21.15.102:6812/13180 295 : cluster [WRN] slow request 30.752278 seconds old, received at 2019-01-11 20:16:19.039513: osd_op(client.4256.0:5504 3.2 3:7f3c8970:::smithi13915529-245 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [read 0~3092653,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e354) currently reached_pg" in cluster log

pass 3448676 2019-01-11 17:31:20 2019-01-11 19:52:06 2019-01-11 20:22:06 0:30:00 0:16:55 0:13:05 smithi master
pass 3448677 2019-01-11 17:31:21 2019-01-11 19:54:09 2019-01-11 20:12:08 0:17:59 0:07:55 0:10:04 smithi master
pass 3448678 2019-01-11 17:31:21 2019-01-11 19:54:48 2019-01-11 20:26:47 0:31:59 0:22:03 0:09:56 smithi master
fail 3448679 2019-01-11 17:31:22 2019-01-11 19:55:53 2019-01-11 20:21:52 0:25:59 0:11:47 0:14:12 smithi master
Failure Reason:

"2019-01-11 20:14:45.348586 osd.1 osd.1 172.21.15.104:6805/13266 48 : cluster [WRN] slow request 30.746980 seconds old, received at 2019-01-11 20:14:14.601522: osd_op(client.4257.0:5771 3.0 3:33dd0458:::smithi10416003-215 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [write 659105~652684] snapc 0=[] ondisk+write+known_if_redirected e61) currently reached_pg" in cluster log

pass 3448680 2019-01-11 17:31:23 2019-01-11 19:55:54 2019-01-11 20:15:53 0:19:59 0:10:28 0:09:31 smithi master
pass 3448681 2019-01-11 17:31:24 2019-01-11 19:55:54 2019-01-11 20:41:54 0:46:00 0:22:59 0:23:01 smithi master
pass 3448682 2019-01-11 17:31:24 2019-01-11 19:55:54 2019-01-11 20:29:54 0:34:00 0:20:34 0:13:26 smithi master
pass 3448683 2019-01-11 17:31:25 2019-01-11 19:56:05 2019-01-11 20:42:05 0:46:00 0:19:37 0:26:23 smithi master
pass 3448684 2019-01-11 17:31:26 2019-01-11 19:57:10 2019-01-11 20:31:09 0:33:59 0:24:00 0:09:59 smithi master
fail 3448685 2019-01-11 17:31:27 2019-01-11 19:57:22 2019-01-11 21:07:22 1:10:00 0:59:44 0:10:16 smithi master
Failure Reason:

"2019-01-11 20:28:00.247112 osd.4 osd.4 172.21.15.184:6805/13050 410 : cluster [WRN] slow request 30.329273 seconds old, received at 2019-01-11 20:27:29.917812: osd_op(client.6440.0:14363 5.es0 5:7ec861c4:::benchmark_data_smithi136_64440_object14362:head [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e379) currently waiting for peered" in cluster log

pass 3448686 2019-01-11 17:31:28 2019-01-11 19:57:22 2019-01-11 20:47:22 0:50:00 0:34:28 0:15:32 smithi master
pass 3448687 2019-01-11 17:31:28 2019-01-11 19:57:33 2019-01-11 20:33:33 0:36:00 0:20:31 0:15:29 smithi master
pass 3448688 2019-01-11 17:31:29 2019-01-11 19:57:46 2019-01-11 20:35:46 0:38:00 0:24:00 0:14:00 smithi master
pass 3448689 2019-01-11 17:31:30 2019-01-11 19:57:46 2019-01-11 20:15:46 0:18:00 0:09:08 0:08:52 smithi master
pass 3448690 2019-01-11 17:31:31 2019-01-11 19:59:04 2019-01-11 20:25:04 0:26:00 0:13:13 0:12:47 smithi master
fail 3448691 2019-01-11 17:31:31 2019-01-11 19:59:55 2019-01-11 20:55:55 0:56:00 0:35:09 0:20:51 smithi master
Failure Reason:

"2019-01-11 20:28:01.932112 osd.1 osd.1 172.21.15.135:6801/12947 76 : cluster [WRN] slow request 30.901904 seconds old, received at 2019-01-11 20:27:31.030172: osd_op(client.4223.0:1216 2.20s1 2.833bfea0 (undecoded) ondisk+retry+read+known_if_redirected e119) currently waiting for peered" in cluster log

pass 3448692 2019-01-11 17:31:32 2019-01-11 20:00:18 2019-01-11 20:34:18 0:34:00 0:16:54 0:17:06 smithi master
pass 3448693 2019-01-11 17:31:33 2019-01-11 20:02:06 2019-01-11 21:06:06 1:04:00 0:25:34 0:38:26 smithi master ubuntu 14.04
pass 3448694 2019-01-11 17:31:34 2019-01-11 20:03:02 2019-01-11 20:29:02 0:26:00 0:12:51 0:13:09 smithi master
pass 3448695 2019-01-11 17:31:34 2019-01-11 20:03:19 2019-01-11 20:33:19 0:30:00 0:13:42 0:16:18 smithi master
pass 3448696 2019-01-11 17:31:35 2019-01-11 20:03:46 2019-01-11 20:47:46 0:44:00 0:09:53 0:34:07 smithi master
fail 3448697 2019-01-11 17:31:36 2019-01-11 20:03:55 2019-01-11 20:31:55 0:28:00 0:13:47 0:14:13 smithi master
Failure Reason:

"2019-01-11 20:26:33.809528 osd.3 osd.3 172.21.15.148:6813/13061 142 : cluster [WRN] slow request 30.242852 seconds old, received at 2019-01-11 20:26:03.566616: osd_op(client.4286.0:11925 3.2 3:745afddc:::smithi14815965-441 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e113) currently reached_pg" in cluster log

pass 3448698 2019-01-11 17:31:37 2019-01-11 20:05:13 2019-01-11 20:43:13 0:38:00 0:23:40 0:14:20 smithi master
pass 3448699 2019-01-11 17:31:38 2019-01-11 20:08:06 2019-01-11 20:32:06 0:24:00 0:11:22 0:12:38 smithi master
pass 3448700 2019-01-11 17:31:38 2019-01-11 20:09:20 2019-01-11 20:41:20 0:32:00 0:19:18 0:12:42 smithi master
pass 3448701 2019-01-11 17:31:39 2019-01-11 20:09:20 2019-01-11 20:27:20 0:18:00 0:05:39 0:12:21 smithi master
pass 3448702 2019-01-11 17:31:40 2019-01-11 20:09:44 2019-01-11 21:15:44 1:06:00 0:54:24 0:11:36 smithi master
fail 3448703 2019-01-11 17:31:41 2019-01-11 20:09:54 2019-01-11 20:55:54 0:46:00 0:28:33 0:17:27 smithi master
Failure Reason:

"2019-01-11 20:32:27.973383 osd.4 osd.4 172.21.15.42:6808/12888 46 : cluster [WRN] slow request 31.192398 seconds old, received at 2019-01-11 20:31:56.780934: osd_op(client.4264.0:2819 3.2 3:752085a4:::smithi11715887-302:head [read 0~3058907,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e69) currently reached_pg" in cluster log

pass 3448704 2019-01-11 17:31:41 2019-01-11 20:10:39 2019-01-11 20:38:38 0:27:59 0:09:49 0:18:10 smithi master
pass 3448705 2019-01-11 17:31:42 2019-01-11 20:12:02 2019-01-11 20:46:02 0:34:00 0:24:09 0:09:51 smithi master
pass 3448706 2019-01-11 17:31:43 2019-01-11 20:12:09 2019-01-11 20:46:09 0:34:00 0:20:35 0:13:25 smithi master
pass 3448707 2019-01-11 17:31:44 2019-01-11 20:12:10 2019-01-11 20:44:10 0:32:00 0:22:09 0:09:51 smithi master
pass 3448708 2019-01-11 17:31:45 2019-01-11 20:13:21 2019-01-11 20:45:21 0:32:00 0:21:25 0:10:35 smithi master
pass 3448709 2019-01-11 17:31:45 2019-01-11 20:13:32 2019-01-11 20:29:31 0:15:59 0:07:11 0:08:48 smithi master
pass 3448710 2019-01-11 17:31:46 2019-01-11 20:13:32 2019-01-11 21:19:33 1:06:01 0:49:43 0:16:18 smithi master
pass 3448711 2019-01-11 17:31:47 2019-01-11 20:13:43 2019-01-11 20:51:43 0:38:00 0:19:38 0:18:22 smithi master
pass 3448712 2019-01-11 17:31:48 2019-01-11 20:15:58 2019-01-11 20:51:57 0:35:59 0:19:46 0:16:13 smithi master
pass 3448713 2019-01-11 17:31:48 2019-01-11 20:15:58 2019-01-11 20:55:57 0:39:59 0:21:00 0:18:59 smithi master
pass 3448714 2019-01-11 17:31:49 2019-01-11 20:16:29 2019-01-11 20:46:28 0:29:59 0:13:30 0:16:29 smithi master
pass 3448715 2019-01-11 17:31:50 2019-01-11 20:17:35 2019-01-11 20:39:35 0:22:00 0:07:04 0:14:56 smithi master
pass 3448716 2019-01-11 17:31:51 2019-01-11 20:17:43 2019-01-11 20:45:43 0:28:00 0:13:26 0:14:34 smithi master
pass 3448717 2019-01-11 17:31:52 2019-01-11 20:17:55 2019-01-11 21:07:55 0:50:00 0:23:04 0:26:56 smithi master centos
pass 3448718 2019-01-11 17:31:52 2019-01-11 20:22:06 2019-01-11 21:10:06 0:48:00 0:15:56 0:32:04 smithi master
pass 3448719 2019-01-11 17:31:53 2019-01-11 20:22:07 2019-01-11 21:02:06 0:39:59 0:31:08 0:08:51 smithi master
pass 3448720 2019-01-11 17:31:54 2019-01-11 20:24:07 2019-01-11 21:12:06 0:47:59 0:12:13 0:35:46 smithi master
pass 3448721 2019-01-11 17:31:55 2019-01-11 20:24:49 2019-01-11 20:56:49 0:32:00 0:19:09 0:12:51 smithi master
pass 3448722 2019-01-11 17:31:55 2019-01-11 20:25:05 2019-01-11 21:03:05 0:38:00 0:23:13 0:14:47 smithi master
fail 3448723 2019-01-11 17:31:56 2019-01-11 20:25:55 2019-01-11 21:01:55 0:36:00 0:19:24 0:16:36 smithi master
Failure Reason:

"2019-01-11 20:50:43.330978 osd.6 osd.6 172.21.15.102:6809/13047 156 : cluster [WRN] slow request 30.999052 seconds old, received at 2019-01-11 20:50:12.331887: osd_op(client.4231.0:6582 2.4as0 2:536faae1:::smithi14814800-304:head [copy-from ver 23] snapc b0=[b0,af,ae,ac,a8,a7,a4,a1] ondisk+write+known_if_redirected e240) currently waiting for peered" in cluster log

pass 3448724 2019-01-11 17:31:57 2019-01-11 20:26:50 2019-01-11 21:00:49 0:33:59 0:20:25 0:13:34 smithi master
pass 3448725 2019-01-11 17:31:58 2019-01-11 20:27:31 2019-01-11 21:01:30 0:33:59 0:08:56 0:25:03 smithi master
pass 3448726 2019-01-11 17:31:59 2019-01-11 20:29:40 2019-01-11 20:49:39 0:19:59 0:06:58 0:13:01 smithi master
pass 3448727 2019-01-11 17:31:59 2019-01-11 20:29:40 2019-01-11 21:03:39 0:33:59 0:23:19 0:10:40 smithi master
pass 3448728 2019-01-11 17:32:00 2019-01-11 20:29:55 2019-01-11 21:57:55 1:28:00 0:12:00 1:16:00 smithi master
pass 3448729 2019-01-11 17:32:01 2019-01-11 20:30:32 2019-01-11 20:52:32 0:22:00 0:11:18 0:10:42 smithi master
pass 3448730 2019-01-11 17:32:01 2019-01-11 20:31:58 2019-01-11 20:49:57 0:17:59 0:09:56 0:08:03 smithi master
pass 3448731 2019-01-11 17:32:02 2019-01-11 20:32:07 2019-01-11 21:24:07 0:52:00 0:28:56 0:23:04 smithi master
pass 3448732 2019-01-11 17:32:03 2019-01-11 20:32:07 2019-01-11 21:00:06 0:27:59 0:17:08 0:10:51 smithi master
pass 3448733 2019-01-11 17:32:04 2019-01-11 20:33:48 2019-01-11 20:49:48 0:16:00 0:06:20 0:09:40 smithi master
pass 3448734 2019-01-11 17:32:04 2019-01-11 20:33:49 2019-01-11 21:01:48 0:27:59 0:12:55 0:15:04 smithi master
pass 3448735 2019-01-11 17:32:05 2019-01-11 20:33:51 2019-01-11 21:29:51 0:56:00 0:40:45 0:15:15 smithi master
pass 3448736 2019-01-11 17:32:06 2019-01-11 20:34:19 2019-01-11 21:10:18 0:35:59 0:26:52 0:09:07 smithi master
fail 3448737 2019-01-11 17:32:07 2019-01-11 20:35:25 2019-01-11 21:09:25 0:34:00 0:18:52 0:15:08 smithi master
Failure Reason:

"2019-01-11 21:00:08.561677 osd.3 osd.3 172.21.15.58:6810/12903 201 : cluster [WRN] slow request 32.321715 seconds old, received at 2019-01-11 20:59:36.239929: osd_op(client.4221.0:7766 2.7 2:e7a4126c:::smithi05814829-649 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [copy-from ver 169] snapc fd=[fd,f8] ondisk+write+known_if_redirected e288) currently started" in cluster log

pass 3448738 2019-01-11 17:32:07 2019-01-11 20:35:47 2019-01-11 21:17:47 0:42:00 0:24:43 0:17:17 smithi master
fail 3448739 2019-01-11 17:32:08 2019-01-11 20:38:50 2019-01-11 21:06:50 0:28:00 0:18:07 0:09:53 smithi master
Failure Reason:

"2019-01-11 20:52:50.391693 osd.1 osd.1 172.21.15.43:6813/15585 1 : cluster [WRN] 7 slow requests, 5 included below; oldest blocked for > 30.639731 secs" in cluster log

pass 3448740 2019-01-11 17:32:09 2019-01-11 20:39:36 2019-01-11 21:03:35 0:23:59 0:12:28 0:11:31 smithi master
pass 3448741 2019-01-11 17:32:10 2019-01-11 20:41:32 2019-01-11 21:15:32 0:34:00 0:11:28 0:22:32 smithi master
pass 3448742 2019-01-11 17:32:10 2019-01-11 20:41:55 2019-01-11 21:13:54 0:31:59 0:20:11 0:11:48 smithi master ubuntu 16.04
pass 3448743 2019-01-11 17:32:11 2019-01-11 20:42:06 2019-01-11 21:06:06 0:24:00 0:11:12 0:12:48 smithi master
pass 3448744 2019-01-11 17:32:12 2019-01-11 20:43:26 2019-01-11 21:11:25 0:27:59 0:15:57 0:12:02 smithi master
pass 3448745 2019-01-11 17:32:13 2019-01-11 20:44:11 2019-01-11 21:26:11 0:42:00 0:24:00 0:18:00 smithi master
pass 3448746 2019-01-11 17:32:13 2019-01-11 20:45:32 2019-01-11 21:19:32 0:34:00 0:12:38 0:21:22 smithi master
pass 3448747 2019-01-11 17:32:14 2019-01-11 20:45:44 2019-01-11 21:27:44 0:42:00 0:29:56 0:12:04 smithi master
pass 3448748 2019-01-11 17:32:15 2019-01-11 20:46:03 2019-01-11 21:26:03 0:40:00 0:20:58 0:19:02 smithi master
pass 3448749 2019-01-11 17:32:16 2019-01-11 20:46:10 2019-01-11 21:02:09 0:15:59 0:06:50 0:09:09 smithi master
pass 3448750 2019-01-11 17:32:16 2019-01-11 20:46:29 2019-01-11 21:18:29 0:32:00 0:21:59 0:10:01 smithi master
fail 3448751 2019-01-11 17:32:17 2019-01-11 20:47:33 2019-01-11 21:25:33 0:38:00 0:19:19 0:18:41 smithi master
Failure Reason:

"2019-01-11 21:12:58.975729 osd.3 osd.3 172.21.15.135:6813/13015 94 : cluster [WRN] slow request 30.180660 seconds old, received at 2019-01-11 21:12:28.795013: osd_op(client.4253.0:5500 2.bs0 2.849c9deb (undecoded) ondisk+retry+read+known_if_redirected e107) currently waiting for peered" in cluster log

pass 3448752 2019-01-11 17:32:18 2019-01-11 20:47:47 2019-01-11 21:15:47 0:28:00 0:17:14 0:10:46 smithi master
pass 3448753 2019-01-11 17:32:18 2019-01-11 20:49:51 2019-01-11 21:11:50 0:21:59 0:11:41 0:10:18 smithi master
pass 3448754 2019-01-11 17:32:19 2019-01-11 20:49:51 2019-01-11 21:47:51 0:58:00 0:46:30 0:11:30 smithi master
fail 3448755 2019-01-11 17:32:20 2019-01-11 20:49:58 2019-01-11 21:33:58 0:44:00 0:32:04 0:11:56 smithi master
Failure Reason:

"2019-01-11 21:07:17.627129 osd.3 osd.3 172.21.15.47:6804/13082 78 : cluster [WRN] slow request 31.783664 seconds old, received at 2019-01-11 21:06:45.843407: osd_op(client.4246.0:1200 2.3 2:ce3edf20:::smithi04715647-27 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [copy-from ver 0] snapc 0=[] ondisk+write+known_if_redirected e120) currently started" in cluster log

fail 3448756 2019-01-11 17:32:21 2019-01-11 20:51:54 2019-01-11 21:29:54 0:38:00 0:17:30 0:20:30 smithi master
Failure Reason:

"2019-01-11 21:16:39.441154 osd.7 osd.7 172.21.15.191:6804/12897 86 : cluster [WRN] slow request 30.505057 seconds old, received at 2019-01-11 21:16:08.936072: osd_op(client.4370.0:1 36.3s0 36.6a4e663b (undecoded) ondisk+write+known_if_redirected e34) currently waiting for peered" in cluster log

pass 3448757 2019-01-11 17:32:21 2019-01-11 20:51:58 2019-01-11 21:25:58 0:34:00 0:23:36 0:10:24 smithi master
pass 3448758 2019-01-11 17:32:22 2019-01-11 20:52:33 2019-01-11 21:08:32 0:15:59 0:06:24 0:09:35 smithi master
pass 3448759 2019-01-11 17:32:23 2019-01-11 20:55:55 2019-01-11 21:29:55 0:34:00 0:21:21 0:12:39 smithi master
pass 3448760 2019-01-11 17:32:23 2019-01-11 20:55:55 2019-01-11 21:21:55 0:26:00 0:14:11 0:11:49 smithi master
pass 3448761 2019-01-11 17:32:24 2019-01-11 20:55:56 2019-01-11 21:27:56 0:32:00 0:18:41 0:13:19 smithi master
pass 3448762 2019-01-11 17:32:25 2019-01-11 20:55:58 2019-01-11 21:35:58 0:40:00 0:23:43 0:16:17 smithi master
pass 3448763 2019-01-11 17:32:26 2019-01-11 20:56:50 2019-01-11 21:32:49 0:35:59 0:15:20 0:20:39 smithi master
pass 3448764 2019-01-11 17:32:27 2019-01-11 21:00:19 2019-01-11 21:22:18 0:21:59 0:10:31 0:11:28 smithi master
pass 3448765 2019-01-11 17:32:27 2019-01-11 21:00:50 2019-01-11 21:36:50 0:36:00 0:18:51 0:17:09 smithi master
pass 3448766 2019-01-11 17:32:28 2019-01-11 21:01:42 2019-01-11 21:31:42 0:30:00 0:20:26 0:09:34 smithi master
pass 3448767 2019-01-11 17:32:29 2019-01-11 21:01:49 2019-01-11 21:33:49 0:32:00 0:17:51 0:14:09 smithi master
pass 3448768 2019-01-11 17:32:29 2019-01-11 21:01:56 2019-01-11 21:21:55 0:19:59 0:10:07 0:09:52 smithi master
pass 3448769 2019-01-11 17:32:30 2019-01-11 21:02:08 2019-01-12 03:32:13 6:30:05 6:16:25 0:13:40 smithi master
fail 3448770 2019-01-11 17:32:31 2019-01-11 21:02:10 2019-01-11 21:44:10 0:42:00 0:26:26 0:15:34 smithi master
Failure Reason:

"2019-01-11 21:23:16.689144 osd.4 osd.4 172.21.15.142:6805/13157 70 : cluster [WRN] slow request 30.277619 seconds old, received at 2019-01-11 21:22:46.411454: osd_op(client.4253.0:3091 3.2 3:5844c247:::smithi00516252-93 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [read 0~3546244,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e114) currently reached_pg" in cluster log

pass 3448771 2019-01-11 17:32:32 2019-01-11 21:03:16 2019-01-11 21:35:16 0:32:00 0:19:05 0:12:55 smithi master
fail 3448772 2019-01-11 17:32:32 2019-01-11 21:03:53 2019-01-11 21:37:53 0:34:00 0:21:59 0:12:01 smithi master
Failure Reason:

"2019-01-11 21:27:03.249075 osd.0 osd.0 172.21.15.171:6814/38155 145 : cluster [WRN] slow request 30.994853 seconds old, received at 2019-01-11 21:26:32.254170: osd_op(client.4272.0:7203 3.1 3:90fb8afb:::smithi17116257-295 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [read 0~2941258,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e407) currently reached_pg" in cluster log

pass 3448773 2019-01-11 17:32:33 2019-01-11 21:03:53 2019-01-11 21:41:53 0:38:00 0:27:08 0:10:52 smithi master
pass 3448774 2019-01-11 17:32:34 2019-01-11 21:06:17 2019-01-11 21:28:17 0:22:00 0:11:28 0:10:32 smithi master
pass 3448775 2019-01-11 17:32:35 2019-01-11 21:06:17 2019-01-11 21:32:22 0:26:05 0:16:51 0:09:14 smithi master
pass 3448776 2019-01-11 17:32:36 2019-01-11 21:06:51 2019-01-11 21:34:50 0:27:59 0:14:33 0:13:26 smithi master
pass 3448777 2019-01-11 17:32:36 2019-01-11 21:07:33 2019-01-11 21:35:33 0:28:00 0:16:35 0:11:25 smithi master
pass 3448778 2019-01-11 17:32:37 2019-01-11 21:07:56 2019-01-11 21:41:56 0:34:00 0:22:11 0:11:49 smithi master
pass 3448779 2019-01-11 17:32:38 2019-01-11 21:08:17 2019-01-11 21:40:16 0:31:59 0:18:43 0:13:16 smithi master
pass 3448780 2019-01-11 17:32:38 2019-01-11 21:08:44 2019-01-11 21:34:43 0:25:59 0:13:26 0:12:33 smithi master centos
pass 3448781 2019-01-11 17:32:39 2019-01-11 21:09:26 2019-01-11 22:03:26 0:54:00 0:43:04 0:10:56 smithi master
pass 3448782 2019-01-11 17:32:40 2019-01-11 21:09:55 2019-01-11 21:39:54 0:29:59 0:17:14 0:12:45 smithi master
pass 3448783 2019-01-11 17:32:41 2019-01-11 21:10:07 2019-01-11 21:50:07 0:40:00 0:19:09 0:20:51 smithi master
pass 3448784 2019-01-11 17:32:41 2019-01-11 21:10:20 2019-01-11 21:30:19 0:19:59 0:09:20 0:10:39 smithi master
pass 3448785 2019-01-11 17:32:42 2019-01-11 21:11:37 2019-01-11 23:05:38 1:54:01 0:11:11 1:42:50 smithi master
fail 3448786 2019-01-11 17:32:43 2019-01-11 21:11:45 2019-01-11 21:51:44 0:39:59 0:27:47 0:12:12 smithi master
Failure Reason:

"2019-01-11 21:34:04.770198 osd.6 osd.6 172.21.15.196:6808/12897 293 : cluster [WRN] slow request 31.068064 seconds old, received at 2019-01-11 21:33:33.702077: osd_op(client.4232.0:4618 2.37 2:ede7a9ef:::smithi18715544-31 oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo:head [copy-from ver 264] snapc 167=[167,163,161,15e,15c,159,155,153,150,14f,14d,149] ondisk+write+known_if_redirected e398) currently started" in cluster log

pass 3448787 2019-01-11 17:32:44 2019-01-11 21:11:51 2019-01-11 21:31:51 0:20:00 0:08:41 0:11:19 smithi master
pass 3448788 2019-01-11 17:32:44 2019-01-11 21:12:08 2019-01-11 21:38:07 0:25:59 0:11:12 0:14:47 smithi master
pass 3448789 2019-01-11 17:32:45 2019-01-11 21:14:07 2019-01-11 21:38:06 0:23:59 0:13:38 0:10:21 smithi master
pass 3448790 2019-01-11 17:32:46 2019-01-11 21:15:43 2019-01-11 21:57:43 0:42:00 0:30:37 0:11:23 smithi master centos 7.4
pass 3448791 2019-01-11 17:32:47 2019-01-11 21:15:45 2019-01-11 21:39:44 0:23:59 0:10:18 0:13:41 smithi master
pass 3448792 2019-01-11 17:32:47 2019-01-11 21:15:48 2019-01-11 21:45:47 0:29:59 0:16:15 0:13:44 smithi master
pass 3448793 2019-01-11 17:32:48 2019-01-11 21:17:59 2019-01-11 21:41:58 0:23:59 0:11:28 0:12:31 smithi master
pass 3448794 2019-01-11 17:32:49 2019-01-11 21:18:30 2019-01-11 21:48:30 0:30:00 0:19:08 0:10:52 smithi master
pass 3448795 2019-01-11 17:32:50 2019-01-11 21:19:44 2019-01-11 21:49:43 0:29:59 0:09:12 0:20:47 smithi master