joshd-2016-05-31_23:01:34-rados-wip-pg-log-errors-2---basic-mira

226177

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira034 mira075 mira090 mira119

rados/thrash-erasure-code-shec/{leveldb.yaml rados.yaml clusters/{fixed-4.yaml openstack.yaml} fs/xfs.yaml msgr-failures/fastclose.yaml thrashers/default.yaml workloads/ec-rados-plugin=shec-k=4-m=3-c=2.yaml}

'/home/ubuntu/cephtest/archive/syslog/kern.log:2016-06-01T09:02:12.851604+00:00 mira034 kernel: [ INFO: possible circular locking dependency detected ] ' in syslog

226179

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira077 mira090

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/async.yaml msgr-failures/few.yaml thrashers/default.yaml workloads/cache-pool-snaps.yaml}

timed out waiting for admin_socket to appear after osd.4 restart

226183

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira038 mira072

rados/thrash-erasure-code/{leveldb.yaml rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/few.yaml thrashers/fastread.yaml workloads/ec-rados-plugin=jerasure-k=3-m=1.yaml}

'/home/ubuntu/cephtest/archive/syslog/kern.log:2016-06-01T06:16:31.243056+00:00 mira038 kernel: [ INFO: possible circular locking dependency detected ] ' in syslog

226189

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira082 mira084

rados/singleton/{rados.yaml all/thrash_cache_writeback_proxy_none.yaml fs/xfs.yaml msgr/async.yaml msgr-failures/many.yaml}

"2016-06-01 06:40:30.554071 osd.5 172.21.7.116:6808/18898 1 : cluster [WRN] pg 2.2 has invalid (post-split) stats; must scrub before tier agent can activate" in cluster log

226190

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira077 mira090

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/async.yaml msgr-failures/few.yaml thrashers/pggrow.yaml workloads/pool-snaps-few-objects.yaml}

timed out waiting for admin_socket to appear after osd.4 restart

226193

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira034 mira059

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/osd-delay.yaml thrashers/default.yaml workloads/rados_api_tests.yaml}

"2016-06-01 06:52:22.748732 osd.3 172.21.4.108:6800/30802 114 : cluster [WRN] pg 108.7 has invalid (post-split) stats; must scrub before tier agent can activate" in cluster log

226195

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira016 mira052

rados/basic/{rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml mon_kv_backend/leveldb.yaml msgr/random.yaml msgr-failures/many.yaml tasks/rados_api_tests.yaml}

"2016-06-01 07:10:25.006355 osd.3 172.21.5.108:6800/2475 14 : cluster [WRN] pg 100.7 has invalid (post-split) stats; must scrub before tier agent can activate" in cluster log

226205

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira083 mira102

rados/verify/{rados.yaml 1thrash/default.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml mon_kv_backend/rocksdb.yaml msgr/random.yaml msgr-failures/few.yaml tasks/rados_api_tests.yaml validater/lockdep.yaml}

Command failed (workunit test rados/test.sh) on mira083 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=50da4f168a23404e478b98bda16351646fc6fbe6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/workunit.client.0/rados/test.sh'

226207

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira075 mira089

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rgw_snaps.yaml}

"2016-06-01 07:11:44.746813 osd.1 172.21.7.106:6800/22466 115 : cluster [ERR] 3.4 shard 1 missing 3:2b04a3e9:::notify.6:head" in cluster log

226217

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira042 mira061

rados/monthrash/{rados.yaml ceph/ceph.yaml clusters/9-mons.yaml fs/xfs.yaml mon_kv_backend/rocksdb.yaml msgr/simple.yaml msgr-failures/few.yaml thrashers/sync.yaml workloads/rados_5925.yaml}

SELinux denials found on ubuntu@mira061.front.sepia.ceph.com: ['type=AVC msg=audit(1464764886.487:38280): avc: denied { chown } for pid=25064 comm="radosgw" capability=0 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability permissive=1', 'type=AVC msg=audit(1464764886.517:38281): avc: denied { setattr } for pid=25064 comm="radosgw" name="ceph-client.rgw.mira061.asok" dev="tmpfs" ino=671315 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:ceph_var_run_t:s0 tclass=sock_file permissive=1', 'type=AVC msg=audit(1464764887.563:38300): avc: denied { chown } for pid=25112 comm="radosgw" capability=0 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability permissive=1']

226220

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira035 mira072

rados/thrash-erasure-code/{leveldb.yaml rados.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr-failures/fastclose.yaml thrashers/morepggrow.yaml workloads/ec-small-objects-fast-read.yaml}

timed out waiting for admin_socket to appear after osd.4 restart

226229

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira034 mira042

rados/singleton/{rados.yaml all/lost-unfound-delete-upgrade.yaml fs/xfs.yaml msgr/async.yaml msgr-failures/few.yaml}

SELinux denials found on ubuntu@mira034.front.sepia.ceph.com: ['type=AVC msg=audit(1464766439.424:8500): avc: denied { open } for pid=10725 comm="ceph-osd" path="/var/lib/ceph/osd/ceph-1/magic" dev="sdf" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1464766437.449:8419): avc: denied { open } for pid=10482 comm="ceph-osd" path="/var/lib/ceph/osd/ceph-0/magic" dev="sdh" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1464766437.449:8419): avc: denied { read } for pid=10482 comm="ceph-osd" name="magic" dev="sdh" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1464766439.424:8500): avc: denied { read } for pid=10725 comm="ceph-osd" name="magic" dev="sdf" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1']

226230

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira039 mira090

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/osd-delay.yaml thrashers/mapgap.yaml workloads/cache-agent-small.yaml}

Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op copy_from 50 --op write_excl 50 --op delete 50 --pool base'

226236

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira048 mira061 mira089 mira094

rados/thrash-erasure-code-big/{leveldb.yaml rados.yaml cluster/{12-osds.yaml openstack.yaml} fs/btrfs.yaml msgr-failures/few.yaml thrashers/mapgap.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml}

SELinux denials found on ubuntu@mira061.front.sepia.ceph.com: ['type=AVC msg=audit(1464770533.953:3638): avc: denied { chown } for pid=19928 comm="radosgw" capability=0 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability permissive=1', 'type=AVC msg=audit(1464770534.003:3639): avc: denied { setattr } for pid=19928 comm="radosgw" name="ceph-client.rgw.mira061.asok" dev="tmpfs" ino=72667 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:ceph_var_run_t:s0 tclass=sock_file permissive=1', 'type=AVC msg=audit(1464770534.836:3654): avc: denied { setattr } for pid=19957 comm="radosgw" name="ceph-client.rgw.mira061.asok" dev="tmpfs" ino=72675 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:ceph_var_run_t:s0 tclass=sock_file permissive=1']

226237

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira039 mira112

rados/singleton/{rados.yaml all/lost-unfound-upgrade.yaml fs/xfs.yaml msgr/simple.yaml msgr-failures/few.yaml}

SELinux denials found on ubuntu@mira112.front.sepia.ceph.com: ['type=AVC msg=audit(1464766051.318:22579): avc: denied { open } for pid=16593 comm="ceph-osd" path="/var/lib/ceph/osd/ceph-1/magic" dev="sdg" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1464766052.010:22604): avc: denied { open } for pid=16668 comm="ceph-osd" path="/var/lib/ceph/osd/ceph-0/magic" dev="sdf" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1464766051.318:22579): avc: denied { read } for pid=16593 comm="ceph-osd" name="magic" dev="sdg" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1', 'type=AVC msg=audit(1464766052.010:22604): avc: denied { read } for pid=16668 comm="ceph-osd" name="magic" dev="sdf" ino=26 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:unlabeled_t:s0 tclass=file permissive=1']

226238

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira032 mira094

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/3-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr/async.yaml msgr-failures/few.yaml thrashers/pggrow.yaml workloads/cache-pool-snaps.yaml}

timed out waiting for admin_socket to appear after osd.2 restart

226240

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira051 mira122

rados/monthrash/{rados.yaml ceph/ceph.yaml clusters/3-mons.yaml fs/xfs.yaml mon_kv_backend/leveldb.yaml msgr/async.yaml msgr-failures/mon-delay.yaml thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml}

"2016-06-01 07:24:41.093356 osd.2 172.21.5.110:6808/30255 5 : cluster [WRN] pg 92.4 has invalid (post-split) stats; must scrub before tier agent can activate" in cluster log

226245

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira062 mira103

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/2-size-2-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/ext4.yaml msgr/simple.yaml msgr-failures/fastclose.yaml thrashers/mapgap.yaml workloads/cache.yaml}

Command crashed: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op copy_from 50 --op write_excl 50 --op delete 50 --pool base'

226248

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira061

rados/singleton/{rados.yaml all/mon-thrasher.yaml fs/xfs.yaml msgr/simple.yaml msgr-failures/many.yaml}

SELinux denials found on ubuntu@mira061.front.sepia.ceph.com: ['type=AVC msg=audit(1464766121.010:3603): avc: denied { setattr } for pid=20265 comm="radosgw" name="ceph-client.rgw.mira061.asok" dev="tmpfs" ino=77052 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:ceph_var_run_t:s0 tclass=sock_file permissive=1', 'type=AVC msg=audit(1464766120.957:3602): avc: denied { chown } for pid=20265 comm="radosgw" capability=0 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability permissive=1']

226252

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira052 mira089

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/osd-delay.yaml thrashers/pggrow.yaml workloads/rados_api_tests.yaml}

"2016-06-01 07:46:58.975504 osd.2 172.21.7.106:6808/26096 143 : cluster [WRN] pg 110.0 has invalid (post-split) stats; must scrub before tier agent can activate" in cluster log

226254

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira048 mira061 mira089 mira094

rados/thrash-erasure-code-big/{leveldb.yaml rados.yaml cluster/{12-osds.yaml openstack.yaml} fs/ext4.yaml msgr-failures/osd-delay.yaml thrashers/morepggrow.yaml workloads/ec-rados-plugin=lrc-k=4-m=2-l=3.yaml}

SELinux denials found on ubuntu@mira061.front.sepia.ceph.com: ['type=AVC msg=audit(1464771981.663:3645): avc: denied { chown } for pid=20305 comm="radosgw" capability=0 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability permissive=1', 'type=AVC msg=audit(1464771981.708:3646): avc: denied { setattr } for pid=20305 comm="radosgw" name="ceph-client.rgw.mira061.asok" dev="tmpfs" ino=76203 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:ceph_var_run_t:s0 tclass=sock_file permissive=1']

226265

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira059 mira069

rados/thrash/{hobj-sort.yaml rados.yaml rocksdb.yaml 0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml msgr/random.yaml msgr-failures/osd-delay.yaml thrashers/morepggrow.yaml workloads/rgw_snaps.yaml}

"2016-06-01 08:05:21.118039 osd.4 172.21.6.110:6804/24697 124 : cluster [ERR] 3.3 shard 4 missing 3:c609908c:::notify.5:head" in cluster log

226276

FAIL

wip-pg-log-errors-2

master

master

mira

centos 7.2

mira034 mira061

rados/thrash-erasure-code-isa/{leveldb.yaml rados.yaml arch/x86_64.yaml clusters/{fixed-2.yaml openstack.yaml} fs/xfs.yaml msgr-failures/fastclose.yaml supported/centos_7.2.yaml thrashers/morepggrow.yaml workloads/ec-rados-plugin=isa-k=2-m=1.yaml}

'/home/ubuntu/cephtest/archive/syslog/kern.log:2016-06-01T09:34:02.475331+00:00 mira061 kernel: [ INFO: possible circular locking dependency detected ] ' in syslog

226277

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira061

rados/objectstore/objectcacher-stress.yaml

SELinux denials found on ubuntu@mira061.front.sepia.ceph.com: ['type=AVC msg=audit(1464767864.411:3603): avc: denied { setattr } for pid=19701 comm="radosgw" name="ceph-client.rgw.mira061.asok" dev="tmpfs" ino=76935 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:object_r:ceph_var_run_t:s0 tclass=sock_file permissive=1', 'type=AVC msg=audit(1464767864.356:3602): avc: denied { chown } for pid=19701 comm="radosgw" capability=0 scontext=system_u:system_r:ceph_t:s0 tcontext=system_u:system_r:ceph_t:s0 tclass=capability permissive=1']

226284

FAIL

wip-pg-log-errors-2

master

master

mira

 

mira030 mira088

rados/monthrash/{rados.yaml ceph/ceph.yaml clusters/3-mons.yaml fs/xfs.yaml mon_kv_backend/leveldb.yaml msgr/simple.yaml msgr-failures/mon-delay.yaml thrashers/one.yaml workloads/snaps-few-objects.yaml}

'/home/ubuntu/cephtest/archive/syslog/kern.log:2016-06-01T08:28:42.363686+00:00 mira030 kernel: [ INFO: possible circular locking dependency detected ] ' in syslog