Pulpito

julpark-2021-03-03_20:03:25-fs:workload-master-distro-basic-smithi/5931175

Posted Started Updated
Runtime
Duration
In Waiting
 Machine Teuthology Branch OS Type OS Version Nodes Status
2021-03-03 20:05:35 2021-03-04 19:20:54 2021-03-04 19:46:14 0:25:20 0:12:24 0:12:56 smithi master centos 8.2 5 fail

Description: fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}

Log: http://qa-proxy.ceph.com/teuthology/julpark-2021-03-03_20:03:25-fs:workload-master-distro-basic-smithi/5931175/teuthology.log

Sentry event: https://sentry.ceph.com/organizations/ceph/?query=0f9e3b223aed4ee5861797765a08e308

Failure Reason:

Command failed on smithi103 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:bfde964cf0400f19b7ae46122039982198a0f4db shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 62f1f3a4-7d21-11eb-9065-001a4aab830c -- ceph orch daemon add mon smithi103:172.21.15.103=smithi103'

All Details...

  • log_href: http://qa-proxy.ceph.com/teuthology/julpark-2021-03-03_20:03:25-fs:workload-master-distro-basic-smithi/5931175/teuthology.log
  • archive_path: /home/teuthworker/archive/julpark-2021-03-03_20:03:25-fs:workload-master-distro-basic-smithi/5931175
  • description: fs:workload/{begin clusters/1a5s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{frag_enable osd-asserts session_timeout whitelist_health whitelist_wrongly_marked_down} ranks/3 scrub/no tasks/{0-check-counter workunit/suites/iozone}}
  • duration: 0:12:24
  • email:
  • failure_reason: Command failed on smithi103 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:bfde964cf0400f19b7ae46122039982198a0f4db shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 62f1f3a4-7d21-11eb-9065-001a4aab830c -- ceph orch daemon add mon smithi103:172.21.15.103=smithi103'
  • flavor: basic
  • job_id: 5931175
  • kernel:
    • sha1: distro
    • kdb: True
  • last_in_suite: False
  • machine_type: smithi
  • name: julpark-2021-03-03_20:03:25-fs:workload-master-distro-basic-smithi
  • nuke_on_error: True
  • os_type: centos
  • os_version: 8.2
  • overrides:
    • ceph-deploy:
      • fs: xfs
      • conf:
        • client:
          • log file: /var/log/ceph/ceph-$name.$pid.log
        • mon:
          • osd default pool size: 2
        • osd:
          • mon osd full ratio: 0.9
          • mon osd backfillfull_ratio: 0.85
          • bluestore fsck on mount: True
          • mon osd nearfull ratio: 0.8
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • bluestore block size: 96636764160
          • debug rocksdb: 10
          • osd failsafe full ratio: 0.95
      • bluestore: True
    • check-counter:
      • dry_run: True
      • counters:
        • mds:
          • mds.exported
          • mds.imported
    • selinux:
      • whitelist:
        • scontext=system_u:system_r:logrotate_t:s0
    • workunit:
      • sha1: ac62ca61d46e87208fa6292c9152faacd08c23b1
      • branch: julpark_workload
    • ceph:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • fs: xfs
      • sha1: bfde964cf0400f19b7ae46122039982198a0f4db
      • conf:
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • osd op complaint time: 180
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • mds bal fragment size max: 10000
          • mds op complaint time: 180
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
          • mds debug scatterstat: True
          • mds debug frag: True
        • client:
          • rados osd op timeout: 15m
          • debug ms: 1
          • rados mon op timeout: 15m
          • debug client: 20
          • client mount timeout: 600
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 20
          • debug journal: 20
          • osd_max_omap_entries_per_request: 10
          • debug osd: 25
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • osd op complaint time: 180
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • bluestore fsck on mount: True
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
          • mon op complaint time: 120
        • mgr:
          • debug ms: 1
          • debug mgr: 20
      • cephfs:
        • session_timeout: 300
        • ec_profile:
          • m=2
          • k=2
          • crush-failure-domain=osd
        • max_mds: 3
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
    • install:
      • ceph:
        • sha1: bfde964cf0400f19b7ae46122039982198a0f4db
    • admin_socket:
      • branch: master
    • thrashosds:
      • bdev_inject_crash_probability: 0.5
      • bdev_inject_crash: 2
  • owner: scheduled_julpark@teuthology
  • pid:
  • roles:
    • ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0']
    • ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1']
    • ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0']
    • ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'osd.1', 'osd.4', 'osd.7', 'osd.10']
    • ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'osd.2', 'osd.5', 'osd.8', 'osd.11']
  • sentry_event: https://sentry.ceph.com/organizations/ceph/?query=0f9e3b223aed4ee5861797765a08e308
  • status: fail
  • success: False
  • branch: master
  • seed:
  • sha1: bfde964cf0400f19b7ae46122039982198a0f4db
  • subset:
  • suite:
  • suite_branch: julpark_workload
  • suite_path:
  • suite_relpath:
  • suite_repo:
  • suite_sha1: ac62ca61d46e87208fa6292c9152faacd08c23b1
  • targets:
    • smithi082.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEIm0SfLUySsV6tO2SHsF93V2U43SY32gTxQQreYVhof8rUoEUc0Rfr3JDfzfAWY9dtbgwOrdqEeVv0ynm6pTeZOjhqpArJet/6TWQVRu90Cl4uEygIrdGjMDIQn8Ei7zGpRmpsonK6IIK5xe+UxJwAv/Jm4sdBTWlDPs4raAEPAF7ewItytx7TUcw9Jp1cZ7r+6m2Y37JON7Gb2J3OwOeDqEwsLsc+3a1AXrsHzAfYHaq1f/MKPMXqXIIHXMzWK2rVMN9z9znBtA+x7phKwjkx0eV2oCBNuW35+q/hqsIlB/V5t76remFAI8dr9ta2Ga8g89WYi4Fm1Dgk1q8KhupHRURpO+eDFWTj4LlHjv0M0SKV1n8uM/uCLciRyz7jxXldvmENa6CUzFrDzMKQEZNwCmlcJjMB7Y52rm2XS+B4f7bVehXlVCrf4PCLaE4QB0AjFUTQzlDbA5xSFJEqrzMct7JtAU+gzek+5L3FMw6XOq2oLqcKrUpjoZFDKlgteE=
    • smithi103.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeDsEcnbtvI5JB88zTL/gF4PkkwjnCxde3SyylNbyYHf9xTlwJHa10cnqoT7rdfd0duInAJcRlrmgIHUJS0eQorWlmBdXJXcILAMrCmeYlia3LZfaul3TCsA/IDTrpBCbPLTs92zkUxattHHChx/rT85qIsTorb3MaIh27fwFcvUUQ8dSuqp5/DY/w4tdTfzuESVnyMjyefKu8CrAqd8qPA8VGMEtoz1rqM6gJda5uM/7fDu257dQDjg/2sCHsP5EPA1nVeKWBczPUfHOyXkUAGbIkDNc8JJEk0Ni2ij4b1t+4lFJTOiwlwUqPBuzbcTzSXrkV0Y3TCN+TOtJmXVTcQ8PtbzU8kuiPikDpz5YY0XkWSgzxdjy8UKld698nVppNWnH8MkVhlCjsD9/EKXcy2SbIpzfgXUgMsJMzwYApGz6tNWiXxYQCRZ++Z9SO9wbP20WAlWmMxcKzZnVn4KYTsKqpmhcEeRKzmqB5IG0UEqMx+cgFDpMqRoJOvsViIKs=
    • smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOGWeT/Fmca3aqOeN5j6vlBzKXi9JpMAxKDL3A2+gwO0EFPjpmrBgLk8r6DiGR99ShPgSDCYdu2BBcayDHKPF1tegTCTUiIwrd5k+dBoVZ37KxnEtH5zMgsc6Nd2PiP23ZsqU+kevub0cCcSdhjq0Oa8wj7pptN6p8zu4NeIJqMecMw+dEhl82TBD712Z3UVO7rurV2QJQWBD68iwDwTDQUcbjtcNksCEaxIzJ4qcH8rov+/yWCD1d84zOY/4YKgfTWxb9e1FgufIcKY7NOcCxwt5eJ/LxHgnzoE1vDawjmw0u7kRmG05FCm4RyzGyxQfaB4XQeecHzWfMFKXtajlCC8SmcvOPvY0/b4lYsxOn9tIxnlD4maW9mmjD3cY9yCe75Eiai9fN9ByFkyDMTcYqBUS2Lq1f5GrgkjnRokHCEUVa/vlcBpKEAU3FTbrYJZax+I3lST+R7eXaxN7/jg91Mvepotb50+Ka2PqfKnWmQOJH4ggBZD+RvBIBJgTVQNs=
    • smithi199.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDdvLkOda2qFgOcWZ2ZJ2pJKAiPZGz28hU4jcFG88pptRuw1/Oq9/Sd2AtvXN+/Kd+2nU7VWxfwrj+2qlWjusgmsyQsyxIZt6egGOya7IrKKsRyuRTjkAuqt6/kMkP9cnwDpZRldTZSiqk/TErpFUod6CJuaqW2T2AgiPx2JQyQ4xmOwNn184/k/L65CrSyHelFTFLiUi+jgTNXIUTPOjQRQlXw57lfBnBRnSUXzpY+GaIT8vm8bUcMAoCbE+qlUABHyldv31HMkNSOD+ReYQDy9JAIPjqvxiIBzeiYmhfTmORSo9SG8ikYOcohVFSWW2ZvAj4mtfMcTdcijgHx9NOgrgCo+guOpF9N2DRad/MeMWePKsdQTZaBD4kY2HWgeHzWlWFzGppIqZJ2Hid76qiYTajopZ/UvolRLH/KPVvHm21F3gbvhcyDoyYZbsAA34fglON73FRZxImMF3sSdIRlUwEWz8f0vt6qZQKgGQDJ+7ngirHRRJeMwmTtisxzhCs=
    • smithi175.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDVCNfPMKqoo5vt/kazPCg1D5tKwuZbf6h+ERyJ6kHKaEnUfCUAIcAg601PGnxMYnvet4jEoq2WhC3uNP+eDxnovXmDon56NdDAwRdod+9sy7a7h5yqTWMcObVqFO+FaFxo9zJRbSZli+HiexptcXmVVz8uP7CNOZDrxeymlOA2A6AnNHfnkkxn/Lq+DrIgcLQ+mMbRErcc8sTXqe2p/h1N1ek/MFeTn5lLI9djFpEjIrgMhF6xu9d7zW4mZBsk3TcRSKzA6sSg/L6OWQKTb/xfrvxZBkYTKAR/sO4KgRPtEdXOygo/iEcLS5/5LquRWMTLTtqgeuLqqo6xDJqKrz++eslt2WmKxBt6BjUCrimGaisKvfIVyeTxIb1aK0MdHY7AnbtgGXG2p2tAoGQnDQNyLE6wq3N7AN72KWgmWWMuje4QExr4DPwywRh3IeumF0Hytk47j8BW23MOuY4LdpvVO1RJPv3xa7FTKRWMwHNbSBvoFwMjdpgYt9UIk6mhPsk=
  • tasks:
    • internal.check_packages:
    • internal.buildpackages_prep:
    • internal.save_config:
    • internal.check_lock:
    • internal.add_remotes:
    • console_log:
    • internal.connect:
    • internal.push_inventory:
    • internal.serialize_remote_roles:
    • internal.check_conflict:
    • internal.check_ceph_data:
    • internal.vm_setup:
    • kernel:
      • sha1: distro
      • kdb: True
    • internal.base:
    • internal.archive_upload:
    • internal.archive:
    • internal.coredump:
    • internal.sudo:
    • internal.syslog:
    • internal.timer:
    • pcp:
    • selinux:
    • ansible.cephlab:
    • clock:
    • install:
    • cephadm:
      • log-whitelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
      • roleless: True
      • cephadm_mode: root
      • sha1: bfde964cf0400f19b7ae46122039982198a0f4db
      • cluster: ceph
      • fs: xfs
      • conf:
        • mds:
          • mds bal split bits: 3
          • mds bal split size: 100
          • osd op complaint time: 180
          • debug mds: 20
          • mds bal merge size: 5
          • debug ms: 1
          • mds bal frag: True
          • mds verify scatter: True
          • mds bal fragment size max: 10000
          • mds op complaint time: 180
          • rados mon op timeout: 15m
          • rados osd op timeout: 15m
          • mds debug scatterstat: True
          • mds debug frag: True
        • client:
          • rados osd op timeout: 15m
          • debug ms: 1
          • rados mon op timeout: 15m
          • debug client: 20
          • client mount timeout: 600
        • osd:
          • mon osd full ratio: 0.9
          • debug ms: 20
          • debug journal: 20
          • osd_max_omap_entries_per_request: 10
          • debug osd: 25
          • debug bluestore: 20
          • debug bluefs: 20
          • osd objectstore: bluestore
          • mon osd backfillfull_ratio: 0.85
          • osd op complaint time: 180
          • bluestore block size: 96636764160
          • osd shutdown pgref assert: True
          • debug filestore: 20
          • debug rocksdb: 10
          • mon osd nearfull ratio: 0.8
          • osd failsafe full ratio: 0.95
          • bluestore fsck on mount: True
        • mon:
          • debug paxos: 20
          • debug mon: 20
          • debug ms: 1
          • mon op complaint time: 120
        • mgr:
          • debug ms: 1
          • debug mgr: 20
      • cephfs:
        • session_timeout: 300
        • ec_profile:
          • m=2
          • k=2
          • crush-failure-domain=osd
        • max_mds: 3
      • log-ignorelist:
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • overall HEALTH_
        • \(FS_DEGRADED\)
        • \(MDS_FAILED\)
        • \(MDS_DEGRADED\)
        • \(FS_WITH_FAILED_MDS\)
        • \(MDS_DAMAGE\)
        • \(MDS_ALL_DOWN\)
        • \(MDS_UP_LESS_THAN_MAX\)
        • \(FS_INLINE_DATA_DEPRECATED\)
        • overall HEALTH_
        • \(OSD_DOWN\)
        • \(OSD_
        • but it is still running
        • is not responding
    • cephadm.shell:
      • host.a:
        • ceph orch status
        • ceph orch ps
        • ceph orch ls
        • ceph orch host ls
        • ceph orch device ls
    • cephadm.shell:
      • host.a:
        • ceph fs volume create foo
        • ceph fs volume create bar
    • sleep:
      • interval: 60
    • ceph-fuse:
    • ceph-fuse:
      • client.0:
        • cephfs_name: foo
    • ceph-fuse:
      • client.1:
        • cephfs_name: bar
    • check-counter:
      • workunit:
        • clients:
          • all:
            • suites/iozone.sh
    • teuthology_branch: master
    • verbose: True
    • pcp_grafana_url:
    • priority:
    • user:
    • queue:
    • posted: 2021-03-03 20:05:35
    • started: 2021-03-04 19:20:54
    • updated: 2021-03-04 19:46:14
    • status_class: danger
    • runtime: 0:25:20
    • wait_time: 0:12:56