📦
CHI-in-a-box
  • What is CHI-in-a-Box?
  • Before You Begin
    • Assumed Knowledge
    • Hosts and Services
    • Network Overview
    • CC-Ansible
    • The site configuration
      • inventory
      • defaults.yml
      • passwords.yml
      • certificates/
      • node_custom_config/ (optional)
      • post-deploy.yml (optional)
    • How Deployment Works
    • Security considerations
  • Setup Guides
    • Evaluation Site
      • Bring up the Control Plane
    • Production Baremetal
      • Baremetal QuickStart
      • Host Networking Configuration
    • Troubleshooting
      • Networking
    • Verification Checklist
    • Dev-in-a-Box
    • Edge-in-a-Box
  • Reference
    • Chameleon Identity Federation
    • Ironic Flat Networking
    • Ironic Multi-Tenant Networking
    • Glance Image Storage
    • Resource Reservation
      • Default Resource Properties
    • Monitoring
      • IPMI Metrics
      • SNMP Metrics
  • Example Deployments
    • ARM/x86 mixed architecture
    • Edge computing/container testbed
  • Operations
    • Hardware management
    • Certificate management
    • Chameleon tools
      • Hammers 🔨
        • maintenance_reservation
      • Disk image subscription
      • Usage reporting
    • Troubleshooting
      • Known issues
        • Neutron (networking)
        • Nova (KVM)
        • Ironic (bare metal)
      • Instance networking diagnostics
      • Security incident triage
      • Troublesome Hardware
    • Alert runbooks
      • Cron Job No Recent Success
      • Instance Failure
      • Image Cache Space
      • Ironic Node Error State
      • Jupyter Server Launch Failure
      • MySQL Host Down
      • MySQL Replication Error
      • Node Exporter Down
      • Node Network Bridge Down
      • Node Network Bridge Low Traffic
      • Nova Ironic Instance Launch Failure
      • OpenStack API Down
      • PeriodicTask No Recent Success
      • Portal Down
      • Precis Parsed Events Low
      • Provider Conflict
      • Runbook Template
    • User support guide
    • Upgrading to a new Release
  • Development
    • Developing OpenStack Services
    • Dev-in-a-box
Powered by GitBook
On this page
Edit on GitHub
  1. Operations
  2. Alert runbooks

PeriodicTask No Recent Success

Summary: A periodic task has not had a successful run recently.

Consequences: Periodic tasks serve many purposes so the impact depends on the task, though typically is limited in user impact. Most tasks are responsible for cleaning up testbed state; over time if these jobs are left failing, users may experience nodes that cannot be provisioned. Tasks are also involved in creating backups of data, and if they stop working, the ability to restore from backups diminishes.

Possible causes

Authentication errors: Many jobs interface with OpenStack APIs, and user a special user for this. If this user is changed or disabled, jobs can fail.

  1. Try running the failing job manually (see its entry via systemctl show <task>) to see what type of error is generated. If it is an authentication error, see if the user (defined as the hammers_openstack_user var) is disabled/missing.

Runtime error: A job may encounter input is unfamiliar with. The code for the job will have to be fixed; an issue should be raised for this.

Task-specific runbooks

keycloak_tas_sync

If the sync is running for a while but then encounters a 401 unauthorized error, it can be because the service account's token has too short of an expiration. The current configured TTL is in the configuration for the "user-group-import" client in the master Keycloak realm as an advanced setting (Access Token Lifespan).

PreviousOpenStack API DownNextPortal Down

Last updated 2 years ago