Register trainee teachers - Healthcheck and Ping Endpoints

Healthchecks are handled by HeartbeatController.

Ping

Use the /ping endpoint as a lightweight check to see if the app is up. This can be used by load-balancers, for example, which may frequently test whether to send traffic to a given host running this app. The response will simply be PONG, with a 200 status code, and not specific resources are checked when this endpoint is activated.

Healthcheck

Use the /healthcheck to check whether that the app’s dependencies are working. This returns a JSON structure that lists the checks it performs and their results. For example:

{
  "checks": {
    "database": true,
    "redis": true,
    "sidekiq_processes": true,
    "solid_queue_processes": true
  }
}

The available checks:

database

Check that ‘ActiveRecord’ connection to the database is active.

redis

Checks that Sidekiq can connect to the Redis server.

sidekiq_processes

Check which queues currently have jobs, as reported by Sidekiq, and checks that there is a Sidekiq running for each one. In theory there should be a process for each queue that is being used, and we determine which queues are being used by looking at what jobs we have queued up.

Failures caused by stale jobs

This check will fail if there are stale jobs in a queue that does not have a processor running. You can check which queues have jobs by starting the Rails console and running:

stats = Sidekiq::Stats.new
processes = Sidekiq::ProcessSet.new
stats.queues.keys.each do |queue|
  running = processes.any? { |process| queue.in? process["queues"] }
  puts "#{queue} running? #{running}"
end

If you see jobs in a queue that should not be running any more (for example, we’ve retired find_sync and should not see any jobs for it) then you’ll need to remove any job queues that exist for it for this test to return true. If, on the other hand, you only have the queues you expected listed, but no process is listed for them, then you have a problem.

solid_queue_processes

Check which queues have jobs in Solid Queue. Then check that there is a live Solid Queue worker for each one. A worker is live if it sent a heartbeat within SolidQueue.process_alive_threshold (5 minutes by default).

Solid Queue keeps finished jobs for a day, so a queue counts as being used if it had a job in that time.

Failures caused by stale jobs

You can check which queues have a live worker by starting the Rails console and running:

live_queues = SolidQueue::Process
  .where(kind: "Worker", last_heartbeat_at: SolidQueue.process_alive_threshold.ago..)
  .flat_map { |process| process.metadata["queues"].to_s.split(",") }
SolidQueue::Queue.all.each do |queue|
  puts "#{queue.name} running? #{queue.name.in?(live_queues)}"
end

If a queue should not be used any more, discard its jobs in Mission Control (/system-admin/jobs). Finished jobs are cleared after a day by the clear_solid_queue_finished_jobs recurring task.