Skip to main content

Register trainee teachers - API Monitoring Guide for Register

  • We use yabeda with the yabeda-rails gem to define and collect custom metrics from our Rails application.
  • Yabeda is a framework that provides a simple DSL to declare and manage monitoring metrics. The yabeda-rails gem automatically configures Yabeda with Rails specific hooks.
  • We then use yabeda-prometheus to expose our collected metrics to Prometheus using a /metrics endpoint.
  • Prometheus then ingests them and allows us to build dashboards around them using Grafana. At the time of writing, we do this in our routes file like so:
mount Yabeda::Prometheus::Exporter => "/metrics"

Configuration

The configuration of Yabeda within our application is defined in the initializer app/initializers/yabeda.rb. Here we define all of the metrics that we want to track throughout our app. Things like the total number of requests, the request duration etc.

Yabeda.configure do
  group :register_api do
    counter   :requests_total,
              comment: "Total number of Register API requests",
              tags: %i[method controller action]
  end
end

We then add to these metrics from anywhere in our app that we choose. At the time of writing we’re doing this in our API’s base controller with an around_action:

...
  included do
    around_action :monitor_api_request
  end

  private

  def monitor_api_request
    start = Time.zone.now
    track_total_requests
    begin
      yield
    rescue => ex
      track_unsuccessful_requests(ex)
      raise ex
    ensure
      track_request_duration(start)
      track_response_size
    end
  end
...

Grafana Dashboards

Grafana is configured to visualize the metrics collected by Prometheus. It supports additional data sources, providing a comprehensive view of our application’s operational metrics.

Accessing Grafana Instances

Adding New Dashboards

To add new dashboards or modify existing ones in Grafana, follow the Monitoring documentation in the teacher-services-cloud repository. Or reach out to someone on the infrastructure team to help with this.

Grafana Overview

Once we know that Prometheus is ingesting the data correctly, you’ll likely want to setup a dashboard in Grafana to visualise the data. To do this, follow these steps:

  1. Log in to Grafana and choose “Dashboards” from the menu on the right:
  1. Either choose from the list of dashboards, or choose “New” to create a new dashboard. Here’s the Register API dashboard for example:
  1. Choose “Add” from the menu at the top of the page, and then choose “Visualisation”
  1. You should now see a blank visualisation with no data. To add data, click on the “Metric” dropdown to choose which metric you want to see:
  1. From the dropdown, click on the option at the top for “Metrics Explorer”. Not all available metrics will appear in the initial list, so it’s best to search here.
  1. Some metrics are tracked by default, without anything custom needed from the developer. These metrics are prefixed with "rails":
  1. From there, you should see some data displayed in the viewer, we can now refine our query from this point to show us what we want.
  1. Choose the code option for the metric to edit the query:
  1. The metric can now be queried using the following syntax:

This is filtering by the “controller” property and providing a regular expression to show only those controllers with the “api” prefix.

  1. You can also sum the results to show the data across all instances as a single value:

There are many different ways to query display data in Grafana, and this is just one example. The menu on the right can show you different types of charts and graphs available.