At work we have a Sentry subscription which we use primarily for issue tracking but recently I noticed the “Crons” entry in the sidebar and decided to give it a try as we have a bunch of Nomad periodic jobs that aren’t exactly easy to monitor with the Nomad WUI.

Screenshot of Sentry's Cron monitors page.
Sentry highlights as red when an error occurred and yellow when an expected check-in was missed.

For context, our periodic jobs are basically invocations of bundle exec rake <task> commands, so each task is running within a ruby process and Sentry provides an easy to use API via their SDK.

The “Set Up Crons” document is straightforward and easy to follow. Given we’re not using Sidekiq we couldn’t just include their modules and call it a day. Instead we used the “Manual Setup” approach but it quickly became a repetitive task to do a check-in for every job.

Upserting Cron Monitors

We already have a YAML file used to provision the nomad jobs with a cron schedule, command and per-environment resources. We added a few more properties to support “upsert” easily:

  • slug: used to identify Sentry monitors
  • max_runtime: to let Sentry know how long the job is expected to run
  • checkin_margin: we trust Nomad to spin-up the job on time but it doesn’t hurt to add this measure.

So the YAML file ended up looking like the following:

# config/nomad_jobs.yml
default: &default
  cron: "* * * * *"
  # Sentry cron monitors settings
  checkin_margin: 5 # Optional check-in margin in minutes
  max_runtime: 15 # minutes

send_daily_reports:
  <<: *default
  cron: "30 15 * * *"
  command: "bundle exec rake geckox:send_daily_reports"
  slug: "daily-reports"
  max_runtime: 3

# ...

And with this in place, we fully took advantage of Sentry’s capture_check_in method:

module Geckox
  class << self
    # Execute block within a Sentry cron check-in for the given slug.
    #
    # This will "upsert" the monitor if <code>slug</code> is found in
    # <code>config/nomad_jobs.yml</code>.
    #
    # @param slug [String] Slug matching a manually created Sentry monitor
    # @yield [] Block receives no arguments
    #
    # @see https://docs.sentry.io/platforms/ruby/crons/#manual-setup
    # @see https://docs.sentry.io/platforms/ruby/crons/#upserting-cron-monitors
    def sentry_check_in(slug)
      monitor_config = nil
      nomad_job = Rails.configuration
        .nomad_jobs
        .select { |key| Rails.configuration.nomad_jobs.dig(key, "slug") == slug }

      if nomad_job.present?
        nomad_config = nomad_job.values.first
        monitor_config = Sentry::Cron::MonitorConfig.from_crontab(
          nomad_config["cron"],
          checkin_margin: nomad_config["checkin_margin"],
          max_runtime: nomad_config["max_runtime"]
        )
      end

      check_in_id = Sentry.capture_check_in(slug, :in_progress, monitor_config: monitor_config)
      yield
      Sentry.capture_check_in(slug, :ok, check_in_id: check_in_id, monitor_config: monitor_config)
    rescue => e
      if defined?(check_in_id)
        Sentry.capture_check_in(slug, :error, check_in_id: check_in_id)
      end
      raise e
    end
  end
end

# To trigger a monitored job
Geckox.sentry_check_in("daily-reports") do
  SendDailyReportsJob.perform_now
end

And with this in place, we got a nice UI informing us if something isn’t right at a glance.

Additionally, whenever an error is raised from a periodic job, it’s automatically linked from the monitor page to a Sentry issue with the relevant context at hand. Great product integration!