I was bored and wanted to get some kicks so I decided to automate the deployment of a cron(8) job I have running on a server, so in this post we see how to setup a CI/CD pipeline relying on GitLab Webhooks and custom deployment scripts.

NOTE: you can check the source code for this post at juankman94/githooks-receiver.

Disclosure: I did this mostly because I wanted to practice my ruby and configure a server for the fun of it, take that into account when reading this.

This exercise assumes:

  • An application is already deployed somewhere
  • Development is still being made
  • Proper/secure configuration exists
  • The application server uses systemd(1)

The CI/CD flow used in this exercise goes like this:

  1. Development is done locally
  2. Changes are commited and pushed to GitLab (or any git provider that supports webhooks)
  3. GitLab notifies the receiver of the Push Event
  4. The receiver does something™

The Receiver

GitLab shows a basic WEBrick server to receive the event, but we’ll go with Sinatra ‘cuz I’ve never used it before. From the website: “Sinatra is a DSL for quickly creating web applications in Ruby with minimal effort”. Having said that, let’s create the project by declaring the dependencies in a Gemfile:

# Gemfile
source 'https://rubygems.org'

gem 'sinatra'
Install dependencies by running bundle install (click for details)

For Fedora 30 I had to install some dependencies (to build with native extensions), namely:

% dnf install -y ruby ruby-dev redhat-rpm-config
% dnf groupinstall -y "Development Tools"

The GitLab documentation states that it doesn’t care the response status and that the receiver SHOULD reply as soon as possible:

Your endpoint should send its HTTP response as fast as possible. If you wait too long, GitLab may decide the hook failed and retry it.

So the receiver (app.rb) looks like this:

# frozen_literal_string: true
require "sinatra"

ALLOWED_REPOS = %w( myproject )
DEPLOYMENTS = {
  "myproject" => "./deployments/myproject",
}

post "/repo/:repo" do
  repo = params["repo"]
  resp = {
    status: "ok",
    code: 200,
  }

  if ALLOWED_REPOS.include?(repo)
    deploy_pid = Process.spawn(DEPLOYMENTS[repo])
    Process.detach(deploy_pid)
  else
    status 403
    resp = { status: "forbidden", code: 403 }
  end

  body JSON.dump(resp)
end

As we can see in the DEPLOYMENTS hash, we map a repository name to a deployment script (or command of your choice, really). In the snippet above it is stated that the script deployments/myproject will be executed in the background, so let’s see what it does:

#!/bin/sh
# Update `myproject` source code & restart service

HOOKS_DIR="$( dirname "$(realpath $0)" )"/..
LOG_FILE="$HOOKS_DIR/log/myproject.log"
MYPROJECT_DIR="$HOME/myproject"
VERSION="$1"

function log_message() {
	echo "[$(date --iso-8601=seconds)] $@" | tee -a $LOG_FILE
}

mkdir -p "$(dirname "$LOG_FILE")"

log_message "downloading version $VERSION"

log_message "changing directory to $MYPROJECT_DIR"
cd "$MYPROJECT_DIR"

log_message "stashing directory"
git stash | tee -a $LOG_FILE

log_message "pulling changes"
git pull | tee -a $LOG_FILE

log_message "restarting application"
sudo systemctl restart myproject

With these files in place, we have all we need: a trigger, an HTTP server to receive the events and a script to run the pipeline. Do note that these files reveal several things from the environment:

  • The receiver user is able to execute the deployment commands (some configuration may be needed depending on the cmd)
  • We’re not making sure the deployment is successful
  • myproject is running as an systemd.unit(5) service

So now let’s (manually) run the receiver server and see how it works (we can test webhooks by pushing events at will from our project’s hooks menu, but not just yet):

$ bundle exec ruby app.rb -p 8008 -o 0.0.0.0
[2020-03-30 17:22:58] INFO  WEBrick 1.4.2
[2020-03-30 17:22:58] INFO  ruby 2.5.7 (2019-10-01)
== Sinatra (v2.0.8.1) has taken the stage on 4567 for development with backup from WEBrick
[2020-03-30 17:22:58] INFO  WEBrick::HTTPServer#start: pid=22884 port=8008

We’re running the server on port 8008 but you can set whichever port you want. And if we POST a message we should see a log message:

$ curl -X POST --data '{"a":"b"}' http://localhost:8008/repo/myproject
{"status":"ok","code":200}

# and in the sinatra process log
...
200.200.0.146 - - [30/Mar/2020:17:23:03 +0000] "POST /repo/myproject HTTP/1.1" 200 33 0.0102

The Host

So we have the webhook receiver server, but how can we ensure that it’ll be running to receive the notifications? For that we’ll setup a systemd(1) service. Now, I won’t go into the details of how this works, that’s out of the scope of this post, but we’ll get to see the results after placing files in the proper locations.

SystemD

Since we’re running an application that will be spawning new processes from events received from the wild, it’s better to do it with as little privileges as possible, so we’ll run the receiver as an systemd.unit(5) service. First, we place the githooks.service file in the /etc/systemd/system/ directory:

# githooks.service
[Unit]
Description=GitLab Webhooks Receiver
DefaultDependencies=no
After=network.target

[Service]
Type=simple
ExecStart=/opt/githooks/bin/run
User=githooks

[Install]
WantedBy=default.target

With that in place, let’s load, enable (so it’ll start automatically on (re)boot) and start the service:

% systemctl daemon-reload
% systemctl status githooks
githooks.service - GitLab Webhooks Receiver
   Loaded: loaded (/etc/systemd/system/githooks.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
% systemctl enable githooks
% systemctl start githooks
% systemctl enable githooks
% systemctl status githooks
githooks.service - GitLab Webhooks Receiver
   Loaded: loaded (/etc/systemd/system/githooks.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-31 06:49:27 UTC; 22h ago
 Main PID: 31273 (run)
   CGroup: /user.slice/user-1000.slice/user@1000.service/githooks.service
           ├─31273 /bin/sh /opt/githooks/bin/run
           └─31277 /usr/bin/ruby-mri /opt/githooks/app.rb -p 8008 -o 0.0.0.0

sudo

We need to configure sudo(8) to let the application user restart the service, to do that we use sudoers(5):

% adduser -d /home/githooks githooks
% EDITOR=vim visudo -f /etc/sudoers.d/githooks

And put this in the configuration for the user so the deployment script from before (#receiver) works:

# /etc/sudoers.d/githooks
Cmnd_Alias GITHOOKS_CMD = /usr/bin/systemctl restart githooks

githooks ALL = (root) NOPASSWD: GITHOOKS_CMD

Facing The World

So we got the receiver server running, awesome! But it’s not (or SHOULD NOT) be reachable from the wild yet, we need to update our firewall to let the connections come in on port 8008, in my case I’ll do it using firewall-cmd(1) (as root, of course):

% firewall-cmd --add-port 8008/tcp
% firewall-cmd --permanent --add-port 8008/tcp

So now we can reach the receiver server from the internet and can test it at will through the gitlab project’s hooks menu or your local machine:

user@localhost $ curl -X POST --data '{"push":"event"}' http://www.geckox.mx:8008/repo/myproject
{"status":"ok","code":200}

Conclusions

We have an unprivileged barebone HTTP application to handle events and trigger custom pipelines that Just Works™, but there are some things that can be easily improved, namely:

  • SSL
  • Authorization (via GitLab token)
  • Logging
  • Pipeline reporting
  • Tighten up receiver execution environment

These issues will be are addressed in the part II of this post.

UPDATE: a previous version of this post suggested to run the systemd.service as a regular user, but that causes issues ‘cuz those services only run when the user has an active session.