Write Your Own CI/CD pipeline
I was bored and wanted to get some kicks so I decided to automate the deployment of a cron(8) job I have running on a server, so in this post we see how to setup a CI/CD pipeline relying on GitLab Webhooks and custom deployment scripts.
NOTE: you can check the source code for this post at juankman94/githooks-receiver.
Disclosure: I did this mostly because I wanted to practice my ruby and configure a server for the fun of it, take that into account when reading this.
This exercise assumes:
- An application is already deployed somewhere
- Development is still being made
- Proper/secure configuration exists
- The application server uses systemd(1)
The CI/CD flow used in this exercise goes like this:
- Development is done locally
- Changes are commited and pushed to GitLab (or any git provider that supports webhooks)
- GitLab notifies the receiver of the Push Event
- The receiver does something™
The Receiver
GitLab shows a basic WEBrick server to receive the event, but we’ll go with Sinatra ‘cuz I’ve never used it before. From the website: “Sinatra is a DSL for quickly creating web applications in Ruby with minimal effort”. Having said that, let’s create the project by declaring the dependencies in a Gemfile:
# Gemfile
source 'https://rubygems.org'
gem 'sinatra'
Install dependencies by running
bundle install
(click for details)
For Fedora 30 I had to install some dependencies (to build with native extensions), namely:
% dnf install -y ruby ruby-dev redhat-rpm-config
% dnf groupinstall -y "Development Tools"
The GitLab documentation states that it doesn’t care the response status and that the receiver SHOULD reply as soon as possible:
Your endpoint should send its HTTP response as fast as possible. If you wait too long, GitLab may decide the hook failed and retry it.
So the receiver (app.rb
) looks like this:
# frozen_literal_string: true
require "sinatra"
ALLOWED_REPOS = %w( myproject )
DEPLOYMENTS = {
"myproject" => "./deployments/myproject",
}
post "/repo/:repo" do
repo = params["repo"]
resp = {
status: "ok",
code: 200,
}
if ALLOWED_REPOS.include?(repo)
deploy_pid = Process.spawn(DEPLOYMENTS[repo])
Process.detach(deploy_pid)
else
status 403
resp = { status: "forbidden", code: 403 }
end
body JSON.dump(resp)
end
As we can see in the DEPLOYMENTS
hash, we map a repository name to a
deployment script (or command of your choice, really). In the snippet above it
is stated that the script deployments/myproject
will be executed
in the background, so let’s see what it does:
#!/bin/sh
# Update `myproject` source code & restart service
HOOKS_DIR="$( dirname "$(realpath $0)" )"/..
LOG_FILE="$HOOKS_DIR/log/myproject.log"
MYPROJECT_DIR="$HOME/myproject"
VERSION="$1"
function log_message() {
echo "[$(date --iso-8601=seconds)] $@" | tee -a $LOG_FILE
}
mkdir -p "$(dirname "$LOG_FILE")"
log_message "downloading version $VERSION"
log_message "changing directory to $MYPROJECT_DIR"
cd "$MYPROJECT_DIR"
log_message "stashing directory"
git stash | tee -a $LOG_FILE
log_message "pulling changes"
git pull | tee -a $LOG_FILE
log_message "restarting application"
sudo systemctl restart myproject
With these files in place, we have all we need: a trigger, an HTTP server to receive the events and a script to run the pipeline. Do note that these files reveal several things from the environment:
- The
receiver
user is able to execute the deployment commands (some configuration may be needed depending on thecmd
) - We’re not making sure the deployment is successful
myproject
is running as an systemd.unit(5) service
So now let’s (manually) run the receiver
server and see how it works (we can
test webhooks
by pushing events at will from our project’s hooks menu, but not just yet):
$ bundle exec ruby app.rb -p 8008 -o 0.0.0.0
[2020-03-30 17:22:58] INFO WEBrick 1.4.2
[2020-03-30 17:22:58] INFO ruby 2.5.7 (2019-10-01)
== Sinatra (v2.0.8.1) has taken the stage on 4567 for development with backup from WEBrick
[2020-03-30 17:22:58] INFO WEBrick::HTTPServer#start: pid=22884 port=8008
We’re running the server on port 8008
but you can set whichever port you
want. And if we POST
a message we should see a log message:
$ curl -X POST --data '{"a":"b"}' http://localhost:8008/repo/myproject
{"status":"ok","code":200}
# and in the sinatra process log
...
200.200.0.146 - - [30/Mar/2020:17:23:03 +0000] "POST /repo/myproject HTTP/1.1" 200 33 0.0102
The Host
So we have the webhook receiver server, but how can we ensure that it’ll be
running to receive the notifications? For that we’ll setup a systemd(1)
service. Now, I won’t go into the details of how this works, that’s out of the
scope of this post, but we’ll get to see the results after placing files in the
proper locations.
SystemD
Since we’re running an application that will be spawning new processes from
events received from the wild, it’s better to do it with as little privileges
as possible, so we’ll run the receiver as an
systemd.unit(5)
service. First, we place the githooks.service
file in the
/etc/systemd/system/
directory:
# githooks.service
[Unit]
Description=GitLab Webhooks Receiver
DefaultDependencies=no
After=network.target
[Service]
Type=simple
ExecStart=/opt/githooks/bin/run
User=githooks
[Install]
WantedBy=default.target
With that in place, let’s load, enable (so it’ll start automatically on (re)boot) and start the service:
% systemctl daemon-reload
% systemctl status githooks
githooks.service - GitLab Webhooks Receiver
Loaded: loaded (/etc/systemd/system/githooks.service; disabled; vendor preset: enabled)
Active: inactive (dead)
% systemctl enable githooks
% systemctl start githooks
% systemctl enable githooks
% systemctl status githooks
githooks.service - GitLab Webhooks Receiver
Loaded: loaded (/etc/systemd/system/githooks.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-03-31 06:49:27 UTC; 22h ago
Main PID: 31273 (run)
CGroup: /user.slice/user-1000.slice/user@1000.service/githooks.service
├─31273 /bin/sh /opt/githooks/bin/run
└─31277 /usr/bin/ruby-mri /opt/githooks/app.rb -p 8008 -o 0.0.0.0
sudo
We need to configure sudo(8) to let the application user restart the service, to do that we use sudoers(5):
% adduser -d /home/githooks githooks
% EDITOR=vim visudo -f /etc/sudoers.d/githooks
And put this in the configuration for the user so the deployment script from before (#receiver) works:
# /etc/sudoers.d/githooks
Cmnd_Alias GITHOOKS_CMD = /usr/bin/systemctl restart githooks
githooks ALL = (root) NOPASSWD: GITHOOKS_CMD
Facing The World
So we got the receiver server running, awesome! But it’s not
(or SHOULD NOT) be reachable from the wild yet, we need to update our
firewall to let the connections come in on port 8008
, in my case I’ll do it
using firewall-cmd(1) (as root, of course):
% firewall-cmd --add-port 8008/tcp
% firewall-cmd --permanent --add-port 8008/tcp
So now we can reach the receiver server from the internet and can test it at will through the gitlab project’s hooks menu or your local machine:
user@localhost $ curl -X POST --data '{"push":"event"}' http://www.geckox.mx:8008/repo/myproject
{"status":"ok","code":200}
Conclusions
We have an unprivileged barebone HTTP application to handle events and trigger custom pipelines that Just Works™, but there are some things that can be easily improved, namely:
- SSL
- Authorization (via GitLab token)
- Logging
- Pipeline reporting
- Tighten up
receiver
execution environment
These issues will be are addressed in the
part II
of this post.
UPDATE: a previous version of this post suggested to run the
systemd.service
as a regular user, but that causes issues ‘cuz those services
only run when the user has an active session.