Optimize Docker build image in GitHub Actions

Building docker images on every deployment can be time consuming but there are several ways to speed up the process.

The Setup

At my current job we have a CI/CD pipeline that builds a Docker image that is later deployed to replace the current version in production. It is a multi-step process (workflows, in GH parlance) that used to consist of:

Run CI workflow (~50 minutes)
1. Make sure application is installable (using cache to optimize dependencies downloads)
2. Unit tests
3. Integration tests (selenium)
Run CD workflow (~30 minutes)
1. Build image: some dependencies need to be compiled (~25 minutes)
2. Deploy to production

The second workflow is run automatically by GH Actions when the first workflow completes, i.e., on: workflow_run in the Workflow spec, so we would need to wait for about 1 hour and a half before we actually see our changes in production.

The Refactor

Workflow graph generated by GitHub Actions. — After refactoring, the CI workflow diagram looks like this.

Since we use Docker for the build process, there’s already a tried and tested cache layer that I know we were not taking advantage of. After ducking around a bit, I arrived at Docker’s GitHub Actions cache documentation which outlines the (currently experimental) integration with Actions’ cache. To optimize image caching I decided to split the image into two:

base: install system dependencies, e.g., runtimes and dependencies that need to be compiled and don’t change as often, e.g., HTTP server.
application: install dependencies that can simply be downloaded or change more often.

After testing locally this reduced the build process from 20 to ~5 minutes, but when merged to the main branch and tested in the GH Actions servers, the cache didn’t work as expected.

This is because GH Actions cache was being created on the feature branch but, after merging, it would not be picked up by the main branch build process. There were multiple ways to solve this but I decided to restructure the pipeline to run some steps in parallel:

Run CI workflow (~50 minutes): all steps in parallel
1. Build image only for main branch
2. Unit tests
3. Integration tests
Run CD workflow (~5 minutes)
1. Deploy to production

With the new layout the CI process still takes a long time (due to selenium tests) but the whole pipeline runtime was reduced by about 30 minutes.

The above mentioned pipeline doesn’t illustrate that the “Build image” step is actually building both the base image and the application image which can be improved further by NOT building the base (i.e., go straight to cache) image unless specific files changed. In our case, we want to build the base image only if the base Dockerfile changes. To achieve this, I used the dorny/paths-filter action to detect changes on specific files, further reducing the processing needed to build the application image.

The Rails

This setup was conceived specifically for a Ruby on Rails application with several gems that require a compilation with native extensions which is a redundant step for dependencies like puma or sassc that rarely change. So at the end of Dockerfile.base, after all “OS” dependencies have been installed, we install these gems individually which will become CACHED layers for the image build process, e.g.:

FROM --platform=linux/amd64 debian:12-slim AS geckox

RUN apt-get update -q && apt-get install -qqqqy ...

...

RUN gem install --no-document puma -v 6.4.0
RUN gem install --no-document sassc -v 2.4.0

As a rule of thumb (which I break when my setup grows), I DO NOT include ADD or COPY statements in my Dockerfile.base file to avoid unintentional layer cache expiration; I add those to the Dockerfile.app image which usually copies the application code to the image.

The CI Workflow

After all was said and done, the CI workflow looked something like this:

name: CI
on: push

jobs:
  setup:
    steps:
      - Download dependencies...

  tests:
    steps:
      - ...

  integration_tests:
    steps:
      - ...

  # Build docker images only for main
  build_base_image:
    if: ${{ github.ref == 'refs/heads/main' }}
    env:
      REGISTRY: ghcr.io
      IMAGE_NAME: orgname/geckox/geckox-base-image
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      # dorny/paths-filter (git) errors out because of "dubious ownership in repository"
      - name: "Change repository directory permissions"
        run: |
          git config --global --add safe.directory /data/runners/geckox/work/geckox/geckox || true

      # https://github.com/dorny/paths-filter
      - name: Check Dockerfile.base
        id: changes_filter
        uses: dorny/paths-filter@v2
        with:
          filters: |
            dockerfile_base:
              - added|modified: 'build/Dockerfile.base'

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Skip image build process for Dockerfile.base
        if: steps.changes_filter.outputs.dockerfile_base == 'false'
        run: echo "Skipping..."

      # https://github.com/docker/build-push-action/tree/v4
      - name: Build and push Docker image
        if: steps.changes_filter.outputs.dockerfile_base == 'true'
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
          file: build/Dockerfile.base
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest

  build_image:
    needs: build_base_image
    if: ${{ github.ref == 'refs/heads/main' }}
    env:
      REGISTRY: ghcr.io
      IMAGE_NAME: orgname/geckox/geckox-app
      GITHUB_SHA: ${{ github.sha }}
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Log in to the Container registry
        uses: docker/login-action@v2
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Set IMAGE_TAG variable
        run: |
          echo "IMAGE_TAG=$GITHUB_SHA"
          echo "IMAGE_TAG=$GITHUB_SHA" >> $GITHUB_ENV

      # https://github.com/docker/build-push-action/tree/v4
      - name: Build and push Docker image
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          cache-from: type=gha
          cache-to: type=gha,mode=max
          file: build/Dockerfile.app
          build-args: |
            BASE_TAG=latest
          tags: |
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }}