Github Actions Protobuf

My GitHub project rec requires Google’s Protocol Buffers to build. I want to add a GitHub Action to verify my project builds, triggered by a push. Google provides pre-built binaries as a means to use the library, but I need the protobuf runtime as well, which means I must compile the library from source. This presents a problem, because it takes ~30 minutes to build the protobuf library from scratch, and GitHub Actions always run starting from a clean virtual environment.

A request was made to add protobufs to the default GitHub Action runner, but was denied with the suggestion to build from source once and cache the result. This post covers the creation of a GitHub Action to build protobuf from source and cache it for subsequent runs.

Create a Simple Protobuf Application

To test the action, I’ll be using a simple protobuf application. The application has three files: main.cpp, person.proto, and Makefile. See the code on GitHub, or view the files below.

// main.cpp

#include <iostream>

#include "person.pb.h"

int main() {
  Person p;
  p.set_name("Person");
  p.set_id(1);
  std::cout << "Hello, " << p.DebugString() << std::endl;
}
// person.proto

syntax = "proto3";

message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}
# Makefile

all:
	protoc --proto_path=. --cpp_out . person.proto
	g++ -std=c++17 person.pb.cc -c -o person.o
	g++ -std=c++17 main.cpp -c -o main.o
	g++ person.o main.o -o main -lprotobuf

Create a GitHub Action

Create an initial workflow file in your project at .github/workflows/build.yml. Initially, the action will only run make.

name: build
on: push
jobs:
   build-ubuntu:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - name: make
         run: make

Pushing this action to a GitHub repository along with the project files above will result in an error specifying that protoc cannot be found.

GitHub Action build failed

Install protoc

Google provides a pre-built protoc binary, but the binary only provides compilation support. We need the runtime system as well since we’ll be building and linking our code. Instead, we’ll use GitHub Actions to build the protobuf library from source. Modify build.yml to download, compile, and install the protobuf library.

name: build
on: push
jobs:
   build-ubuntu:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - name: Build protobuf library
         run: |
           git clone https://github.com/protocolbuffers/protobuf.git
           cd protobuf
           git submodule update --init --recursive
           ./autogen.sh
           ./autogen.sh  # run autogen twice, see below
           ./configure
           make
           make check
           sudo make install
           sudo ldconfig
       - name: make
         run: make

The build instructions are pulled directly from the protobuf library. Due to a bug with the autogen script, autogen.sh needs to be run twice.

Now, the build should succeed and the action should pass.

GitHub Action build succeeded

But it took 25 minutes to build the protobuf library! Every future commit will rebuild the library, giving us a minimum build time of 25 minutes, way too long for such a simple project.

Cache protobuf build

To speed up protobuf installation, we can cache the build state and reuse it on future invocations. A cache action exists that will cache folders for future builds. To use it, we provide a path to cache and a key to identify the cached data. Modify build.yml to cache the protobuf folder.

name: build
on: push
jobs:
   build-ubuntu:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - name: Cache protobuf library
         id: cache-protobuf
         uses: actions/cache@v1
         with:
           path: protobuf
           key: ${{ runner.os }}-protobuf
       - name: Build protobuf library
         run: |
           git clone https://github.com/protocolbuffers/protobuf.git
           cd protobuf
           git submodule update --init --recursive
           ./autogen.sh
           ./autogen.sh
           ./configure
           make
           make check
           sudo make install
           sudo ldconfig
       - name: make
         run: make

The runner.os environment variable is set based on the operating system running the job, and makes sure cached build files from a Linux environment don’t get loaded on a Windows environment. Running on ubuntu-latest, the cache key will be Linux-protobuf. It’s also important to provide an ID so we can check the cache result in future steps.

When the action first runs, the runner will attempt to restore cached data with the key Linux-protobuf. If no cached data is found and the action completes successfully, the protobuf folder will be cached for future use.

Currently, we never check whether the protobuf folder has been cached and instead always rebuild it, negating any benefit of caching. Fortunately, GitHub Actions provides a way to check for a cache hit and selectively run a step.

Modify build.yml to only build the protobuf library if a cached result is not found.

name: build
on: push
jobs:
   build-ubuntu:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
       - name: Cache protobuf library
         id: cache-protobuf
         uses: actions/cache@v1
         with:
           path: protobuf
           key: ${{ runner.os }}-protobuf
       - name: Build protobuf library
         if: steps.cache-protobuf.outputs.cache-hit != 'true'
         run: |
           git clone https://github.com/protocolbuffers/protobuf.git
           cd protobuf
           git submodule update --init --recursive
           ./autogen.sh
           ./autogen.sh
           ./configure
           make
           make check
       - name: Install protobuf library
         run: |
           cd protobuf
           sudo make install
           sudo ldconfig
       - name: make
         run: make

Note that building and installing have been split into two steps. The installation step is always necessary to install the protoc binary and supporting runtime files to a location in the PATH.

Running the final GitHub Action with a cached protobuf folder results in greatly improved build times.

Improved build times with cached protobuf library

Caching the protobuf build reduces build times from 25 minutes to ~45 seconds (the amount of time it takes to load the protobuf folder from cache). See the result of a build where protoc has been cached, versus one where it has not been cached.

It’s also possible to move the protobuf build logic from the GitHub Action build file to a separate installation script. See rec/install for an example.