Github Actions Protobuf
My GitHub project rec requires Google’s Protocol Buffers to build. I want to add a GitHub Action to verify my project builds, triggered by a push. Google provides pre-built binaries as a means to use the library, but I need the protobuf runtime as well, which means I must compile the library from source. This presents a problem, because it takes ~30 minutes to build the protobuf library from scratch, and GitHub Actions always run starting from a clean virtual environment.
A request was made to add protobufs to the default GitHub Action runner, but was denied with the suggestion to build from source once and cache the result. This post covers the creation of a GitHub Action to build protobuf from source and cache it for subsequent runs.
Create a Simple Protobuf Application
To test the action, I’ll be using a simple protobuf application. The application has three files: main.cpp, person.proto, and Makefile. See the code on GitHub, or view the files below.
// main.cpp
#include <iostream>
#include "person.pb.h"
int main() {
Person p;
p.set_name("Person");
p.set_id(1);
std::cout << "Hello, " << p.DebugString() << std::endl;
}
// person.proto
syntax = "proto3";
message Person {
string name = 1;
int32 id = 2;
string email = 3;
}
# Makefile
all:
protoc --proto_path=. --cpp_out . person.proto
g++ -std=c++17 person.pb.cc -c -o person.o
g++ -std=c++17 main.cpp -c -o main.o
g++ person.o main.o -o main -lprotobuf
Create a GitHub Action
Create an initial workflow file in your project at .github/workflows/build.yml. Initially, the action will only run make.
name: build
on: push
jobs:
build-ubuntu:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: make
run: make
Pushing this action to a GitHub repository along with the project files above will result in an error specifying that protoc cannot be found.
Install protoc
Google provides a pre-built protoc binary, but the binary only provides compilation support. We need the runtime system as well since we’ll be building and linking our code. Instead, we’ll use GitHub Actions to build the protobuf library from source. Modify build.yml to download, compile, and install the protobuf library.
name: build
on: push
jobs:
build-ubuntu:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Build protobuf library
run: |
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./autogen.sh # run autogen twice, see below
./configure
make
make check
sudo make install
sudo ldconfig
- name: make
run: make
The build instructions are pulled directly from the protobuf library. Due to a bug with the autogen script, autogen.sh needs to be run twice.
Now, the build should succeed and the action should pass.
But it took 25 minutes to build the protobuf library! Every future commit will rebuild the library, giving us a minimum build time of 25 minutes, way too long for such a simple project.
Cache protobuf build
To speed up protobuf installation, we can cache the build state and reuse it on future invocations. A cache action exists that will cache folders for future builds. To use it, we provide a path to cache and a key to identify the cached data. Modify build.yml to cache the protobuf folder.
name: build
on: push
jobs:
build-ubuntu:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Cache protobuf library
id: cache-protobuf
uses: actions/cache@v1
with:
path: protobuf
key: ${{ runner.os }}-protobuf
- name: Build protobuf library
run: |
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./autogen.sh
./configure
make
make check
sudo make install
sudo ldconfig
- name: make
run: make
The runner.os environment variable is set based on the operating system running the job, and makes sure cached build files from a Linux environment don’t get loaded on a Windows environment. Running on ubuntu-latest, the cache key will be Linux-protobuf. It’s also important to provide an ID so we can check the cache result in future steps.
When the action first runs, the runner will attempt to restore cached data with the key Linux-protobuf. If no cached data is found and the action completes successfully, the protobuf folder will be cached for future use.
Currently, we never check whether the protobuf folder has been cached and instead always rebuild it, negating any benefit of caching. Fortunately, GitHub Actions provides a way to check for a cache hit and selectively run a step.
Modify build.yml to only build the protobuf library if a cached result is not found.
name: build
on: push
jobs:
build-ubuntu:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Cache protobuf library
id: cache-protobuf
uses: actions/cache@v1
with:
path: protobuf
key: ${{ runner.os }}-protobuf
- name: Build protobuf library
if: steps.cache-protobuf.outputs.cache-hit != 'true'
run: |
git clone https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./autogen.sh
./configure
make
make check
- name: Install protobuf library
run: |
cd protobuf
sudo make install
sudo ldconfig
- name: make
run: make
Note that building and installing have been split into two steps. The installation step is always necessary to install the protoc binary and supporting runtime files to a location in the PATH.
Running the final GitHub Action with a cached protobuf folder results in greatly improved build times.
Caching the protobuf build reduces build times from 25 minutes to ~45 seconds (the amount of time it takes to load the protobuf folder from cache). See the result of a build where protoc has been cached, versus one where it has not been cached.
It’s also possible to move the protobuf build logic from the GitHub Action build file to a separate installation script. See rec/install for an example.