Caching & change detection Turborepo and Github Actions
“Remote Caching”
I recently worked on a project with Turborepo and wanted to setup caching without using Vercel’s proprietary Remote Caching. I also wanted change detection to only deploy things that have changed. Here’s how I did it.
First, let’s just setup a basic CI workflow that runs our test suite.
name: ci
on:
pull_request:
types: [opened, synchronize]
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 2
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: 18
cache: 'npm'
- run: npm ci
- run: npx turbo run test
This file will run our tests, but does not setup any turborepo caching.
Here’s how we add the turborepo cache:
Make sure you add the --cache-dir
argument to your test call. This will tell turborepo where to save and restore your cache files.
name: ci
on:
pull_request:
types: [opened, synchronize]
jobs:
test:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 2
- name: Cache turborepo
uses: actions/cache@v3
with:
path: .turbo
key: ${{ runner.os }}-turbo-${{ github.sha }}
restore-keys: |
${{ runner.os }}-turbo-
- name: Setup Node
uses: actions/setup-node@v3
with:
node-version: 18
cache: 'npm'
- run: npm ci
- run: npx turbo run test --cache-dir=.turbo
Change Detection
Alright, we have that setup, but how do we do change detection for deployments? There are honestly several ways to do this but the most generic and simplest way is to use turborepo to detect if a package has changed and deploy it.
Let’s start with defining our own custom Github Action that we can use within our repo for change detection.
You do have to put it within a folder and the file must be called _action.yml`
name: Has Changed?
description: Checks if a Turborepo Workspace has changed
inputs:
workspace_name:
required: true
description: |-
Name of Turborepo workspace
from_ref:
required: true
description: |-
Github Ref to detect changes from
to_ref:
required: true
description: |-
Github ref to detect changes to
cache_dir:
required: false
default: .turbo
description: |-
Custom cache directory for turborepo
turbo_version:
required: false
default: 1.9.3
description: |-
Turborepo version
force:
required: false
default: false
description: |-
Used to force this action to return true
outputs:
changed:
description: |-
'true' or 'false' value indicating whether the workspace changed
value: ${{ steps.turbo_check_changed.outputs.changed }}
runs:
using: 'composite'
steps:
- name: Setup Node.js environment
uses: actions/setup-node@v3
with:
node-version: 18
cache: 'npm'
- run: npm install -g turbo@${{ inputs.turbo_version }}
shell: bash
- id: turbo_check_changed
shell: bash
run: |
if [[ "${{ inputs.force }}" == 'true' ]]; then
echo "changed=true" >> $GITHUB_OUTPUT
else
HAS_CHANGED=$(npx turbo build --cache-dir=${{ inputs.cache_dir }} --filter="${{ inputs.workspace_name }}...[${{ inputs.from_ref }}...${{ inputs.to_ref }}]" --dry-run=json | jq ".packages|any(. == \"${{ inputs.workspace_name }}\")")
echo "changed=${HAS_CHANGED}" >> $GITHUB_OUTPUT
fi
This code gets pretty complex in the last bash script. Basically what we are doing is npx turbo build
and having it run a “dry run” of the build. It won’t actually build the packages, it is just going to tell you what it will do.
We set the output to json using the --dry-run=json
argument. This allows us to detect which packages would be built and we can assume have changed.
We use the "echo changed=<value>"" >> $GITHUB_OUTPUT
syntax to set the output for the action. The output can be used in our deployment workflow to decide if we should deploy it. Let’s dive into that code.
name: ci
on:
push:
branches: ['main']
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
jobs:
ci:
uses: ./.github/workflows/ci.yml # use our exisiting CI workflow to run tests
deploy:
runs-on: ubuntu-latest
needs: [ci] # make sure our tests pass before trying to deploy
steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 2
- id: has-changed
uses: ./.github/actions/has-changed # note: this the folder name we stored the "action.yml" file in earlier
with:
workspace_name: <your-package.json-name>
from_ref: ${{ github.ref_name }}
to_ref: HEAD^1
force: ${{ github.event_name == 'workflow_dispatch' }} # this flag forces the package to be marked as changed so it gets deployed. This is useful when you manually trigger a deploy like with a workflow_dispatch event
- name: Cache turborepo
if: steps.has-changed.outputs.changed == 'true' # we only run this if the package has changed
uses: actions/cache@v3
with:
path: .turbo
key: ${{ runner.os }}-turbo-${{ github.sha }}
restore-keys: |
${{ runner.os }}-turbo-
- name: Setup Node
if: steps.has-changed.outputs.changed == 'true'
uses: actions/setup-node@v3
with:
node-version: 18
cache: 'npm'
- run: npm ci
if: steps.has-changed.outputs.changed == 'true'
- run: <insert your deploy command>
if: steps.has-changed.outputs.changed == 'true'
There is a lot here to unpack, but this workflow will only deploy if our code has actually changed. It also allows us to use Github’s workflow dispatch feature to force deployments if we need to redeploy for some reason.
Let me know what you think or if you have any suggestions to make this better!