Caching & change detection Turborepo and Github Actions

June 1st, 2023

“Remote Caching”

I recently worked on a project with Turborepo and wanted to setup caching without using Vercel’s proprietary Remote Caching. I also wanted change detection to only deploy things that have changed. Here’s how I did it.

First, let’s just setup a basic CI workflow that runs our test suite.

name: ci
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 2
      
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: 'npm'
      
      - run: npm ci
      - run: npx turbo run test

This file will run our tests, but does not setup any turborepo caching.

Here’s how we add the turborepo cache:

Make sure you add the --cache-dir argument to your test call. This will tell turborepo where to save and restore your cache files.

name: ci
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 2

      - name: Cache turborepo
        uses: actions/cache@v3
        with:
          path: .turbo
          key: ${{ runner.os }}-turbo-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-turbo-
      
      - name: Setup Node
        uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: 'npm'
      
      - run: npm ci
      - run: npx turbo run test --cache-dir=.turbo

Change Detection

Alright, we have that setup, but how do we do change detection for deployments? There are honestly several ways to do this but the most generic and simplest way is to use turborepo to detect if a package has changed and deploy it.

Let’s start with defining our own custom Github Action that we can use within our repo for change detection.

You do have to put it within a folder and the file must be called _action.yml`

name: Has Changed?
description: Checks if a Turborepo Workspace has changed
inputs:
  workspace_name:
    required: true
    description: |-
      Name of Turborepo workspace
  from_ref:
    required: true
    description: |-
      Github Ref to detect changes from
  to_ref:
    required: true
    description: |-
      Github ref to detect changes to
  cache_dir:
    required: false
    default: .turbo
    description: |-
      Custom cache directory for turborepo
  turbo_version:
    required: false
    default: 1.9.3
    description: |-
      Turborepo version
  force:
    required: false
    default: false
    description: |-
      Used to force this action to return true

outputs:
  changed:
    description: |-
      'true' or 'false' value indicating whether the workspace changed
    value: ${{ steps.turbo_check_changed.outputs.changed }}
runs:
  using: 'composite'
  steps:
    - name: Setup Node.js environment
      uses: actions/setup-node@v3
      with:
        node-version: 18
        cache: 'npm'
    - run: npm install -g turbo@${{ inputs.turbo_version }}
      shell: bash
    - id: turbo_check_changed
      shell: bash
      run: |
        if [[ "${{ inputs.force }}" == 'true' ]]; then
          echo "changed=true" >> $GITHUB_OUTPUT
        else
          HAS_CHANGED=$(npx turbo build --cache-dir=${{ inputs.cache_dir }} --filter="${{ inputs.workspace_name }}...[${{ inputs.from_ref }}...${{ inputs.to_ref }}]" --dry-run=json | jq ".packages|any(. == \"${{ inputs.workspace_name }}\")")
          echo "changed=${HAS_CHANGED}" >> $GITHUB_OUTPUT
        fi

This code gets pretty complex in the last bash script. Basically what we are doing is npx turbo build and having it run a “dry run” of the build. It won’t actually build the packages, it is just going to tell you what it will do. We set the output to json using the --dry-run=json argument. This allows us to detect which packages would be built and we can assume have changed.

We use the "echo changed=<value>"" >> $GITHUB_OUTPUT syntax to set the output for the action. The output can be used in our deployment workflow to decide if we should deploy it. Let’s dive into that code.

name: ci
on:
  push:
    branches: ['main']
  workflow_dispatch:

concurrency:
  group: ${{ github.workflow }}
  cancel-in-progress: false

jobs:
  ci:
    uses: ./.github/workflows/ci.yml # use our exisiting CI workflow to run tests

  deploy:
    runs-on: ubuntu-latest
    needs: [ci] # make sure our tests pass before trying to deploy

    steps:
      - name: Checkout code
        uses: actions/checkout@v3
        with:
          fetch-depth: 2

      - id: has-changed
        uses: ./.github/actions/has-changed # note: this the folder name we stored the "action.yml" file in earlier
        with:
          workspace_name: <your-package.json-name>
          from_ref: ${{ github.ref_name }}
          to_ref: HEAD^1
          force: ${{ github.event_name == 'workflow_dispatch' }} # this flag forces the package to be marked as changed so it gets deployed. This is useful when you manually trigger a deploy like with a workflow_dispatch event

      - name: Cache turborepo
        if: steps.has-changed.outputs.changed == 'true' # we only run this if the package has changed
        uses: actions/cache@v3
        with:
          path: .turbo
          key: ${{ runner.os }}-turbo-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-turbo-
      
      - name: Setup Node
        if: steps.has-changed.outputs.changed == 'true'
        uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: 'npm'
      
      - run: npm ci
        if: steps.has-changed.outputs.changed == 'true'

      - run: <insert your deploy command>
        if: steps.has-changed.outputs.changed == 'true'

There is a lot here to unpack, but this workflow will only deploy if our code has actually changed. It also allows us to use Github’s workflow dispatch feature to force deployments if we need to redeploy for some reason.

Let me know what you think or if you have any suggestions to make this better!