Lee Cheng Hui

Docker layer caching in Woodpecker CI

Table of Contents

tldr: SOURCE_DATE_EPOCH value changes will invalidate WORKDIR’s cache.

The problem

Recently, I switched my CI to Woodpecker, amid Cirrus CI’s shutdown. I’m both happy and sad, happy for the team at Cirrus Lab for joining one of the prestigious AI companies, and sad because the market lost such an amazing product.

Back to the topic, I was setting up the workflow file for building a Docker image, which includes caching. Here is my config, and I’m using the official Docker Buildx plugin:

variables:
  - &docker_repo "chenghuilee/url-loader"

steps:
  - name: build-and-push-docker-image
    image: woodpeckerci/plugin-docker-buildx
    settings:
      repo: *docker_repo
      platforms: linux/amd64
      tags:
        - latest
        - "${CI_COMMIT_SHA:0:8}"
      username: chenghuilee
      password:
        from_secret: docker_hub_password
      cache_to: type=registry,ref=chenghuilee/url-loader:buildcache,mode=max
      cache_from: type=registry\\,ref=chenghuilee/url-loader:buildcache

when:
  - event: push
    branch: main

I’m using the registry as my cache storage, as you can see from the part type=registry. However, the cache missed every time, despite having no change to my application dependency. Here is the build log, reconstructed based on chronological order.

#6 [internal] load build context
#6 transferring context: 5.57kB done
#6 DONE 0.0s

#7 [builder 1/6] FROM docker.io/library/python:3.14-slim@sha256:c845af9399020c7e562969a13689e929074a10fd057acd1b1fad06a2fb068e97
#7 resolve docker.io/library/python:3.14-slim@sha256:c845af9399020c7e562969a13689e929074a10fd057acd1b1fad06a2fb068e97 done
<omitted>
#7 DONE 1.9s

## cache from registry is imported
#8 importing cache manifest from chenghuilee/url-loader:buildcache
#8 inferred cache manifest type: application/vnd.oci.image.manifest.v1+json done
#8 DONE 2.6s

#9 [auth] chenghuilee/url-loader:pull token for registry-1.docker.io
#9 DONE 0.0s


#11 [builder 2/6] WORKDIR /build
#11 DONE 0.2s

## cache miss, not good.
#12 [builder 3/6] RUN apt-get update && apt-get install -y --no-install-recommends     gcc     libxml2-dev     libxslt-dev     zlib1g-dev     && rm -rf /var/lib/apt/lists/*
#12 5.839 Fetched 60.9 MB in 1s (92.3 MB/s)
#12 5.870 Selecting previously unselected package libsframe1:amd64.
#12 5.870 (Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%

I decided to look at the buildx command that the plugin generates, and here it is:

docker buildx build --rm=true -f Dockerfile . --pull=true --cache-from type=registry,ref=chenghuilee/url-loader:buildcache --cache-to type=registry,ref=chenghuilee/url-loader:buildcache,mode=max --build-arg DOCKER_IMAGE_CREATED=2026-05-28T13:37:43Z --build-arg SOURCE_DATE_EPOCH=1779975090 --output type=image,push=true,rewrite-timestamp=true --platform linux/amd64 -t chenghuilee/url-loader:latest -t chenghuilee/url-loader:5f01fe32 --label org.opencontainers.image.created=2026-05-28T13:37:36Z --label org.opencontainers.image.source=https://codeberg.org/chenghui-lee/url-loader.git --label org.opencontainers.image.url=https://codeberg.org/chenghui-lee/url-loader --label org.opencontainers.image.revision=5f01fe32ae3c4cf79dc79f5d4f293dbf9e35067a

That’s quite long for a simple build, I thought. Looking at the command, some parts caught my attention:

My immediate thought was that the files’ modification time was changed and caused the layer hash to differ across commits, which eventually led to a cache miss. However, upon further investigation, I realized the mod time is not taken into account when calculating the cache checksum, source here:

The modification time of a file (mtime) is not taken into account when calculating the cache checksum. If only the mtime of the copied files have changed, the cache is not invalidated.

Scrolling down the official docs, I found another section:

The WORKDIR instruction respects the SOURCE_DATE_EPOCH build argument when determining cache validity. Changing SOURCE_DATE_EPOCH between builds invalidates the cache for WORKDIR and all subsequent instructions.

SOURCE_DATE_EPOCH sets timestamps for files created during the build. If you set this to a dynamic value like a Git commit timestamp, the cache breaks with each commit. This is expected behavior when tracking build provenance.

Bingo. I was using WORKDIR in my Dockerfile during that time and the SOURCE_DATE_EPOCH caused my cache to fail on the second step onwards:

# Build stage
FROM python:3.14-slim AS builder

WORKDIR /build

# Install build dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    gcc \
    libxml2-dev \
    libxslt-dev \
    zlib1g-dev \
    && rm -rf /var/lib/apt/lists/*

# and so on...

I tried to configure the build_args settings of the buildx plugin, but it seems like the SOURCE_DATE_EPOCH argument is set regardless.

steps:
  - name: build-and-push-docker-image
    image: woodpeckerci/plugin-docker-buildx
    settings:
      repo: *docker_repo
      platforms: linux/amd64
      tags:
        - latest
        - "${CI_COMMIT_SHA:0:8}"
      username: chenghuilee
      password:
        from_secret: docker_hub_password
      cache_to: type=registry,ref=chenghuilee/url-loader:buildcache,mode=max
      cache_from: type=registry\\,ref=chenghuilee/url-loader:buildcache
      output: type=image,push=true   # override the default that adds rewrite-timestamp
      build_args:
        - SOURCE_DATE_EPOCH=0 # to avoid cache invalidation, https://docs.docker.com/build/cache/invalidation/

docker buildx build --rm=true -f Dockerfile . --pull=true --cache-from type=registry,ref=chenghuilee/url-loader:buildcache --cache-to type=registry,ref=chenghuilee/url-loader:buildcache,mode=max --build-arg SOURCE_DATE_EPOCH=1780061229 --build-arg *=SOURCE_DATE_EPOCH=0 --build-arg DOCKER_IMAGE_CREATED=2026-05-29T14:13:35Z --output type=image,push=true --platform linux/amd64 -t chenghuilee/url-loader:latest -t chenghuilee/url-loader:c38f9bd0 --label org.opencontainers.image.created=2026-05-29T14:13:33Z --label org.opencontainers.image.source=https://codeberg.org/chenghui-lee/url-loader.git --label org.opencontainers.image.url=https://codeberg.org/chenghui-lee/url-loader --label org.opencontainers.image.revision=c38f9bd09759cbf0429f672715825c8986d61fd9

and the cache missed again…

So I had no choice but not to use WORKDIR in my Dockerfile, and… it works! Here’s the satisfying build log:

#7 [builder 1/4] FROM docker.io/library/python:3.14-slim@sha256:c845af9399020c7e562969a13689e929074a10fd057acd1b1fad06a2fb068e97
#7 resolve docker.io/library/python:3.14-slim@sha256:c845af9399020c7e562969a13689e929074a10fd057acd1b1fad06a2fb068e97 done
#7 DONE 0.0s

#8 importing cache manifest from chenghuilee/url-loader:buildcache
#8 inferred cache manifest type: application/vnd.oci.image.manifest.v1+json done
#8 DONE 2.5s

#9 [auth] chenghuilee/url-loader:pull token for registry-1.docker.io
#9 DONE 0.0s

#11 [builder 2/4] RUN apt-get update && apt-get install -y --no-install-recommends gcc libxml2-dev libxslt-dev zlib1g-dev && rm -rf /var/lib/apt/lists/*
#11 CACHED

#12 [builder 3/4] RUN python -m venv /opt/venv
#12 CACHED

Summary

Using a prebuilt plugin is nice, but sometimes the default values may not be the best for everyone. Values like SOURCE_DATE_EPOCH and rewrite-timestamp are nice for build reproducibility, but come with hidden quirks that break my workflow in this case. Perhaps I should write my own plugin, or even better, just pure bash command? That’s a question for my future self if my workflow breaks again, lol.

↑