Preview Environment Manager
This project is built and tested for Woodpecker CI. The goal is to spawn and tear down preview environments on PR actions.
This article is not meant to be copy/pasta; you're gonna have to adapt everything to fit your actual cluster.
How it works
- The repository
fleet-previewswill be updated from the different codebases to register (or unregister) theFlux Kustomization, thus providing ephemeral environments. - The Woodpecker pipeline uses a golang image to simplify the process of deploying and tearing down those environments. When it tears down an environment, it also cleans up the container images in the registry. The golang layer has access to the API and also contains a template to quickly create environments and push the file directly to the repo so FluxCD can detect it.
- FluxCD periodically scans the
fleet-previewsrepository, deploys the resources, and tears down the kustomizations that aren't in the repo anymore.
Preview Image Tags
The pipeline must build and push the image to the registry. I used the following tag structure: preview-<pr>-<sha>. The cleanup process will automatically delete everything that starts with preview-<pr>-*, and FluxCD will detect changes and release the commit on the same endpoint.
Endpoint
It uses the application name set in FluxCD plus a slug of the branch name. For example: https://webuxlab-article-webux-pages.webux.app, where webuxlab is the application name and article-webux-pages is the slug of the branch article/webux-pages.
Requirements
- Preview Bot (Golang Container)
- Gitea
- Woodpecker CI
- Kubernetes
- FluxCD Configured
- Cert Manager
- Relevant Admin Access
- A wildcard certificate & domain name
- Network and gateway configured
- I use Cilium & Gateway API
Gitea Specifics
Repository
fleet-previews, dedicated and managed by the preview bot.
User & Team
- User:
ci-previews - Team:
ci-preview
Token
- Gitea User
ci-previewswith read/write onfleet-previewsrepository.
WoodpeckerCI Specifics
- I use the built-in container registry in Gitea.
- If you use cleanup rules for the container images, make sure they fit the naming convention of your preview images.
- For each project with this preview feature, you need a pipeline to hook on PR events (See below).
Secrets
- Container Registry
ci-previewstoken
FluxCD Specifics
- You need to configure a preview environment for each of your apps; the template allows deploying and running many PRs in parallel.
- This article assumes that FluxCD is already set up and used for your application. (The initial setup will not be covered in here.)
Domain name & Wildcard Certificate
- It depends entirely on your existing setup.
- I use Cloudflare with a dedicated domain
*.webux.app, Cert-Manager, HAProxy in TCP Passthrough, Cilium and Gateway API. (Not covered in this article)
Preview Bot Code
go.mod
module git.webux.dev/studiowebux/preview-bot
go 1.26
main.go
Using this script allows you to standardize the whole flow by templating it. Then, for each of your applications, you only need to configure FluxCD once.
// preview-bot is a Woodpecker plugin that manages a preview environment:
//
// - sync: create/update the Flux pointer file in the fleet-previews repo
// (Gitea contents API) so Flux spins the preview up.
// - teardown: delete that pointer (Flux prunes the preview) AND delete the
// preview container image `preview-<pr>` (Gitea package API).
//
// It never clones a repo and never touches the fleet repo.
//
// Configuration (Woodpecker `settings:` become PLUGIN_* env vars):
//
// app required app name / image name, e.g. "whats-my-ip"
// action required "sync" or "teardown"
// token required Gitea token: write to the previews repo only
// image_token optional Gitea token with package delete (teardown only);
// reuse the registry push token. If unset, the image
// is left for GC.
// gitea optional base URL (default https://git.webux.dev)
// repo optional previews owner/name (default studiowebux/fleet-previews)
// owner optional image/package owner (default: repo owner)
// domain optional preview base domain (default webux.app)
//
// The preview image tag is preview-<pr>-<sha> (immutable per commit): built on
// PR events, pinned by the overlay, deleted as a group (preview-<pr>-*) on PR
// close.
package main
import (
"bytes"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"regexp"
"strings"
"text/template"
"time"
)
const pointerTmpl = `apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: {{.App}}-preview-{{.Slug}}
namespace: flux-system
spec:
interval: 5m
prune: true
sourceRef:
kind: GitRepository
name: flux-system
path: ./apps/{{.App}}/overlays/preview
postBuild:
substitute:
PREVIEW_NAME: {{.Slug}}
PREVIEW_NAMESPACE: {{.App}}-{{.Slug}}
PREVIEW_HOST: {{.App}}-{{.Slug}}.{{.Domain}}
PREVIEW_IMAGE_TAG: {{.ImageTag}}
`
var nonAlnum = regexp.MustCompile(`[^a-z0-9]+`)
// slugify turns a branch name into a DNS-label-safe slug (no truncation).
func slugify(branch string) string {
s := nonAlnum.ReplaceAllString(strings.ToLower(branch), "-")
return strings.Trim(s, "-")
}
func env(key, def string) string {
if v := os.Getenv(key); v != "" {
return v
}
return def
}
func mustEnv(key string) string {
v := os.Getenv(key)
if v == "" {
log.Fatalf("missing required setting: %s", key)
}
return v
}
type config struct {
app, action, token, imageToken, gitea, repo, owner, domain, slug, imageTag string
}
func load() config {
branch := env("CI_COMMIT_SOURCE_BRANCH", os.Getenv("CI_COMMIT_BRANCH"))
if branch == "" {
log.Fatal("no branch in CI_COMMIT_SOURCE_BRANCH / CI_COMMIT_BRANCH")
}
slug := slugify(branch)
if len(slug) > 40 { // keep app-<slug> under the 63-char label limit
log.Fatalf("slug %q too long (%d); rename the branch", slug, len(slug))
}
repo := env("PLUGIN_REPO", "studiowebux/fleet-previews")
return config{
app: mustEnv("PLUGIN_APP"),
action: mustEnv("PLUGIN_ACTION"),
token: mustEnv("PLUGIN_TOKEN"),
imageToken: os.Getenv("PLUGIN_IMAGE_TOKEN"),
gitea: env("PLUGIN_GITEA", "https://git.webux.dev"),
repo: repo,
owner: env("PLUGIN_OWNER", strings.SplitN(repo, "/", 2)[0]),
domain: env("PLUGIN_DOMAIN", "webux.app"),
slug: slug,
imageTag: env("PLUGIN_IMAGE_TAG", previewTag(slug)),
}
}
// previewTag is preview-<pr>-<sha>: immutable per commit (so Flux redeploys on
// each PR push) and grouped by PR (so teardown can delete the whole group by
// prefix). Falls back to the branch slug when there is no PR number.
func previewTag(slug string) string {
sha := os.Getenv("CI_COMMIT_SHA")
group := imageGroup(slug)
if group == "" {
return "latest"
}
if sha != "" {
return group + sha
}
return strings.TrimSuffix(group, "-")
}
// imageGroup is the shared prefix for all of a PR's preview images,
// e.g. "preview-42-". Teardown deletes every version with this prefix.
func imageGroup(slug string) string {
if pr := os.Getenv("CI_COMMIT_PULL_REQUEST"); pr != "" {
return "preview-" + pr + "-"
}
if slug != "" {
return "preview-" + slug + "-"
}
return ""
}
func (c config) pointer() string {
var b bytes.Buffer
template.Must(template.New("p").Parse(pointerTmpl)).Execute(&b, map[string]string{
"App": c.app, "Slug": c.slug, "Domain": c.domain, "ImageTag": c.imageTag,
})
return b.String()
}
func (c config) path() string { return fmt.Sprintf("%s-%s.yaml", c.app, c.slug) }
// url is the public preview URL, e.g. https://webuxlab-fix-mobile.webux.app.
func (c config) url() string { return fmt.Sprintf("https://%s-%s.%s", c.app, c.slug, c.domain) }
// do sends an authenticated JSON request and returns status + body.
func do(method, url, token string, body any) (int, []byte) {
var r io.Reader
if body != nil {
buf, _ := json.Marshal(body)
r = bytes.NewReader(buf)
}
req, err := http.NewRequest(method, url, r)
if err != nil {
log.Fatal(err)
}
req.Header.Set("Authorization", "token "+token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
out, _ := io.ReadAll(resp.Body)
return resp.StatusCode, out
}
func (c config) contentsURL() string {
return fmt.Sprintf("%s/api/v1/repos/%s/contents/%s", c.gitea, c.repo, c.path())
}
// fileSHA returns the current pointer file sha, or "" if absent.
func (c config) fileSHA() string {
code, out := do(http.MethodGet, c.contentsURL(), c.token, nil)
if code == http.StatusNotFound {
return ""
}
if code != http.StatusOK {
log.Fatalf("GET %s -> %d: %s", c.path(), code, out)
}
var f struct {
SHA string `json:"sha"`
}
json.Unmarshal(out, &f)
return f.SHA
}
func (c config) sync() {
content := base64.StdEncoding.EncodeToString([]byte(c.pointer()))
msg := fmt.Sprintf("preview: %s (PR #%s)", c.path(), os.Getenv("CI_COMMIT_PULL_REQUEST"))
body := map[string]string{"content": content, "message": msg}
method := http.MethodPost // create
if sha := c.fileSHA(); sha != "" {
method = http.MethodPut // update existing
body["sha"] = sha
}
code, out := do(method, c.contentsURL(), c.token, body)
if code != http.StatusCreated && code != http.StatusOK {
log.Fatalf("%s %s -> %d: %s", method, c.path(), code, out)
}
log.Printf("synced %s (image %s)", c.path(), c.imageTag)
log.Printf("preview: %s", c.url())
}
func (c config) teardown() {
// 1. delete the Flux pointer -> Flux prunes the running preview.
if sha := c.fileSHA(); sha != "" {
msg := fmt.Sprintf("preview teardown: %s (PR #%s)", c.path(), os.Getenv("CI_COMMIT_PULL_REQUEST"))
code, out := do(http.MethodDelete, c.contentsURL(), c.token, map[string]string{"message": msg, "sha": sha})
if code != http.StatusOK {
log.Fatalf("DELETE %s -> %d: %s", c.path(), code, out)
}
log.Printf("removed pointer %s", c.path())
} else {
log.Printf("pointer %s absent", c.path())
}
// 2. delete every preview image for this PR (needs a package-scoped token).
c.deleteImages()
}
// deleteImages removes all container versions of the app whose tag starts with
// this PR's group prefix (preview-<pr>-*).
func (c config) deleteImages() {
prefix := imageGroup(c.slug)
if c.imageToken == "" || prefix == "" {
log.Printf("no image_token/prefix; leaving images %s:%s* for GC", c.app, prefix)
return
}
deleted := 0
for page := 1; ; page++ {
listURL := fmt.Sprintf("%s/api/v1/packages/%s?type=container&q=%s&page=%d&limit=50",
c.gitea, c.owner, c.app, page)
code, out := do(http.MethodGet, listURL, c.imageToken, nil)
if code != http.StatusOK {
log.Fatalf("list packages -> %d: %s", code, out)
}
var pkgs []struct{ Name, Version string }
json.Unmarshal(out, &pkgs)
if len(pkgs) == 0 {
break
}
for _, p := range pkgs {
if p.Name != c.app || !strings.HasPrefix(p.Version, prefix) {
continue
}
delURL := fmt.Sprintf("%s/api/v1/packages/%s/container/%s/%s",
c.gitea, c.owner, c.app, p.Version)
dc, dout := do(http.MethodDelete, delURL, c.imageToken, nil)
switch dc {
case http.StatusNoContent, http.StatusOK, http.StatusAccepted, http.StatusNotFound:
deleted++
default:
log.Fatalf("DELETE %s:%s -> %d: %s", c.app, p.Version, dc, dout)
}
}
}
log.Printf("removed %d image(s) matching %s:%s*", deleted, c.app, prefix)
}
func wipCommit(msg string) bool {
return strings.HasPrefix(strings.ToLower(strings.TrimSpace(msg)), "wip")
}
func main() {
log.SetFlags(0)
c := load()
switch c.action {
case "sync":
if msg := os.Getenv("CI_COMMIT_MESSAGE"); wipCommit(msg) {
log.Printf("WIP commit; skipping preview sync for %s", c.path())
return
}
c.sync()
case "teardown":
c.teardown()
default:
log.Fatalf("unknown action %q (want sync|teardown)", c.action)
}
}
Dockerfile
Adapt this Dockerfile to fit your standards.
FROM golang:1.26 AS build
RUN apt-get update && apt-get install -y --no-install-recommends upx-ucl && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod ./
COPY main.go ./
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /preview-bot .
RUN upx --best --lzma /preview-bot
FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build /preview-bot /preview-bot
USER 65532:65532
ENTRYPOINT ["/preview-bot"]
FluxCD Setup
This is only the configuration for the preview system.
- Update the
urlto fit your configuration. - Create a read-only deploy key for the
fleet-previewsrepository. - You then need to configure the key as shown below.
The Source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: previews
namespace: flux-system
spec:
interval: 1m
ref:
branch: main
url: ssh://[email protected]:22/studiowebux/fleet-previews.git
secretRef:
name: previews-read-key # read-only deploy key
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: previews
namespace: flux-system
spec:
interval: 2m
prune: true
sourceRef:
kind: GitRepository
name: previews
path: ./
The Deploy Key Structure
---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: previews-read-key
namespace: flux-system
spec:
encryptedData:
identity: REDACTED
known_hosts: REDACTED
template:
metadata:
name: previews-read-key
namespace: flux-system
Configuring your Application
/previews/preview-component
Adapt to fit your existing configuration.
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
namespace: ${PREVIEW_NAMESPACE}
commonLabels:
env: preview
preview: ${PREVIEW_NAME}
resources:
- networkpolicy.yaml # Not provided, as it depends entirely on your cluster.
patches:
- target:
kind: HTTPRoute
patch: |-
- op: replace
path: /spec/parentRefs
value:
- name: gateway-external
namespace: kube-system
sectionName: https-preview-wildcard
- op: replace
path: /spec/hostnames
value:
- ${PREVIEW_HOST}
/apps/webuxlab/overlays/preview/kustomization.yaml
Using this preview/kustomization.yaml allows spawning many PRs in parallel. You need to add this configuration for each application you want to enable the preview on.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../../../previews/preview-component
# Run the image built from THIS PR commit (Flux fills ${PREVIEW_IMAGE_TAG}).
images:
- name: git.webux.dev/studiowebux/webuxlab
newTag: ${PREVIEW_IMAGE_TAG}
Example (Application Side)
This blog uses that workflow to deploy and validate changes.
To skip the preview deployment, you can prefix your git message with
wiporWIP.
The CI looks like:
when:
- event: [pull_request, pull_request_closed]
steps:
build-preview:
image: woodpeckerci/plugin-docker-buildx:6.1.0
privileged: true
when:
- event: pull_request
settings:
repo: git.webux.dev/studiowebux/webuxlab
tags:
- preview-${CI_COMMIT_PULL_REQUEST}-${CI_COMMIT_SHA}
platforms:
- linux/amd64
registry: https://git.webux.dev
username:
from_secret: registry_username
password:
from_secret: registry_password
storage_driver: vfs
dockerfile: Dockerfile
context: .
no_cache: true
build_args:
- APP_VERSION=${CI_COMMIT_SHA}
# Only after the image exists: write/update the Flux pointer.
preview-sync:
image: git.webux.dev/studiowebux/preview-bot:latest
when:
- event: pull_request
settings:
app: webuxlab
action: sync
token:
from_secret: fleet_previews_token
# On PR close: delete the pointer (Flux prunes) AND the preview-<pr> image.
preview-teardown:
image: git.webux.dev/studiowebux/preview-bot:latest
when:
- event: pull_request_closed
settings:
app: webuxlab
action: teardown
token:
from_secret: fleet_previews_token
image_token:
from_secret: registry_password