Preview Environment Manager

This project is built and tested for Woodpecker CI. The goal is to spawn and tear down preview environments on PR actions.

This article is not meant to be copy/pasta; you're gonna have to adapt everything to fit your actual cluster.

How it works

  1. The repository fleet-previews will be updated from the different codebases to register (or unregister) the Flux Kustomization, thus providing ephemeral environments.
  2. The Woodpecker pipeline uses a golang image to simplify the process of deploying and tearing down those environments. When it tears down an environment, it also cleans up the container images in the registry. The golang layer has access to the API and also contains a template to quickly create environments and push the file directly to the repo so FluxCD can detect it.
  3. FluxCD periodically scans the fleet-previews repository, deploys the resources, and tears down the kustomizations that aren't in the repo anymore.

Preview Image Tags

The pipeline must build and push the image to the registry. I used the following tag structure: preview-<pr>-<sha>. The cleanup process will automatically delete everything that starts with preview-<pr>-*, and FluxCD will detect changes and release the commit on the same endpoint.

Endpoint

It uses the application name set in FluxCD plus a slug of the branch name. For example: https://webuxlab-article-webux-pages.webux.app, where webuxlab is the application name and article-webux-pages is the slug of the branch article/webux-pages.

Requirements

  • Preview Bot (Golang Container)
  • Gitea
  • Woodpecker CI
  • Kubernetes
  • FluxCD Configured
  • Cert Manager
  • Relevant Admin Access
  • A wildcard certificate & domain name
  • Network and gateway configured
    • I use Cilium & Gateway API

Gitea Specifics

Repository

  • fleet-previews, dedicated and managed by the preview bot.

User & Team

  • User: ci-previews
  • Team: ci-preview

Token

  • Gitea User ci-previews with read/write on fleet-previews repository.

WoodpeckerCI Specifics

  • I use the built-in container registry in Gitea.
    • If you use cleanup rules for the container images, make sure they fit the naming convention of your preview images.
  • For each project with this preview feature, you need a pipeline to hook on PR events (See below).

Secrets

  • Container Registry
  • ci-previews token

FluxCD Specifics

  • You need to configure a preview environment for each of your apps; the template allows deploying and running many PRs in parallel.
  • This article assumes that FluxCD is already set up and used for your application. (The initial setup will not be covered in here.)

Domain name & Wildcard Certificate

  • It depends entirely on your existing setup.
  • I use Cloudflare with a dedicated domain *.webux.app, Cert-Manager, HAProxy in TCP Passthrough, Cilium and Gateway API. (Not covered in this article)

Preview Bot Code

go.mod
module git.webux.dev/studiowebux/preview-bot

go 1.26
main.go

Using this script allows you to standardize the whole flow by templating it. Then, for each of your applications, you only need to configure FluxCD once.

// preview-bot is a Woodpecker plugin that manages a preview environment:
//
//   - sync:     create/update the Flux pointer file in the fleet-previews repo
//               (Gitea contents API) so Flux spins the preview up.
//   - teardown: delete that pointer (Flux prunes the preview) AND delete the
//               preview container image `preview-<pr>` (Gitea package API).
//
// It never clones a repo and never touches the fleet repo.
//
// Configuration (Woodpecker `settings:` become PLUGIN_* env vars):
//
//	app          required   app name / image name, e.g. "whats-my-ip"
//	action       required   "sync" or "teardown"
//	token        required   Gitea token: write to the previews repo only
//	image_token  optional   Gitea token with package delete (teardown only);
//	                         reuse the registry push token. If unset, the image
//	                         is left for GC.
//	gitea        optional   base URL            (default https://git.webux.dev)
//	repo         optional   previews owner/name (default studiowebux/fleet-previews)
//	owner        optional   image/package owner (default: repo owner)
//	domain       optional   preview base domain (default webux.app)
//
// The preview image tag is preview-<pr>-<sha> (immutable per commit): built on
// PR events, pinned by the overlay, deleted as a group (preview-<pr>-*) on PR
// close.
package main

import (
	"bytes"
	"encoding/base64"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"regexp"
	"strings"
	"text/template"
	"time"
)

const pointerTmpl = `apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: {{.App}}-preview-{{.Slug}}
  namespace: flux-system
spec:
  interval: 5m
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/{{.App}}/overlays/preview
  postBuild:
    substitute:
      PREVIEW_NAME: {{.Slug}}
      PREVIEW_NAMESPACE: {{.App}}-{{.Slug}}
      PREVIEW_HOST: {{.App}}-{{.Slug}}.{{.Domain}}
      PREVIEW_IMAGE_TAG: {{.ImageTag}}
`

var nonAlnum = regexp.MustCompile(`[^a-z0-9]+`)

// slugify turns a branch name into a DNS-label-safe slug (no truncation).
func slugify(branch string) string {
	s := nonAlnum.ReplaceAllString(strings.ToLower(branch), "-")
	return strings.Trim(s, "-")
}

func env(key, def string) string {
	if v := os.Getenv(key); v != "" {
		return v
	}
	return def
}

func mustEnv(key string) string {
	v := os.Getenv(key)
	if v == "" {
		log.Fatalf("missing required setting: %s", key)
	}
	return v
}

type config struct {
	app, action, token, imageToken, gitea, repo, owner, domain, slug, imageTag string
}

func load() config {
	branch := env("CI_COMMIT_SOURCE_BRANCH", os.Getenv("CI_COMMIT_BRANCH"))
	if branch == "" {
		log.Fatal("no branch in CI_COMMIT_SOURCE_BRANCH / CI_COMMIT_BRANCH")
	}
	slug := slugify(branch)
	if len(slug) > 40 { // keep app-<slug> under the 63-char label limit
		log.Fatalf("slug %q too long (%d); rename the branch", slug, len(slug))
	}
	repo := env("PLUGIN_REPO", "studiowebux/fleet-previews")
	return config{
		app:        mustEnv("PLUGIN_APP"),
		action:     mustEnv("PLUGIN_ACTION"),
		token:      mustEnv("PLUGIN_TOKEN"),
		imageToken: os.Getenv("PLUGIN_IMAGE_TOKEN"),
		gitea:      env("PLUGIN_GITEA", "https://git.webux.dev"),
		repo:       repo,
		owner:      env("PLUGIN_OWNER", strings.SplitN(repo, "/", 2)[0]),
		domain:     env("PLUGIN_DOMAIN", "webux.app"),
		slug:       slug,
		imageTag:   env("PLUGIN_IMAGE_TAG", previewTag(slug)),
	}
}

// previewTag is preview-<pr>-<sha>: immutable per commit (so Flux redeploys on
// each PR push) and grouped by PR (so teardown can delete the whole group by
// prefix). Falls back to the branch slug when there is no PR number.
func previewTag(slug string) string {
	sha := os.Getenv("CI_COMMIT_SHA")
	group := imageGroup(slug)
	if group == "" {
		return "latest"
	}
	if sha != "" {
		return group + sha
	}
	return strings.TrimSuffix(group, "-")
}

// imageGroup is the shared prefix for all of a PR's preview images,
// e.g. "preview-42-". Teardown deletes every version with this prefix.
func imageGroup(slug string) string {
	if pr := os.Getenv("CI_COMMIT_PULL_REQUEST"); pr != "" {
		return "preview-" + pr + "-"
	}
	if slug != "" {
		return "preview-" + slug + "-"
	}
	return ""
}

func (c config) pointer() string {
	var b bytes.Buffer
	template.Must(template.New("p").Parse(pointerTmpl)).Execute(&b, map[string]string{
		"App": c.app, "Slug": c.slug, "Domain": c.domain, "ImageTag": c.imageTag,
	})
	return b.String()
}

func (c config) path() string { return fmt.Sprintf("%s-%s.yaml", c.app, c.slug) }

// url is the public preview URL, e.g. https://webuxlab-fix-mobile.webux.app.
func (c config) url() string { return fmt.Sprintf("https://%s-%s.%s", c.app, c.slug, c.domain) }

// do sends an authenticated JSON request and returns status + body.
func do(method, url, token string, body any) (int, []byte) {
	var r io.Reader
	if body != nil {
		buf, _ := json.Marshal(body)
		r = bytes.NewReader(buf)
	}
	req, err := http.NewRequest(method, url, r)
	if err != nil {
		log.Fatal(err)
	}
	req.Header.Set("Authorization", "token "+token)
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("Accept", "application/json")
	resp, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()
	out, _ := io.ReadAll(resp.Body)
	return resp.StatusCode, out
}

func (c config) contentsURL() string {
	return fmt.Sprintf("%s/api/v1/repos/%s/contents/%s", c.gitea, c.repo, c.path())
}

// fileSHA returns the current pointer file sha, or "" if absent.
func (c config) fileSHA() string {
	code, out := do(http.MethodGet, c.contentsURL(), c.token, nil)
	if code == http.StatusNotFound {
		return ""
	}
	if code != http.StatusOK {
		log.Fatalf("GET %s -> %d: %s", c.path(), code, out)
	}
	var f struct {
		SHA string `json:"sha"`
	}
	json.Unmarshal(out, &f)
	return f.SHA
}

func (c config) sync() {
	content := base64.StdEncoding.EncodeToString([]byte(c.pointer()))
	msg := fmt.Sprintf("preview: %s (PR #%s)", c.path(), os.Getenv("CI_COMMIT_PULL_REQUEST"))
	body := map[string]string{"content": content, "message": msg}
	method := http.MethodPost // create
	if sha := c.fileSHA(); sha != "" {
		method = http.MethodPut // update existing
		body["sha"] = sha
	}
	code, out := do(method, c.contentsURL(), c.token, body)
	if code != http.StatusCreated && code != http.StatusOK {
		log.Fatalf("%s %s -> %d: %s", method, c.path(), code, out)
	}
	log.Printf("synced %s (image %s)", c.path(), c.imageTag)
	log.Printf("preview: %s", c.url())
}

func (c config) teardown() {
	// 1. delete the Flux pointer -> Flux prunes the running preview.
	if sha := c.fileSHA(); sha != "" {
		msg := fmt.Sprintf("preview teardown: %s (PR #%s)", c.path(), os.Getenv("CI_COMMIT_PULL_REQUEST"))
		code, out := do(http.MethodDelete, c.contentsURL(), c.token, map[string]string{"message": msg, "sha": sha})
		if code != http.StatusOK {
			log.Fatalf("DELETE %s -> %d: %s", c.path(), code, out)
		}
		log.Printf("removed pointer %s", c.path())
	} else {
		log.Printf("pointer %s absent", c.path())
	}
	// 2. delete every preview image for this PR (needs a package-scoped token).
	c.deleteImages()
}

// deleteImages removes all container versions of the app whose tag starts with
// this PR's group prefix (preview-<pr>-*).
func (c config) deleteImages() {
	prefix := imageGroup(c.slug)
	if c.imageToken == "" || prefix == "" {
		log.Printf("no image_token/prefix; leaving images %s:%s* for GC", c.app, prefix)
		return
	}
	deleted := 0
	for page := 1; ; page++ {
		listURL := fmt.Sprintf("%s/api/v1/packages/%s?type=container&q=%s&page=%d&limit=50",
			c.gitea, c.owner, c.app, page)
		code, out := do(http.MethodGet, listURL, c.imageToken, nil)
		if code != http.StatusOK {
			log.Fatalf("list packages -> %d: %s", code, out)
		}
		var pkgs []struct{ Name, Version string }
		json.Unmarshal(out, &pkgs)
		if len(pkgs) == 0 {
			break
		}
		for _, p := range pkgs {
			if p.Name != c.app || !strings.HasPrefix(p.Version, prefix) {
				continue
			}
			delURL := fmt.Sprintf("%s/api/v1/packages/%s/container/%s/%s",
				c.gitea, c.owner, c.app, p.Version)
			dc, dout := do(http.MethodDelete, delURL, c.imageToken, nil)
			switch dc {
			case http.StatusNoContent, http.StatusOK, http.StatusAccepted, http.StatusNotFound:
				deleted++
			default:
				log.Fatalf("DELETE %s:%s -> %d: %s", c.app, p.Version, dc, dout)
			}
		}
	}
	log.Printf("removed %d image(s) matching %s:%s*", deleted, c.app, prefix)
}

func wipCommit(msg string) bool {
	return strings.HasPrefix(strings.ToLower(strings.TrimSpace(msg)), "wip")
}

func main() {
	log.SetFlags(0)
	c := load()
	switch c.action {
	case "sync":
		if msg := os.Getenv("CI_COMMIT_MESSAGE"); wipCommit(msg) {
			log.Printf("WIP commit; skipping preview sync for %s", c.path())
			return
		}
		c.sync()
	case "teardown":
		c.teardown()
	default:
		log.Fatalf("unknown action %q (want sync|teardown)", c.action)
	}
}
Dockerfile

Adapt this Dockerfile to fit your standards.

FROM golang:1.26 AS build
RUN apt-get update && apt-get install -y --no-install-recommends upx-ucl && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod ./
COPY main.go ./
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /preview-bot .
RUN upx --best --lzma /preview-bot

FROM scratch
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=build /preview-bot /preview-bot
USER 65532:65532
ENTRYPOINT ["/preview-bot"]

FluxCD Setup

This is only the configuration for the preview system.

  • Update the url to fit your configuration.
  • Create a read-only deploy key for the fleet-previews repository.
  • You then need to configure the key as shown below.

The Source

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: previews
  namespace: flux-system
spec:
  interval: 1m
  ref:
    branch: main
  url: ssh://[email protected]:22/studiowebux/fleet-previews.git
  secretRef:
    name: previews-read-key # read-only deploy key

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: previews
  namespace: flux-system
spec:
  interval: 2m
  prune: true
  sourceRef:
    kind: GitRepository
    name: previews
  path: ./

The Deploy Key Structure

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: previews-read-key
  namespace: flux-system
spec:
  encryptedData:
    identity: REDACTED
    known_hosts: REDACTED
  template:
    metadata:
      name: previews-read-key
      namespace: flux-system

Configuring your Application

/previews/preview-component

Adapt to fit your existing configuration.

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
namespace: ${PREVIEW_NAMESPACE}
commonLabels:
  env: preview
  preview: ${PREVIEW_NAME}
resources:
  - networkpolicy.yaml # Not provided, as it depends entirely on your cluster.
patches:
  - target:
      kind: HTTPRoute
    patch: |-
      - op: replace
        path: /spec/parentRefs
        value:
          - name: gateway-external
            namespace: kube-system
            sectionName: https-preview-wildcard
      - op: replace
        path: /spec/hostnames
        value:
          - ${PREVIEW_HOST}

/apps/webuxlab/overlays/preview/kustomization.yaml

Using this preview/kustomization.yaml allows spawning many PRs in parallel. You need to add this configuration for each application you want to enable the preview on.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
components:
  - ../../../../previews/preview-component
# Run the image built from THIS PR commit (Flux fills ${PREVIEW_IMAGE_TAG}).
images:
  - name: git.webux.dev/studiowebux/webuxlab
    newTag: ${PREVIEW_IMAGE_TAG}

Example (Application Side)

This blog uses that workflow to deploy and validate changes.

To skip the preview deployment, you can prefix your git message with wip or WIP.

The CI looks like:

when:
  - event: [pull_request, pull_request_closed]

steps:
  build-preview:
    image: woodpeckerci/plugin-docker-buildx:6.1.0
    privileged: true
    when:
      - event: pull_request
    settings:
      repo: git.webux.dev/studiowebux/webuxlab
      tags:
        - preview-${CI_COMMIT_PULL_REQUEST}-${CI_COMMIT_SHA}
      platforms:
        - linux/amd64
      registry: https://git.webux.dev
      username:
        from_secret: registry_username
      password:
        from_secret: registry_password
      storage_driver: vfs
      dockerfile: Dockerfile
      context: .
      no_cache: true
      build_args:
        - APP_VERSION=${CI_COMMIT_SHA}

  # Only after the image exists: write/update the Flux pointer.
  preview-sync:
    image: git.webux.dev/studiowebux/preview-bot:latest
    when:
      - event: pull_request
    settings:
      app: webuxlab
      action: sync
      token:
        from_secret: fleet_previews_token

  # On PR close: delete the pointer (Flux prunes) AND the preview-<pr> image.
  preview-teardown:
    image: git.webux.dev/studiowebux/preview-bot:latest
    when:
      - event: pull_request_closed
    settings:
      app: webuxlab
      action: teardown
      token:
        from_secret: fleet_previews_token
      image_token:
        from_secret: registry_password