There is a particular kind of engineering project that starts with a sentence like:

This shouldn't be too complicated.

We had a website. We had Git. We had a deployment process that worked.

A webhook reached the web server, the repository was updated, Composer did its thing, Yarn built the frontend assets, a few custom deployment commands ran, and a short while later the new version was online.

Perfectly reasonable.

Except that this deployment model had slowly become one of the older parts of a stack that had changed substantially around it.

We were working toward version 4 of Charm, our own PHP framework. We had just built an entirely new Neoground website on top of it. The website itself was a clean break with the old one: a new repository, a new company identity, a much more deliberate frontend architecture, custom UI rather than another Bootstrap-shaped foundation, and a stronger engineering structure underneath the visual polish.

The new company line is:

Strategic technology, carried through to execution.

At some point it became difficult to ignore the irony that the website communicating that message was still assembled on the production server after a Git webhook.

So we decided to modernize the deployment.

We just wanted to deploy a PHP website.

Naturally, two days later we had built a small software supply chain.

The old system was not actually bad

This is worth stating first.

There was no dramatic production incident forcing a rewrite. No heroic migration away from an unmaintainable shell script held together with duct tape.

The previous process was approximately:

push
  ↓
Forgejo webhook
  ↓
production server
  ↓
update Git working tree
  ↓
composer install
  ↓
yarn install
  ↓
build CSS and JavaScript
  ↓
custom deployment hooks
  ↓
done

For a small team and relatively straightforward applications, this can work remarkably well.

Charm applications also have their own deployment semantics. There may be database migrations, generated sitemaps, caches, application state and other tasks which have to happen after the code changes. We already had commands for that.

The problem was less what the deployment did and more where responsibilities lived.

Production was simultaneously:

• serving the application;
• holding the source checkout;
• resolving PHP dependencies;
• resolving frontend dependencies;
• compiling assets;
• executing deployment logic.

That means your production machine is not merely running software. It is also one of the places where the software is manufactured.

And once you notice that distinction, it becomes difficult to unsee.

Build somewhere else, deploy the result

The conceptual change was simple:

OLD

source code
    ↓
production server
    ↓
build + run


NEW

source code
    ↓
CI
    ↓
build artifact
    ↓
production
    ↓
run

The production server should receive something that is already complete.

No yarn install.

No Sass compiler.

No esbuild.

No Composer dependency resolution.

No dependency ecosystem touching production simply because I changed a heading somewhere.

The obvious answer was CI.

We use Forgejo, which already provides Actions and runners, so there was very little reason to introduce another CI platform.

Easy.

Except the first question was immediately more interesting:

Where should the runner actually execute arbitrary build code?

CI is controlled arbitrary code execution

It is tempting to put a runner onto an existing server.

There is plenty of unused CPU. Plenty of RAM. Docker already exists there.

Why waste another machine?

The problem is that a CI runner is a rather unusual service. Its entire purpose is to accept instructions from repositories and execute them.

Once Docker enters the picture, this becomes even more important.

Giving a workflow access to the Docker daemon can effectively give it enormous control over the Docker host. That may be entirely acceptable on a dedicated CI machine. It is much less attractive on a machine carrying unrelated services.

I didn't want a compromised dependency, erroneous workflow or future configuration mistake to transform:

build this website

into:

welcome to the infrastructure

So CI needed its own trust boundary.

That requirement stayed constant even while the implementation changed several times.

The technically elegant solution became the wrong solution

One obvious option was a VM on one of our Linux hosts.

KVM. Libvirt. Debian cloud image. Automated provisioning. Disposable CI machine.

Technically, I liked it.

Operationally, less so.

The host in question already has a fairly substantial networking configuration. Docker alone has given it enough bridges, virtual interfaces and veth devices that I have encountered software choosing surprising interfaces or becoming confused by what 0.0.0.0 means in an increasingly creative topology.

Adding another virtualization network, bridge, routing layer and firewall policy was completely possible.

It was also exactly the kind of complexity I did not want to add to an otherwise stable host just to compile CSS occasionally.

A separate cheap VPS would solve that.

So would a dedicated mini PC.

I even have an old Intel NUC which would make a charming little low-power CI appliance. Wake it on demand, let the runner execute jobs, shut it down again.

We went quite far down that thought.

And then the embarrassingly obvious solution appeared.

The CI server was sitting under my desk already

I am currently the person doing the development.

When I am deploying something, my workstation is therefore rather likely to be switched on.

VMware Workstation was already there.

So I created a Debian VM.

That was it.

No new host networking on production.

No VPS bill.

No resurrected NUC.

No Wake-on-LAN automation.

No reason for the CI system to be available at three in the morning when nobody is developing anything.

The VM gets a substantial chunk of workstation CPU and memory while it is running, which also means it is considerably faster than the little NUC would have been.

It has a small XFCE desktop because sometimes having an actual graphical console nearby is convenient. The Forgejo runner itself runs as a normal systemd service.

When I want to deploy:

start VM
  ↓
runner comes online
  ↓
queued job executes

When I'm finished, I turn it off.

If the environment becomes messy, VMware snapshots exist.

Sometimes the sophisticated architectural choice is simply identifying which complexity you do not currently need.

If the workload grows, the same runner role can move to a small VPS or dedicated appliance later.

The workflows don't particularly care.

Building our own build environment

The VM still isn't the actual application build environment.

Docker is.

That gives every CI job a fresh container and lets us define the exact environment a Charm application expects.

So we created our own CI image.

Its current foundation looks roughly like this:

Debian 13
PHP 8.5
Composer 2
Node.js 24 LTS
Yarn 4
Corepack
Valkey-compatible PHP tooling
Git
SSH
rsync
tar
zstd
jq
required PHP extensions

There are a couple of nice details here.

Node comes from the official Node image.

Composer comes from the official Composer image.

PHP comes from the official PHP 8.5 Trixie image.

Docker's multi-stage builds let us use these other images as sources while constructing the final environment.

The result is effectively a tiny standardized Neoground build workstation:

Forgejo job
    ↓
Docker
    ↓
our PHP build image
    ↓
project source

Project-specific commands stay inside the project workflow.

Stable tooling stays inside the image.

That distinction turned out to be important.

CI immediately found something I had forgotten

Then the first real application build failed.

Good.

That is one of the jobs of CI.

The offending command was:

yarn install --immutable

Yarn responded that the lockfile would have to change.

Which was forbidden.

After a little investigation, the reason turned out to be wonderfully mundane:

the project was still on Yarn 1.

Somehow that ancient local assumption had quietly survived while Node, PHP, the application and almost everything around it had moved on.

My development workstation knew how to make it work because it had accumulated the right historical state.

A clean CI environment did not.

That is precisely the kind of thing reproducible builds are supposed to expose.

We migrated properly to Yarn 4, added the project configuration, updated the lockfile and pinned the package-manager version.

Then:

yarn install --immutable

worked.

No implicit migration.

No silently modified lockfile.

No dependency state invented while building production.

The repository now describes what it actually needs.

One small failure had already justified the exercise.

Charm enters the pipeline

The project we used for the first real deployment is the new Neoground website.

It is also an increasingly good test of where Charm v4 is heading.

Charm is our PHP framework and application foundation. It has grown around a fairly simple philosophy: provide the infrastructure real applications need while keeping the programming model compact and understandable.

It is modular, uses modern PHP, integrates things such as Twig, database access, queues, events, caching and configuration, and powers our own applications rather than existing primarily as a framework experiment.

The deployment pipeline was therefore another useful test:

Can a Charm application move cleanly through a modern build and release process without bending the application around the CI system?

Turns out: yes.

The production build now essentially does this:

checkout repository
    ↓
set production environment
    ↓
composer install --no-dev
    ↓
yarn install --immutable
    ↓
build Sass + JavaScript
    ↓
remove node_modules
    ↓
verify expected output
    ↓
package release

The frontend build uses our existing Sass and esbuild pipeline.

Composer installs the exact production dependency set.

The resulting directory is no longer source waiting to become an application.

It is the application.

A website becomes an artifact

This was the point where things became oddly satisfying.

Instead of shipping a Git repository around, CI turns the complete production tree into a Zstandard-compressed tar archive.

A typical build currently ends with something around:

source repository: ~20 MB
production artifact: ~13 MB

Alongside the archive we generate a SHA-256 checksum and structured JSON metadata.

Conceptually:

website-<build>.tar.zst
website-<build>.tar.zst.sha256
website-<build>.json

The metadata records things such as:

{
  "project": "...",
  "source": {
    "commit": "...",
    "ref": "..."
  },
  "artifact": {
    "filename": "...",
    "size_bytes": 0,
    "sha256": "..."
  },
  "build": {
    "environment": "Prod",
    "php": "8.5.x",
    "node": "v24.x",
    "yarn": "4.x"
  }
}

The precise format will continue to evolve, but the principle is useful.

A deployment artifact should be able to answer:

• What project am I?
• Which source commit created me?
• Which build produced me?
• What environment was used?
• How large am I?
• What should my checksum be?

Git remains the source of truth for the code.

The artifact becomes the source of truth for what we actually deploy.

Then “upload the file” became a security problem

At this point we had a beautiful little archive sitting inside CI.

Now it needed to leave CI.

The easiest implementation would have been to give the CI machine normal SSH access somewhere.

Absolutely not.

The entire point of separating CI was to narrow its authority.

So artifact publication itself became capability-like.

The CI environment has a dedicated SSH credential which cannot become a normal shell session. The server restricts it to a narrowly defined rsync operation and an artifact inbox.

The credential cannot simply roam around the server.

It cannot turn into an administrator.

It cannot replace existing canonical releases.

Its purpose is essentially:

You may place new candidate artifacts here.

That is enough.

CI transfers:

archive
checksum
metadata

with the metadata arriving last.

That last detail gives us a wonderfully simple completion protocol.

If an upload dies halfway through transferring the archive, no complete metadata object exists yet.

The ingestion side therefore doesn't have to guess whether a file has finished uploading.

No metadata means:

not ready.

Boring primitives can be remarkably effective.

CI deliberately stops at the inbox

This is another boundary I wanted to keep.

CI does not deploy the website.

CI creates and publishes an artifact.

Deployment belongs to infrastructure.

That is where another piece of our stack enters the story: Compass.

I recently wrote about Compass on the Neoground blog:

Meet Compass: Our Infrastructure Management Layer

Compass sits above our provisioning and infrastructure tooling and gives us a structured way to manage the systems we operate.

Artifact deployment now fits naturally into that model.

The public architectural version looks roughly like this:

Git / Forgejo
    ↓
CI runner
    ↓
custom build container
    ↓
verified artifact
    ↓
restricted inbox
    ↓
Compass + provisioning layer
    ↓
artifact catalog
    ↓
controlled deployment

Once an artifact arrives, the infrastructure side can validate it, ingest its metadata, register it and make it available as a deployable release.

A specific version can later be selected through our tooling.

CI doesn't need authority to perform that operation.

That separation is rather important.

BUILD SYSTEM
produces releases

DEPLOYMENT SYSTEM
decides what goes live

Those are related responsibilities.

They are not the same responsibility.

Real deployments contain state

There was one more reason why simply extracting the archive over the website would have been wrong.

Real applications are not always perfectly immutable directory trees.

Our Charm applications deliberately have some persistent state within their application structure.

For example:

var/logs
var/cache
data

Logs obviously survive deployments.

Site-specific data survives deployments.

The cache directory survives structurally, although its contents may be deliberately cleared.

So our deployment handler works from the extracted artifact as a clean master state, but reconciles it against the existing application rather than blindly replacing everything.

Conceptually:

artifact
    ↓
verify checksum
    ↓
extract into staging
    ↓
validate release
    ↓
reconcile application tree
    │
    ├── preserve persistent data
    ├── preserve logs
    └── clear disposable cache
    ↓
Charm post-deploy command

Charm then gets the final word through an application-level deployment hook.

A project may need to:

• run migrations;
• regenerate a sitemap;
• clear additional caches;
• update generated structures;
• execute other application-specific maintenance.

The infrastructure layer should not need intimate knowledge of those semantics.

It gets the new application state into place.

The application knows how to finish becoming itself.

No /public directory? Still fine.

Charm applications also don't follow one increasingly common PHP convention: putting the entire externally accessible surface beneath a dedicated /public directory.

This is deliberate.

Our application structure predates that convention becoming nearly universal, and keeping static assets directly available has some practical advantages for how Charm projects are organized.

Instead, the web-server configuration defines the security boundary carefully and explicitly controls what can be served.

That architecture works very well for us.

And this entire exercise reinforced something I care about in infrastructure design:

the deployment system should understand the application architecture rather than forcing every application into the deployment system's favorite directory layout.

Generic recipes are useful starting points.

They aren't laws of physics.

Then it worked

Eventually there was nothing left to theorize about.

Push the project.

Forgejo schedules the workflow.

The runner picks it up.

Docker creates the build environment.

PHP 8.5 starts.

Composer installs production dependencies.

Yarn performs an immutable dependency install.

Sass and esbuild build the frontend.

The build tree is cleaned.

The artifact is compressed.

SHA-256 is calculated.

Metadata is generated.

The three release objects are transferred into the restricted inbox.

The infrastructure layer ingests the artifact.

The website is deployed.

And the whole build and publication process takes around 30–40 seconds.

Less than a minute after starting with repository source, we have a complete, verified, deployable website artifact.

The result is almost disappointingly uneventful.

Which is exactly what I want from deployment infrastructure.

So what did we actually improve?

It would be easy to summarize this as:

We replaced git pull with CI/CD.

That misses most of the interesting part.

What really changed was where responsibilities live.

Production stopped being a build machine

It receives prepared software rather than assembling software.

Build state became explicit

PHP, Composer, Node and Yarn versions are controlled by the build environment and project configuration rather than whatever happens to exist on a server.

CI got its own trust boundary

Repository-controlled code doesn't execute alongside unrelated infrastructure.

Releases became identifiable objects

An artifact is attached to a source commit, checksum, metadata and build environment.

CI lost deployment authority

It may publish a candidate release.

Infrastructure decides what gets deployed.

Application semantics stayed with the application

Charm's post-deployment lifecycle handles the work only the application itself understands.

None of this required Kubernetes.

There is no giant orchestration platform hiding somewhere.

The important pieces are still remarkably ordinary:

Git
Forgejo
Linux
Docker
PHP
Node
Yarn
tar
Zstandard
SHA-256
SSH
rsync

The sophistication is mostly in deciding where each one belongs and what it should be allowed to do.

It was still just a website

There is something wonderfully typical about software engineering here.

The visible task was:

Put the newest version of a PHP website onto a server.

Underneath that sentence ended up being source control, CI orchestration, virtualization, containers, package management, PHP runtime design, frontend compilation, artifact engineering, SSH security, filesystem semantics and application lifecycle management.

Five layers become eight rather quickly if you stare at the problem for long enough.

And yet the end result is actually simpler to operate.

That is the part that matters.

The first real application to complete the entire journey was our completely rebuilt Neoground website.

A new company identity.

A new repository.

A new frontend.

Charm moving toward version 4 underneath it.

A new release pipeline around it.

And on the website itself:

Strategic technology, carried through to execution.

Apparently we took that rather literally.