Harden your servers. Own your trust.
Harden your servers. Own your trust.
How to do security right in the time of AI Agents
It's very difficult task to be secure when everybody is acting like "Tony Stark" with his Iron Man suit and JARVIS-like AI Agent. CVE feeds look like weather reports: remote code execution, privilege escalation, sandbox escapes. And this month a public Certificate Authority failed so badly that one private key was the digit 4.
That mess favors attackers. Stacked controls still hold if you treat every layer as disposable and pin the roots yourself. This post is the map. Seal the golden OS (part 2) covers the host; Own the pipeline (part 3) covers supply, CI, and the herd.
You own the trust chain. If a third party is hacked, you should not be owned.
The eMudhra lesson
In October 2026, reporting on the Indian CA eMudhra described private keys for several intermediates that were trivial to guess. One was literally the number 4; others were below 256. Anyone who recovered them could forge TLS for any site those intermediates covered (pretend to be google and send android malicious update, pretend to be microsoft and send malicious update, ..etc).
Bad actors were able to update servers, login pages, "legitimate" software downloads. Mozilla blocked the intermediates within hours, and eMudhra later announced it was leaving the TLS business. Everyone who had pinned production trust to the public bundle still had a cleanup to do.
Sources:
- The Register - eMudhra private keys compromised; one was the number '4' (2026-10-20)
- International Cyber Digest - An Indian CA's Private Key Was Just the Number 4 (2026-10-24)
Certificate pinning, in this series, means pinned to your CA (FreeIPA/Dogtag or equivalent), not to the OS/browser public trust store. Android reverse-engineering talk often means "pinned to OS certs." That is not enough for production trust you care about.
Own the inventory
Everything production trusts is ours, content-addressed and auditable where it matters:
- Our CA - Pin TLS and identity. The public bundle is not the production root.
- Our git on our server - Desired state and code. Commits and tags are signed, and history is content-addressed, which makes git a blockchain of signed commits. A private repo on a public git host does not count as ours.
- Our registry - The only place cattle pull OS and app images from. Inside the LAN.
- Our frozen supply + CI - Audited inputs. Build with network off, or LAN-only. No `curl
- Our Ignition + Ansible - First boot and idempotent orchestration. Ansible is not the OS source of truth.
- Our golden image - Signed bootc / Fedora CoreOS artifact. Canary, then promote. Replace the node; do not repair it.
One rule: our CA, our registry, our signed git, our frozen supply, our CI, our Ansible, our image, or it does not run.
"Private" on someone else's server is still theirs. If that host is breached, the attacker reads and can rewrite your desired state. That is the eMudhra failure again, moved from TLS to source control.
Defend, contain, recover
Assume breach and keep control. Every layer can fail, so design each one so that a single failure cannot become total control.
- Defend. Pin roots and freeze the supply before humans or agents touch the cattle. Nothing runs unless it came from our registry, built by our CI, from our git.
- Contain. A compromised public web box must sit in a vacuum even if it shares a LAN with other hosts. Allow only the east-west paths the app needs and deny unexpected egress. Host-based trust ("on the VPN", "trusted IP range") is never enough: malware on a developer laptop that holds a VPN config is malware with a free pass. Least privilege and tenancy boundaries mean a compromised cow cannot promote the herd.
- Recover. When something breaks, replace the host from a known digest and roll the pipeline back. Do not SSH in to heal a snowflake.
Cattle, not pets
The OS is an immutable(sealed) golden image: reproducible, tested, canaried, then rolled out. A bad node gets replaced. You investigate the image and the pipeline, not the individual VM. Local node state is either disposable or stored elsewhere, and that anti-persistence is a feature: an implant that survives only until the next replacement has a short life.
Day-to-day change does not travel over standing SSH. People push to git, someone reviews, and CI runs Ansible to deploy and promote. Interactive SSH stays rare, is still allowed for emergencies, and still follows the same trust rules.
Agents raise the stakes, but not where most people look. One broken systemd unit on one VM is noise. The blast radius that matters is a poisoned golden image, or a credential that can push to the registry, merge to the promote branch, mint CA material, or rewrite IAM. Lock those paths first.
Six layers, three groups
The map has six layers. They fall into three groups: roots you pin, runtime you confine, and releases you can stand behind.
Roots you pin
- CA. TLS and identity chain to your CA.
- Git and registry. Signed, content-addressed history on your server; images served only from your LAN registry.
- Frozen supply. The wheels, base RPMs, and git SHAs CI may fetch are audited and pinned. Builds run with the network off(eg. with bubble wraps:
bwrap --unshare-netor equivalent), or LAN-only when that is impossible.
Runtime you confine
- Sealed OS. Boot a signed Fedora CoreOS / bootc image. Ignition runs on first boot only: identity, SSH hardening, and systemd units that pull workloads from your registry. An optional local Ansible oneshot handles specialist host setup Ignition cannot express cleanly, always pinned to a
config_ref. - Confined workloads.
- NoNewPrivileges=yes, private tmp, tight capabilities
- one Unix user per app
- the unix user must be non-remote, non-sudoer/wheel, shell=nologin
- SELinux enforcing on the golden image.
Releases you can stand behind
- Releases. Every release must be:
- Integrated: interoperable parts, not a pile of snowflakes.
- Auditable: who built what, when, where, and how.
- Reproducible: the same frozen supply gives the same digest. If two CI runs disagree, stop.
- Definable: desired state declared in a Dockerfile, Ignition, and Ansible, not in tribal knowledge.
- Deliverable: canary a digest from your registry, then promote. Cattle get a digest, never "latest."
Concrete controls that still earn their keep
Passwords are not the root of any critical path. Push code and infrastructure-as-code with long SSH keys (4096-bit or stronger), passphrase-protected on the client. SSH never sends the private key to the server.
Immutable hosts arrive as signed images (Fedora CoreOS, CentOS Stream CoreOS, bootc). Workloads run sandboxed in containers. In compose or Kubernetes, prefer a read-only root filesystem with tmpfs for scratch space, and allow writes only to object storage or a known volume path. Logs go to stderr and on to a central sink.
Static artifacts can live in a private S3 bucket or an on-prem S3-compatible store (eg. RustFS) with rsync replication. Containers are built from your code into your registry. They are not pulled from strangers on the public internet, and not from a "private" registry hosted by a third party.
Database and app roles follow the same least-privilege habit. A report job gets SELECT, not DDL. Prefer insert-only, log-like tables where the data allows. Revoke and rotate when a secret or a host goes bad.
In systemd units, set at least:
[Service]
User=app-reports
NoNewPrivileges=yes
PrivateTmp=yesThen tighten CapabilityBoundingSet, and set PrivateNetwork=yes when the unit needs no network at all. Apply the same idea in compose and Kubernetes: read-only filesystem, immutable image, explicit mounts.
Secrets belong in a secrets engine that requires unseal or authentication to start. In this series that engine is OpenBao. An engine that starts without unseal is not encrypted in any sense that matters. Keep secrets off clients and never ship them to clients. A user-specific secret leaves the server only in exchange for that user's own secret. Prefer short-lived access tokens over long-lived replayable ones.
On public endpoints, proof-of-work challenges (Anubis-style) remain a cheap filter when bots are the weather.
What this series will not pretend
A VPN or private network controls exposure. It does not authenticate anyone, and it does not replace phishing-resistant login. For the forge web UI, the practical default is WebAuthn through Keycloak with users from FreeIPA. Client-certificate mTLS is a sharper channel gate if you want one; you do not need it to claim you own the chain.
An SSH CA with short-lived user certificates is an optional alternative some organizations use. The worked example in this series keeps Ignition-planted authorized_keys on a hardened remote user with AllowGroups remote, because you can inspect the machine and see who can log in. Own the pipeline covers that tradeoff honestly. It is not a mandatory checkbox.
Where to go next
- Seal the golden OS (
part 2) is the host deep dive: Dockerfile through bootc into your LAN registry, minimal Fedora CoreOS, Ignition, SSH and app-user hardening, composefs and signed UKIs, and Quadlets that pull only your digests. Start there if Zincati or bootc still tracks an upstream stream you never staged, or if hosts still log in as the distro default. - Own the pipeline (
part 3) is the supply and herd deep dive: golden supply versus golden image, net-off CI, author then reviewer then CI promote, Ansible as herd orchestration, WebAuthn on the forge, OpenBao, and canary rollout against a digest you already signed. Start there if merge-to-promote, registry push, and secrets unseal still sit on one shared role.

Comments
Post a Comment