I spent an afternoon locking down a Loki instance, checked the change three different ways, and all three passed. Then I tried it from the wrong side and got straight in. After all that, the filter wasn’t doing anything, it never had, and nothing in the system was going to tell me.
The change was small. Loki listens on 3100, and I only wanted it reachable from the handful of hosts that write to it or query it. systemd has directives for exactly this, so I added them to the unit, deployed, and started checking my work.
[Service]
IPAddressAllow=localhost 192.0.2.10 192.0.2.11
IPAddressDeny=any
The unit file had the directives in it. systemctl status loki said active, running, nothing in the recent log. And because I wanted to see that the directives had been read, systemctl show loki -p IPAddressAllow, which printed the whole allowlist back at me, every address I’d put in. Three for three.
Then I tried the door, from a host that had no business getting through.
$ curl -s -o /dev/null -w '%{http_code}\n' http://loki:3100/loki/api/v1/labels
200
A 200 is HTTP for yes: the request arrived and Loki answered it, from a host that was supposed to be turned away at the door. What I wanted was no answer at all. A dropped packet doesn’t get a refusal, it gets silence, and curl gives up waiting.
systemd implements those directives by attaching an eBPF program to the unit’s cgroup. This Loki lives in an unprivileged LXC container, which has no BPF delegation, so the attach never happens. Nothing in my logs surfaced it. The unit loaded, the config parsed, and the filter was decoration.
I’d like to say this was obscure. It isn’t. Here’s the systemd.resource-control man page, in the section for the exact directive I used:
Note that these settings might not be supported on some systems (for example if eBPF control group support is not enabled in the underlying kernel or container manager). These settings will have no effect in that case. If compatibility with such systems is desired it is hence recommended to not exclusively rely on them for IP security.
That’s not buried in a footnote. It sits directly under the option, it names the container manager case, and it says in plain words not to rely on this for IP security. I had that page open while I wrote the change, because nobody types IPAddressAllow= and its special address names from memory. I was reading it for syntax. The caveat was a paragraph about somebody else’s system.
systemd’s code knows about it too. There is a warning for this situation, “unit configures an IP firewall, but the local system does not support BPF/cgroup firewalling”, printed once, for the first unit that asks. And the same function drops that message from warning to debug level when the cause is a permission error and systemd is running inside a container. Fair enough: in a container it would fire constantly. It also means the one setup where the filter can’t work is the one where systemd keeps it to itself.
Three of my four checks were asking the configuration whether it agreed with itself. systemctl status says the process is up, and systemctl show reports parsed config. Neither is applied kernel state. Only the fourth check, curling from the wrong side, was asking about reality.
The first three could not have failed. There was no state of the world in which the filter was dead and systemctl status looked any different. A check that can’t fail isn’t telling me anything, and a control I’ve never watched refuse something isn’t one I can vouch for.
This isn’t new
None of this is a discovery. It shows up in the literature as fail-open behavior, as control plane versus data plane, as absence-of-evidence reasoning, and as the SRE argument for alerting on symptoms rather than on configuration. If you’ve read the Google SRE book chapter on monitoring, you’ve met the general case.
What I haven’t found is a usable list of the specific shapes this takes when you’re the one at the terminal, and what each one looks like from the outside. That’s what this is. I’ve hit all five, in stacks that have nothing to do with each other, all on my own hardware or in my own projects.
Five shapes
| Shape | What it looks like | The tell |
|---|---|---|
| Configured but not applied | The setting is present and readable. The kernel, CNI, agent, or running process never picked it up. | Nothing ever gets refused. Try it from the wrong side. |
| Scheduled but not triggered | The job is defined and its last run was green. It hasn’t run in months. | Check when it last ran, not only how it ended. |
| Ran but produced no evidence | Exit code 0, no output, no artifacts. | Ask what it produced. Assert a count. |
| Measured with the wrong population | The number is plausible. The denominator contains things you didn’t mean. | Enumerate the population by hand, once. |
| Enforced, but the wrong thing | The control works exactly as built, against the wrong target or in the wrong mode. | Test the allowed path too, not only the denied one. |
The Loki filter is the first row. The rest follow, and Loki comes back at the end.
Ran but produced no evidence
I had a visual regression gate on a game project. It ran the real build headless, took captures, and compared them against a baseline. The invocation used --quit-after 600.
Godot documents that flag as main loop iterations. Not seconds. Six hundred frames is about ten seconds, and the comparison needs about two minutes. So every run got killed a tenth of the way in, printed no result, and exited 0. The gate had been green for months, including across a release that changed how every character in the game rendered.
The tell was in the CI UI the whole time: 42 seconds, for a job that should take two minutes. I’d been reading that number as good news.
Scheduled but not triggered
Same repo. A workflow dropped staging from its push.branches list, and the commit message explained why that was fine: there was an always-open staging to main pull request, so every staging commit would still get tested through the pull_request trigger.
That pull request was a release cut. It merged six minutes after the commit that removed the trigger.
For the next four months the suite ran on staging only on days a staging to main PR happened to be open, so basically at release time. Nothing reported it. A trigger that matches nothing looks exactly like a trigger with nothing to match. I only found it because I went looking for a result that turned out never to have existed.
The commit message was confident, and wrong inside six minutes. The rule I took from it: don’t delegate a trigger to the continued existence of a PR, a branch, or anything else that somebody else’s workflow is allowed to close.
Removed, and still running
I removed Performance Co-Pilot from my fleet. apt purge pcp pcp-conf succeeded everywhere it ran, dpkg -l 'pcp*' came back empty on every host, and I wrote it up as done.
A port scan the next day, for something unrelated, found PCP still at its post on twenty-four hosts. Some of those daemons had been there since May.
The obvious reading is that apt failed. It didn’t. A process outliving the unlink of its own binary is ordinary Unix behavior, and apt purge doesn’t promise to stop running daemons. PCP runs four of them. Two, pmcd and pmlogger, happened to stop on their own. The other two, pmproxy and pmie, kept running with their ports bound. Half the ports went away, which is what made my spot check look like progress.
The bug was that I accepted package state as evidence of process state. The property I wanted was “these ports aren’t listening any more.” The property I measured was “the package is gone.” Only one of those was the point. This is the first row run backwards: gone from the configuration, still there in the running system. And this goes well past apt: disabling a systemd unit doesn’t stop the running instance, removing a firewall rule doesn’t kill established connections, revoking a token doesn’t invalidate the sessions already issued.
Who’s in the denominator
Telemetry for that same game reported installs, retention, and crash rates. Google Play runs a pre-launch report on every upload: a farm of virtual devices boots your actual build and pokes at it. Those devices send real telemetry, with a fresh install id every run.
They were 63 of the 95 Android installs. The top device model in my own census was a OnePlus 8 Pro rendering through Google SwiftShader, which is a software rasterizer, which means it was a phone nobody owns.
Every per-install rate I’d been reading was computed over a population that was two-thirds robots. And because the robots arrive in a burst right after each release upload, they produce exactly the pattern my crash-spike alert was built to fire on.
Nothing about the report looked broken. It counted everything that arrived, and everything that arrived had the right paperwork.
A detection rule evaluating a population that no longer contains the thing you care about is this failure in different clothes. A wrong denominator still produces a plausible number, and plausible numbers don’t trip anyone’s suspicion. A number that’s obviously garbage gets investigated. A number that’s quietly garbage gets put in a slide.
I’d already written this down. A strategy doc from four months earlier says, correctly, that most of the installs for one release were pre-launch robots. The fact existed in prose. Prose doesn’t filter anything. The eventual fix was one condition: a real player’s phone has a GPU, so anything reporting a software rasterizer isn’t a player.
Enforcing, and still wrong
Back to Loki.
I replaced the systemd filter with nftables inside the container, curled from the wrong side again, and this time got refused. The control seemed to work. I’d got the result I wanted. It was still aimed at the wrong target.
I’d built the allowlist by reading every service config for things that talk to Loki. That inventory missed a script on my workstations that queries Loki by IP, which no service config mentions. The filter did exactly what I’d told it to and turned away one of my own.
Every test I ran that day was from the wrong side, because that was the test I’d just learned to run. It passed. The test that would have caught this one was the opposite: check that your own side still gets in. A control can be perfectly enforcing and still be pointed at the wrong set, and “it blocks” is only half of “it works.”
Where this bites hardest
None of what follows happened to me. These are well-known public failure modes, listed to show the same shapes where the stakes are higher than a homelab.
- A Kubernetes
NetworkPolicythat is present and valid while the CNI doesn’t implement policy at all, or whose pod selector matches zero pods. Configured but not applied. - An admission webhook with
failurePolicy: Ignoreand an expired serving certificate. Every request sails through and the cluster reports compliant. Configured but not applied. - A firewall rule installed in
iptables-legacyagainst annftdataplane, or in the host namespace when the workload lives in its own. Configured but not applied. - Monitor mode that never graduated: a WAF rule in count rather than block, a DMARC record left at
p=none, a CSP shipped report-only, SELinux or AppArmor left permissive. Everyone gets logged, nobody gets stopped, and the report pipeline nobody reads is technically working. Enforced, but the wrong thing. - A SIEM detection rule that’s green and has matched nothing since its log source stopped ingesting. Scheduled but not triggered.
- Secret scanning or SAST running clean against zero files, because a path or branch filter excludes where the code actually lives. Ran but produced no evidence.
Every one of these renders correctly in a dashboard.
Compliance evidence is usually collected from the configuration surface: a screenshot of the policy, an export of the rule, a report that the agent is installed. That is the surface that’s wrong in every example above, so the artifact you hand an auditor and the artifact you’d need to catch the failure are the same document, and it says yes either way. Auditors separate design effectiveness from operating effectiveness and test them differently, one by reading the configuration and the other by watching the control act on something. Most of what gets collected is the first kind, filed as the second.
What I ask now
Have I ever seen it fail? Before trusting a check, hand it something broken and watch it go red. I once evaluated a config validator that exited 0 having written no output at all, because it couldn’t create its own lockfile. I also shipped a test whose fake server answered “success” to any request, including one the real server rejects. Both were green. Neither had ever been red.
Did it enforce? Try it from the wrong side. Where you have a choice, pick mechanisms whose applied state you can inspect. That’s why I went to nftables: nft list table shows what the kernel is actually holding, and the rule has a counter on it.
# nft list table inet service_acl
tcp dport 3100 ip saddr { 192.0.2.10, 192.0.2.11 } accept
tcp dport 3100 counter packets 14 bytes 840 drop
Fourteen packets turned away is a thing systemctl status could never have told me. Then I added a post-deploy assertion that the drop rule is in the live table, because the next silent no-op is going to look exactly like success too.
Did it run, and when? “The last check was green” and “the last check was green, four months ago” are the same badge. Look at the timestamp. Look at the duration too. A job that passed in 42 seconds when the work takes two minutes didn’t do the work.
What did it produce? If the answer is “no errors,” it isn’t a check yet. Make it emit a count and assert on the count: records written, bytes moved, captures taken, ports bound. A check whose success condition is the absence of a complaint can’t tell “ran and found nothing” from “never ran.” Don’t take the exit code’s word for it either. Proxmox’s qm guest exec exits 0 whether the command in the guest succeeded, failed, or was still running when the timeout hit. The real answer is in the JSON it prints.
One caveat. A count is not a set. Two DNS servers of mine each reported eight names published, moving in lockstep, while one of them was missing a record the other had. Eight and eight, and not the same eight. When the property you care about is agreement, compare the contents, or a hash of them, not the lengths.
Who’s in the denominator? Print the population once and read the top few rows. My robots were obvious the first time anyone looked at the device list.
Is it aimed at the right set? Test the right side too, not only the wrong one. Build the list of who gets in from who actually shows up, not from what the configs say should.
What was missing
Every failure here started as a decision that was right when it was made and quietly stopped being true. The reasoning always existed somewhere: a commit message, a strategy doc, a man page I had open in another tab, my own head. What didn’t exist was anything that would notice when the reasoning expired.
I’m not going to tell you automation fixes this. Half of this essay is automation failing quietly. The narrower claim is the one I actually trust:
A comment explaining why a check is safe to remove is a claim. A test that fails when the claim stops holding is a record. A decision belongs in automation only when the automation asserts evidence that the decision is still true.
I still run systemctl status first, out of habit. Then I go around to the wrong side.