Shieldlist — product specification

Cette page n'existe qu'en anglais pour le moment.

Self-hosted, multi-server intrusion detection and remediation platform for hosting infrastructures. Written in Go.

Open decisions are collected in §11; everything else in this document is settled.

1. Purpose#

Shieldlist watches the logs of a fleet of servers, detects attacks, applies sanctions, enforces them on every machine, reports offenders, and keeps the whole history queryable, and every action available on it after the fact.

It is also, deliberately, the fleet's log platform: centralising the logs, making them searchable, and holding them for the durations the law requires are product goals in their own right, not by-products of detection. The history must make hindsight easy: what attacked this machine last month, and what did we let through.

The name is the design: the durable output of detection is accurate lists of attackers. Bans stay as close as possible to proven attacks, the widest sanctions are earned collectively rather than guessed (§5.8), and what the lists accumulate — who acted together, from where — is the intelligence the platform exists to build.

Everything the platform does must be drivable three ways, at parity: web interface, CLI, and declarative config files. No feature may be UI-only; no feature may be config-only. Interface and CLI are both clients of one API (§5.9), so whatever a person can do, a script can.

2. Naming#

The product is Shieldlist. The CLI verb is shield, and the shield binary ships with every install, controller and agent machines alike (§6).

3. Vocabulary#

Two rules govern every name in this product:

1. No metaphor that needs explaining. A word that requires a paragraph of documentation before it means anything is the wrong word.
2. Internal names may be technical; visible names must be plain. If a word appears in the UI, the CLI or a config file, it must be understandable without documentation.

A rule is in one of three states: on, test, off. In test the rule triggers and records everything, but applies no sanction — it decides exactly as in prod, and nothing is banned; the console shows the two apart (prod / test). Files written before 2026-08-23 say detect-only for the same state; it is still read, never written back.

The chain, end to end:

log line → parser → hit → threshold → trigger → sanction (ban) → enforced by agents

4. Principles#

1. History is the product. Every event, hit, sanction and report is retained for a configurable duration (default 1 year, must hold 3) and stays queryable. Expiry is a state, not a deletion: offender history and escalation counters survive within the retention window, so repeat offending stays visible.
2. Everything can be replayed. Any action available in real time must apply to the past: ban, unban, report, exempt, add to a list — individually or in bulk, from a saved query. Parsing included: fixing a parser must allow re-deriving fields from stored raw lines.
3. Detection assists the operator, it does not surprise them. A rule is armed by a human, after seeing what it would have done against real stored history.
4. No attacker-controlled input in a synchronous path. No reverse DNS, no remote fetch, no callback in the sanction path.
5. Everything runs locally. No third-party service is needed to detect, decide or enforce. The only outbound traffic the platform originates is abuse reporting, list federation, update checks and fetching parser or rule sets from a catalogue — all opt-in.
6. Fail open on detection, fail closed on enforcement. A detection outage must never block traffic; a transport outage must never silently unban.
7. Configuration is data, never code. Nothing the controller distributes can execute on an agent. See §5.2.
8. Two concepts, not four. Parser and rule; everything else parameterises them. Configuration is local files the operator versions. Parser and rule sets may come from anywhere — shipped with the product, written in place, or fetched from a catalogue — but they always arrive as ordinary files, copied into the configuration and armed like anything else: what the platform evaluates is what is in the files, never a remote subscription evaluated behind the operator's back.

5. Functional requirements#

5.1 Collection#

• The agent tails its declared log_inputs: HTTP access and error logs, mail, ssh, journald units, arbitrary files.
• Raw lines are shipped and stored. The controller parses; the agent parses only what its local rules need (§5.3). The raw line remains the truth, which is what makes re-parsing history possible and a replay honest.
• Rotation- and truncation-proof. File identity is (device, inode), not path. Rotation yields a new identity and closes the previous one; a size below the current offset means truncation and resets it.
• At-least-once delivery, deduplicated at ingest. A batch may be persisted without its acknowledgement reaching the agent, so the agent re-sends and the controller deduplicates on the batch's position in its source: (agent, log_input, position), where position is (inode, offset) for a file, the cursor for a journald unit, and an agent-assigned sequence number for a socket. Exactly-once is a property of the store, not of the transport.
• Local queue. When the controller is unreachable the agent buffers — in memory by default, on disk if configured (§6) — with a size cap and an explicit oldest-first drop policy that is logged and exposed as a metric. Never block the protected service. A memory queue lost to a restart costs central history, never enforcement or detection: a live tail joins the present (§6) rather than trusting a saved position, so the unshipped lines stay in the source logs but are not re-read. What the agent was counting is remembered, not re-read: the rules' windows — the hits below a threshold — go to disk at shutdown and every few minutes, come back at start with what the windows still hold, and cross an in-place reload; an attacker seen an hour ago does not get a clean slate because the agent restarted. Choosing the memory queue is choosing that trade; completeness of the central history is the disk queue's job.
• Enrichment at ingest, from local databases only: geolocation, ASN, network, known-actor classification. Plain files on the controller, updated out of band — or fetched by the controller itself when the operator names a source (the free DB-IP databases, monthly; nothing is fetched unless named, and the attribution the data's licence asks for is shown where the data is).
• Ownership attribution at ingest (§5.6), mandatory.

Discovery. Configuring log inputs by hand on every machine is the tedium that stops this kind of tool from being deployed, so the agent discovers them.

• Panel detection. The agent recognises the hosting control panel installed on the machine and derives the log inputs from it. A shipped catalogue covers the main panels and the services they front — the cPanel, Plesk, DirectAdmin, ISPConfig, HestiaCP, CyberPanel, CloudPanel, Froxlor and Virtualmin families, each with its own web, mail and FTP stack — plus plain installations with no panel at all.
• Format detection. For a log with no known origin, the agent samples lines and scores them against the shipped parsers, then proposes the match. A proposal is never armed silently (principle 3); the operator confirms, or a policy accepts automatically for formats above a confidence threshold.
• Discovery feeds ownership. Panels already encode which domain belongs to which account in their log layout — per user, per domain, or both. That layout is what makes ownership attribution (§5.6) work without anyone typing a mapping. Where the panel's own configuration must be read to resolve it, it is read locally and read-only, never written to and never depended on synchronously.
• Detection reads local files only. It never fetches anything remote, and it never runs a command the panel supplies.
• Volume target: a single controller must sustain millions of lines per day with an order of magnitude of headroom, and a single machine under attack — tens of thousands of requests a minute — must be parsed, evaluated and shipped without the protected service noticing; both figures are held as benchmark gates like §5.4's throughput. Back-pressure degrades in a fixed, documented order: ingestion latency first, then local queue depth, then the oldest queued events (logged, metered) — never live parsing on the agent, never enforcement, never the protected service. And when lines are written faster than they can be read, the reader serves detection from the newest lines first and back-fills history later: detection latency stays bounded at any write rate, completeness is history's concern, not detection's.

5.2 Rules: language and safety#

A rule is data. It is written by an operator in the panel or in a file, and the controller pushes it to the agents, which evaluate it locally. That is what makes a sanction on the attacked machine immediate, and what lets a machine keep defending itself when the controller is unreachable.

Pushing rules is only safe under a strict constraint, and it is the reason principle 7 exists:

• The rule language is not executable. No command, no shell, no plugin, no arbitrary file path. A rule expresses patterns, conditions, thresholds and a choice of sanction — nothing else.
• Sanctions are a fixed set compiled into the agent. A rule selects and parameterises one; it can never supply one. Whoever controls the controller must not thereby control root on every protected machine.
• The controller never pushes code. Binaries come from the release channel and are verified against a key that is not the controller's (§6.1).

Regular expressions are the sharp edge of the language, and are handled as such:

• RE2 only — Go's regexp. Linear time is guaranteed, so catastrophic backtracking (ReDoS) is impossible by construction. The cost is no backreferences and no lookaround; that trade is accepted, and no backtracking engine may be added for compatibility.
• A mandatory literal prefilter. Each parser declares a cheap substring that must be present before its regex runs. This removes the overwhelming majority of lines without touching the engine.
• A pattern lives in a parser file, or inside the rule that scores it — a match entry may write pattern and prefilter in place of a parser name, and the loader makes a parser of it (<rule>#<n>), so that a rule with no reason to share its regex keeps regex and rule in one file, and everything downstream (evidence, cost accounting, the console) sees one kind of parser.
• Cost is measured, not estimated. Replaying a rule over real history yields a concrete ns/line figure. Each rule carries a CPU budget, with a shipped default; a rule measured over budget is not pushed.
• Per-rule CPU and memory accounting on the agent — sampled, so the measuring itself costs nothing worth counting. A rule over its budget is warned about locally and reported to the controller with its measured figures, so the fleet's dashboards show which rules cost what; the agent never disarms a rule on its own — switching detection off is an operator's act, and an automatic one would be a way for load to blind the machine. Rule state is bounded too: the groups and distinct values a rule tracks have a cap, and eviction is a metric, never silent. A rule must never be able to kill the machine it protects — by cycles or by memory — which the process-wide ceiling (§6) guarantees, not a kill switch per rule.
• No regex is ever built at runtime from data an attacker controls.

5.3 Detection#

Rules are the core of the product. They support:

• counting and thresholds over sliding windows — counting events, or counting distinct values of a field, so a range rule can require three distinct attacking addresses rather than three requests before it fires: proof of multiplicity, not volume;
• several thresholds on one rule, each with its own window — five in a minute or ten in ten minutes — and the rule fires when any one is crossed. One rule, one sanction: a crossing while a sanction is already in force is absorbed by escalation (§5.4, check 5), never duplicated, and the trigger records which threshold fired;
• thresholds expressed as a count or as a share of a finite population, so a rule watching a list can fire when a fifth of its members have been seen attacking;
• grouping by ip, range, asn, country, session, user-agent family, membership of a list, or an arbitrary key expression;
• conditions over any parsed field;
• what it does: a sanction (a policy by name), a notification with a severity (§5.11) — besides the sanction or instead of it —, or, with neither, a recorded trigger only.

Whatever the grouping, the offender is where the traffic came from — an address, a range, an ASN or a country. A rule keyed on a request property (a user-agent, a path, a host) sanctions the addresses that exhibit it, not the string that revealed them. Thresholds are scaled by the offender's trust (§5.4).

Weighting. A rule can count two populations rather than one — hits it considers bad against traffic it considers good — and trigger on their ratio. A busy legitimate client producing a handful of failures among thousands of successes is then not treated like a scanner producing nothing but failures.

Two conditions coexist, and either one triggers on its own:

• a ratio over a window, with a minimum volume below which the ratio is not computed at all, since 2 failures out of 3 requests means nothing;
• an absolute ceiling on bad hits, which triggers regardless of how good the ratio looks. Without it, an attacker hides behind volume: enough success traffic makes any amount of abuse proportionally invisible.

Weights are per hit, so a rule can price a failed login above a 404. What counts as good is declared explicitly, never inferred.

Levels. The same rule wants different thresholds on different machines — the same failed-login pattern on a hardened box and on a shared-hosting server — and writing the rule twice is how the two copies drift. So a rule writes its numbers once, and the machine's level scales them, the same way for every rule, in a way a person can predict: instant — the first hit convicts; strict — half the hits; standard — as written; lenient — twice the hits. A machine runs at one level (standard unless said otherwise), set for the fleet, per agent, per pack on an agent, or per rule on an agent (§5.9); a rule may also be given explicit numbers on one machine without touching its file. (Rules carried a set of thresholds per level until 2026-08-18; that spelling was retired for this one — one number set per rule, one knob per machine.)

Where a rule runs. On the agent — the cheap, high-confidence detection with an immediate local sanction and no round trip. Rules that need what only the controller holds (§5.1 enrichment, sessions, lists) are the controller era's, and their placement is the controller's business when it evaluates; the rule file does not say where it runs. Correlation across machines necessarily runs on the controller. Agent-placed rules cost some duplicated parsing on the protected machine; that is the price of immediacy and of surviving a controller outage, and it is bounded by §5.2's CPU budget.

Architectural constraint, non-negotiable: the rule evaluator is a pure function of an ordered event stream and rule state, driven by two interchangeable sources — the live stream and a range of stored history. This is what makes a replay exact rather than approximate, and it is what makes rules debuggable at all. The same evaluator code runs on the agent and on the controller.

Order and time. The canonical order of the stream is by event time, then agent, log input and source position — a total order, the one a replay always uses. Live, events may arrive late (a queue draining, a lagging machine), so the evaluator is order-tolerant: every event counts in the window of its own timestamp, whenever it arrives, and a threshold fires when the evidence completes. The equivalence promised between a live run and a replay is exact on substance and free on wall-clock: same events, same hits, same triggers, same offenders — the moment a trigger fires live may lag by the lateness of its evidence, never differ in what it decides. Agent-placed rules read their own logs in source order and are exact by construction. Clock skew between agents is measured by the controller on every batch, exposed as a metric and alerted on; time synchronisation across the fleet is an operational prerequisite, not something the evaluator pretends to repair.

Replay is therefore a core feature, not an analysis tool: write a rule, run it over a chosen range, get back the hits, the triggers, the sanctions it would have produced, the offenders touched, the overlap with existing sanctions, and any collision with exemptions or own infrastructure. Then arm it. The same mechanism serves the other direction: an operator arriving after an attack began replays the range and enforces what it finds — a crossing counts only if the sanction it earns would still be in force, window and ban duration. test covers the forward-looking equivalent, with a comparison of observed versus enforced outcomes over time.

Deferred, not dropped: behavioural baselines (learned per-domain, per-endpoint, per-network normal rates, with deviation alerting), pattern mining that proposes rules automatically, cross-machine DDoS correlation with graduated response (observe → ban the address → ban the range), linking offender populations across attacks — the same botnet resurfacing with fresh addresses, seen as overlapping lists — and machine-assisted analysis that reads history to explain an incident or draft a candidate rule: proposals only, armed by a human (principle 3). These are out of scope for the first releases. The data model and the evaluator must not make them impossible later, which requires exactly what is already mandated: keep the raw lines, keep the fields addressable, keep every rule runnable on the past.

5.4 Sanctions and enforcement#

Scopes: ip, range, asn, country, and the members of a list (§5.8). useragent and session are detection keys, not sanction scopes: a firewall cannot see them, so a rule grouped on one sanctions the addresses behind it (§5.3).

Enforcement is nftables. Sets with per-element timeouts, giving native ranges and native TTL, updated atomically, and reloaded from the agent's local copy after a reboot. There is no web-layer enforcer today (§11.3).

A ban carries a port scope: the whole machine, or a set of ports — typically the ports of the service the rule protects. The policy ladder can widen it with the offence: first strike on the mail ports, recidivism everywhere. Two properties are mandatory whatever the scope:

• the match is on traffic to the machine — packets belonging to connections the machine itself initiated are never dropped, whatever their source, so a range ban can never cut the replies of an outside service the fleet depends on;
• a ban takes effect immediately: the offender's established connections are terminated when it is applied, not left to finish their work.

Three consequences follow, and they are accepted, not overlooked:

• ban is the only sanction type. throttle and challenge are not expressible in a firewall and are therefore out of scope. They can only return alongside a web-layer enforcer — an open direction (§11.3), not a plan.
• A ban has per-port granularity but no per-domain granularity. nftables does not see virtual hosts: on its ports, a ban blocks the address for every site the machine hosts.
• An offender behind a proxy that legitimate traffic must keep flowing through is detectable but not enforceable at the firewall. When a CDN fronts a site, the log may carry the real client address while the packets carry the proxy's: banning the client drops nothing, and banning the proxy is collateral (check 3 below). Accepted and visible, not papered over.

What makes a scoped exemption meaningful is not where the block applies but what is checked before deciding.

Checks before applying a ban, in order. Exemptions and the own-infrastructure list are pushed to every agent alongside the rules, so checks 1 and 2 run wherever the ban is decided — locally, with no round trip, for an agent-placed rule. Checks 3–5 need history and run only on the controller: a locally decided sanction is therefore always a single address at its policy's first rung, and the controller re-runs the full sequence on receiving it — confirming, adjusting or lifting it — before any propagation.

1. Own infrastructure — permanently exempt, and no rule, import or bulk action may override it. Checked first so nothing else can reach it.
2. Exemptions, evaluated per hit rather than per offender: a hit against a resource whose owner exempts this address does not count, and an exemption may be conditional — exempt this offender, except on these paths. The rule is then re-evaluated on the hits that remain. An address attacking one owner who exempts it and another who does not is still banned, on the second owner's evidence alone.
3. Shared origin — how much of this address's traffic is legitimate, from real history. Carrier NAT, corporate egress, a CDN or proxy front all produce abuse and normal traffic from the same address, and banning it costs the normal traffic too. Above a configurable collateral threshold the sanction is refused, or held for the operator.
4. Expansion guards (below) when the sanction is a range rather than an address.
5. Existing sanction — an active sanction is escalated or expanded, never duplicated.

Checks before lifting a ban. Lifting is a recomputation, not a deletion: the owner's hits are removed from the evidence and the rule is evaluated again on what is left.

1. The requester owns resources that actually produced hits in this sanction — otherwise there is nothing for them to lift.
2. The remaining evidence is re-evaluated. If it still crosses the threshold, the ban stands, and the requester is told that other evidence sustains it, without being told whose.
3. Policy may forbid lifting an escalated or permanent sanction, and may forbid it entirely for a given owner.
4. The lift is recorded as evidence in its own right: an address repeatedly exempted by one owner and repeatedly banned on others' evidence is a signal, not noise.
5. A lift may say "and withdraw the report": what the machine reported about the address (§5.5) is withdrawn at the provider where the provider allows it, and the report's record says so. The operator who found the ban wrong does not leave the report standing; nothing is withdrawn without being asked.

Every one of these checks reads local state only. None of them may consult anything an attacker controls, and none may block the hot path (principle 4).

Durations are a policy, not a rule setting. A policy is a named ladder — first offence, second, third, and so on up to permanent — with a memory: an offender quiet for that long starts back at the first rung (memory: in the file; "decay" in earlier text). Quiet time is measured from the moment their last sanction ended, never from when it began — an offender cannot reoffend while banned, so a long ban must not consume its own memory and hand the returning recidivist a first-offence duration. A repeat is judged per machine and per address, whatever the rule: an address banned under one rule then caught by another is a returning offender. The policy also says which ports a ban covers (all, the service's as the machine declares them, or a list) and may let the controller lengthen a ban when the provider's reputation for the address is at or above a bar — after the fact, never in the path of the sanction (principle 5). A rule references a policy by name; it does not carry durations. Shipped policies cover the common cases, so the usual answer to "how long should this ban be" is choosing a name, not writing a number. Escalation counters survive expiry (§4.1), which is what makes repeat offending visible in the first place.

A policy is written either as an explicit ladder or as a progression — a base duration, a growth factor, a ceiling. Ladders need not be gentle: a banned address cannot reoffend while banned, so gradual doubling spends the sum of every rung in cumulative ban time before it protects anything, and "short first offence, straight to the ceiling on repeat" is a shipped policy, not a hack. Shipped policies lean long: the platform's guards — replays before arming, weighting, trust, the collateral checks — exist precisely so that long bans are safe, and the goal is quiet logs, not short punishments.

A policy may also tighten the evidence a returning offender needs (tighten: 2x): each past episode within decay divides every rule's threshold count by the factor, never below one hit — ten hits convict the first time, five the second, three the third. Duration and evidence escalate together, so a recidivist is caught faster and kept out longer; the counter decays with the same decay. Once a rule convicts, every counter it holds for that offender starts again — a rule with several thresholds is one rule and gives one verdict; the hits already answered for do not go on filling its slower windows (that is what escalation is for).

Where a duration starts is the policy's choice: at the evidence or at the application. The two only diverge when evidence arrives late — a cut-off agent draining its queue, an operator acting the morning after. Counting from the evidence makes a stale short ban expire on arrival, leaving history and escalation but no enforcement; counting from the application is the proactive stance — a ban decided today protects from today, whenever the attack happened. Shipped policies count from the application. Either way the trigger records both times, escalation counts the offence, and reports carry the attack time, never the discovery time. Every rule chooses its policy; durations are never decided by a global chain that individual rules cannot override. And the data plane must honour the duration or refuse it visibly: the agent reports the TTL it actually enforced, and an enforcement layer that silently clamps a long ban has turned the policy into fiction.

Trust is graduated, not binary. Between exempt and neutral lies a scale, and it is a named object, not a condition repeated in every rule. A trust entry names an origin — a country, an ASN or a range; never a target, origins and targets must not be confused — and adjusts how offenders from it are treated: a multiplier on rule thresholds — an address from the operator's own market must produce more evidence than a bulletproof host before the same rule triggers — and bounds on sanctions: never range-ban inside this country, never expand across this ASN, cap the duration, hold for the operator instead of applying. Trust in one direction is severity in the other; the same mechanism marks hostile origins down as it marks home ones up. It never removes logging, and it is not an exemption — the top of the scale is where exemption begins. The shipped default is neutral everywhere: trust is the operator's statement about their own traffic, never shipped geopolitics.

Expansion is built in, not a separate rule. Writing one rule per scope is duplication, so widening is a property of the sanction:

• When several distinct sanctioned addresses fall inside the same prefix within a window, the sanction widens to that prefix, and may cascade further — /24 → /23 → … — up to a hard floor that no policy can cross.
• IPv6 sanctions default to the /64, because a single address is a meaningless unit there; expansion then runs /64 → /56 → /48.
• Widening to the autonomous system is a distinct, harder-guarded step.
• The operator can expand or narrow any existing sanction directly, from the CLI or the UI, without touching a rule.

Guards on expansion, all mandatory:

• never across an exemption, and never covering own infrastructure;
• never inside an origin whose trust forbids it: a trusted country or ASN is not range-banned, however many of its addresses individually earned sanctions;
• a cap on the total number of addresses a single sanction may cover;
• a check against real history before widening: the platform knows how much legitimate traffic that prefix produced, and refuses — or asks — when the collateral exceeds a threshold. This is the reason to keep the whole history, applied to the decision that most often goes wrong.

Distribution. Sanctions propagate through the controller, never agent to agent, and propagation is a choice, not a constant. The pack is the unit of sharing (decided 2026-08-20): a sanction of rule X decided by machine S is applied by machine R exactly when X belongs to a pack both S and R run, S's declaration offers, R's declaration enforces, and the rule does not say share: no — that veto is absolute, no pack overrides it. What follows from the definition, and is intended: a rule outside every pack never travels (machine-specific rules stay local); a machine with no declaration shares nothing (it runs no packs); a pack with one subscriber shares with nobody; distributed copies (their origin names the source machine) are never redistributed. Sharing SSH failures fleet-wide while a hosting pair shares its web bans only with each other is expressed by the packs the machines already run, not by extra configuration; finer sharing than a pack is expressed by splitting the pack. A sanction lifted or shortened at its origin — by the operator or on the machine itself — is lifted on the machines that applied its copy. A machine that starts sharing — enforce switched on, a pack added, a machine enrolled into a pack — is served every still-active shared sanction, never only what comes next: joining the group means holding what the group holds. And a copy this machine applied counts toward escalation like its own record once it has run out: an offender the group already sanctioned climbs the ladder here, never a first offence again. Per agent, two switches govern whether its local sanctions are offered to the fleet and whether it enforces the fleet's. A third per-agent setting names who may lift its sanctions: whoever the owner model allows (the default, §5.6), or the operator alone — an independent machine whose bans no customer, portal or automation token can lift, whatever powers they hold elsewhere. Every lift, wherever it is asked from, is a controller decision run through the checks above; the agent applies the outcome. Whatever the switches, events and evidence always flow — excluding a node from global enforcement never excludes it from observation. Agents hold a persistent local copy of what they enforce, so a controller outage neither unbans nor blanks the ruleset.

Pipelines are multithreaded throughout, transport is chunked and batched by default, and the originating agent is a distinct, unqueued delivery target.

Enforcement sheds detection load. A banned address is dropped before the protected service, so it stops producing log lines the moment the ban lands: the faster the first local ban, the smaller the flood the rest of the pipeline ever sees. Under a flood the system converges to fewer, wider sanctions — escalation deduplicates, expansion and list-wide bans replace tens of thousands of individual entries — and the data plane receives batched atomic set updates, never one entry at a time.

Intense DDoS — hundreds of connections per address — is met below the logs. Three mechanisms, all local to the attacked machine:

• Kernel guardrails, shipped. The agent's nftables table carries optional static limits — concurrent connections per source, new-connection rate per source — enforced by the kernel even while userspace drowns. They are parameterised like sanctions: a fixed compiled set, tuned by config, never supplied by it.
• The connection table is a log_input (§3): sampled per-source connection counts and byte rates (conntrack accounting) reach the evaluator as ordinary events, so an agent-placed rule bans a hundred-connection source — or one saturating bandwidth — without parsing a single log line. The connections the web server never logs (refused past its limits, held open, never completing a request) are exactly what this source sees.
• Detection reads the present (§5.1): when lines are written faster than they can be read, detection is served from the newest lines first and history back-fills later — detection latency stays bounded at any write rate.

One honest limit: a flood that saturates the physical uplink is defeated upstream, not on the host. The machine's job is to refuse to amplify — drop early, kill the flows, widen the ban — and to say plainly, in its metrics, that the pipe itself is full.

5.5 Abuse reporting — core, not a plugin#

Off unless configured. Reporting is an egress of personal data and an act in the operator's name; it never happens because a default said so.

There is no report queue. A sanction already records everything a report needs, so the report's state lives on the sanction itself — whether it was sent, when, to which provider, and what the provider answered. What would have been a queue is a query: the unreported sanctions, ordered by whatever the operator wants to prioritise.

That is not a simplification, it is the only bound that holds. The measured worst case is five million distinct offenders in a day against an allowance in the thousands; a queue of the difference is a structure that grows for ever and drains in a decade, and it is the one part of the system nothing would bound. As a flag on data already kept, the backlog costs nothing and disappears with the retention it belongs to.

• The allowance is the operator's subscription, written in the configuration — not guessed, and not a constant compiled in. The agent counts what it has sent against it and stops attempting rather than discovering exhaustion by being refused. A defence that answers a flood by flooding somebody else's API has failed twice.
• A report gets one chance, at the moment of its detection. Whatever stops it — the allowance spent, the provider unreachable, the credential refused — the outcome is logged, the sanction stays on record still unreported, and the agent does not come back to it: no retry cadence, no automatic backlog drain. An agent that drains a backlog by itself is describing last week as if it were happening now, and spending the operator's allowance without being asked.
• An address that keeps attacking while banned is reported again. A ban extended because the hits go on (check 5) reopens its report once the last one is older than the renew interval (a day by default; the operator sets it, or never): the provider hears that the abuse continues, and the operator sees a report as recent as the attack. A report that never left is not renewed — its one chance was taken.
• Exhaustion is temporary and the agent knows it. Reporting resumes — for what is decided from then on — at the provider's reset, read from its response where the provider says so and assumed at the daily reset otherwise, with a bounded probe meanwhile so an upgraded plan or another machine freeing room is noticed the same hour.
• Reporting after the fact is the operator's deliberate act, a command that selects the unreported by a window the operator names, shows the count before anything leaves, and submits freshest first. Providers discount a report that arrives long after the events it describes, so a backlog drained a week later is worth less than one drained the same hour — which is exactly why draining it is a decision, not a default.
• Every outcome is recorded and distinguishable, because they demand different answers: accepted; refused as a duplicate of a recent report; refused as malformed, which will never succeed and must not be retried; refused for allowance, which is not the report's fault; and the provider being unreachable, which is. The operator sees the counts without reading a log.
• Reporting never touches the path that decides and enforces. It runs apart, on its own schedule, and a provider that is slow, rate limiting or down has no way to slow down a ban. A report is a courtesy to the internet; stopping the attack is the job, and the two must not share a thread. A burst of bans is a burst of requests — a provider expects that from a machine under attack — and when the allowance runs out mid-burst the sanctions keep being written at full speed, unreported and on record.
• A test trigger is never reported. What was not enforced is not submitted; simulation and observation windows produce no egress.
• Whether and from which sighting a rule's sanctions are reported is set per rule — first offence, repeat offenders only, or never. A botnet cycling hundreds of thousands of throwaway addresses a day must be bannable without burning the daily allowance on hosts that will never be seen twice; its offenders stay in the store for deferred bulk submission at allowance pace.
• Providers constrain scope. A provider that only accepts single addresses receives the member addresses that individually earned their sanction, never the range.

What leaves, and who decides

A report's content is a template, and templates are data. Only the provider's API — its endpoint, its authentication, its category vocabulary, how it answers — is compiled in. What a report says ships as presets, one per kind of attack, and an operator may replace any of them without touching the program. The same reasoning as rules: the engine is fixed, the content is theirs.

Redaction is default-deny, and it is the point of templating. A report leaves the operator's machine and is published under their name; what it must never carry is their users. The evidence a template emits is composed from named parser fields, never from a raw log line, so a field that the template does not name cannot leave by accident. The shipped presets are written that way and say why:

• mail — the offending address, what was attempted, never the mailbox or the address it was aimed at;
• SSH — that authentication was attempted and refused, never the account names tried;
• web — the method, status and the shape of the request, never which hosted site received it.

The rule is not "strip what looks sensitive" but "emit only what is named", because the first is a judgement made once and the second is a property.

Who does the reporting

Reporting can be done by the agent or by the controller, and this is configuration, not architecture: both run the same state machine, the same allowance arithmetic and the same templates.

• Who reports a machine's sanctions is set per agent, on the controller: the agent itself, the controller on its behalf, or nobody. An enrolled agent applies what it was assigned; a standalone agent's own configuration is the declaration. The controller reports only what its agents do not, so the same offender is never submitted twice for the same sighting, and neither side has to ask the other at the moment of sending.
• The agent is the right sender when egress must come from the protected machine, or when there is no controller at all (§10). The controller is the right sender when a fleet shares one allowance, or when reports should come from one known address — some providers weigh the reporting address's reputation.

Credentials live in their own file, not in the configuration alongside rules and inputs. One file, restrictive permissions, holding the provider's key and the operator's allowance — separate because a file that exists only to hold secrets is a file that is unambiguous to exclude from a backup, a copy or a repository, whereas "the config, except the parts with keys in them" is a judgement someone eventually gets wrong.

Outbound hooks

Beyond providers, an operator may bind their own action to a sanction — a webhook, a message, a script of their own.

A hook is local operator configuration and can never be pushed. The controller may not supply a hook, a command, an address or a path, and an agent ignores any attempt to send one. This is not a restriction on hooks, it is what keeps §5 true: the day the controller can name a command to run, whoever controls the controller controls root on every protected machine, and every other guarantee here is decoration.

Hooks are subject to the same rules as reports — off by default, templated content, default-deny redaction — and to one more: a hook that is slow or unreachable must never delay a sanction. Enforcement does not wait on anybody's network.

5.6 Ownership: who may see what#

The platform is operated by a hosting provider whose customers — and resellers — get scoped visibility. Scoping is not per machine: one machine hosts many customers, and one customer may own a whole machine.

• An owner holds resources of three kinds, and their scope is the union:
  • domains — a shared-hosting customer sees the traffic of their sites;
  • machines — someone renting a dedicated server or VPS sees everything on it, not one vhost of it;
  • address ranges — the addresses assigned to them, covering protocols with no domain notion and machines they run themselves.
• Hierarchy: reseller → client → resources. A reseller sees the aggregate of its clients. Both levels are optional; a deployment with no customers has one implicit owner holding everything.
• Attribution happens at ingest and is mandatory: each event is bound to the domain it targeted, to its machine, and to the local address it arrived on. Ownership resolves from those three keys. Events matching no known resource land in an explicit unattributed bucket, visible and monitored, never silently discarded.
• Overlapping ownership is normal and resolves deterministically: visibility is granted by any matching resource, and the most specific owner is shown as responsible.
• Isolation is a security boundary, not a display filter. A customer must never observe another customer's data by any path — UI, API, CLI, export or feed. Scoping is applied in the query layer so that no endpoint can omit it.
• Customer powers are chosen by the operator, per owner. The full set is: read the events, hits and sanctions touching their resources; look an offender up and see what it did to them; unban; add an exemption. Each is granted or withheld individually, so a deployment can run anywhere between read-only and fully self-service. Never available at any setting: fleet-wide data, another owner's scope, editing rules, changing policy, or the reporting subsystem. Every customer action is audited like an operator action.
• Owner access is direct, delegated, or both. An owner may hold their own account on the scoped UI, and the operator's existing portal may drive the same actions through an owner-scoped API token (§6) — a deployment enables either or both.
• A scoped action never widens into a global one. An exemption or an unban covers the hits produced against that owner's own resources and nothing else (§5.4). It never revokes a fleet-wide sanction, never reaches a machine outside the owner's resources, and never suppresses evidence gathered on behalf of another owner.
• Aggregate reporting per client and per reseller is a first-class output, in the UI and the CLI.

5.7 Public surface: exposure, amnesty, owner access#

Each instance decides how much of itself it shows. The default is nothing.

• Publishing bans is opt-in and granular: none, a curated list, or a whole category, with per-list visibility (public, unlisted, private). What is published carries the reason and the timestamps, never the customer-identifying context that produced it.
• A banned offender can file an amnesty request. The policy is per instance: refuse outright (no public endpoint at all), manual review, automatic under conditions (first offence, low severity, minimum age, proof of work or captcha), or automatic with a cooldown before it can be used again. The policy is config like everything else, and is audited. The default is manual review: the endpoint exists on a fresh install, and requests queue for the operator. Automatic granting ships in the first release but is off by default — it is a public path to lifting a sanction, so enabling it is a deliberate act, not something inherited from an installer. Granting an amnesty runs the same checks as any other lift (§5.4), so it can never clear a ban that other evidence sustains.
• Domain and machine owners get a scoped window onto the same data (§5.6), and may lift a ban within their own scope where policy allows. Same permission model as the operator UI, narrower scope, not a separate system.

This surface is attacker-facing and is specified as such:

• The amnesty endpoint is never synchronous with enforcement (principle 4). A request enters a queue and is evaluated out of band.
• It must not become an oracle: responses reveal nothing beyond what the instance already publishes. No unauthenticated party may enumerate who is banned, which rule triggered, or what evidence exists.
• Rate limiting per address and per range is mandatory, and a flood of requests is itself a detectable signal.
• Automatic amnesty is bounded: never for an escalated or permanent sanction, and never for a sanction whose scope is wider than a single offender.

5.8 Lists: own, publish, share, ingest#

• Own lists: named, reproducible sets of addresses and ranges built from queries or curated by hand, each membership carrying its reason, source and timestamps.
• Membership is evidence, not a sanction. No list — own, shared or imported — bans anything by itself.
• A rule can watch a list (§5.3): group on its membership and trigger on collective behaviour — a share of the members individually flagged, or a number of distinct members producing hits within a window. Its sanction may then cover the whole list (§5.4): addresses that acted in concert are banned in concert, while everything below the trigger keeps flowing and keeps being logged — which is what preserves the sample that identifies the attack and where it comes from. A collective ban is the widest sanction in the product and passes every §5.4 check like any other: exemptions, trust, the collateral guard and the address cap included.
• Publishing: a public frontend — browsable and searchable — plus machine-readable feeds (JSON, plain CIDR, signed snapshots).
• Sharing: instances subscribe to each other's lists over an authenticated, pull-based, signed and versioned channel.
• Third-party ingestion, treated as hostile input by construction:
  • every source is namespaced, quarantined, never merged into own data;
  • a source may only propose, subject to local exemptions, a maximum entry count, a maximum aggregate prefix size (no /0, no accidental country-wide ban), and sanity checks against own infrastructure;
  • imports are staged in test with an impact report computed from real history before they can be armed;
  • full provenance per entry, one-command revocation of an entire source.
• Published lists, as built (2026-08): a list may name a URL (or several) and a format instead of a network, a country or prefixes; the controller fetches it on its cadence, reads it into prefixes and nothing else, bounds it (count, width, coverage, public addresses only; a line that is not an address refuses the whole fetch; a failed fetch keeps the last good file), and pushes it to the machines in its block scope as data — one file in their bundle — which the agent holds in sets of its own behind one drop rule and one counter per list: no sanction, no expiry, a diff at every refresh, and never the machine itself, its controller, or what the operator exempted. A catalogue of such lists ships, each with its attribution; an operator adds their own by URL. The same list with an exempt scope is an allow list.

5.9 Management: UI, CLI, files at parity#

• Config files are the single source of configuration — parsers, rules, exemptions, trust, policies, owners and their resources, the agents and their log inputs: plain text, versioned by the operator with whatever tool they already use, deployable by anything that can copy files. The format is YAML — one format, no alternatives — and the remote control writes it back in one canonical style. Hand-written config is the primary interface: the common case is a handful of flat keys with a default for everything else, compact scalar forms where they read naturally, and nesting only for what is genuinely structured. A rule an operator writes by hand in five lines is the bar, not the exception. A configuration revision loads atomically: fully valid, or refused whole with visible errors — never partially. A dry-run validator catches errors before load.
• The controller's configuration is the fleet's, in the same files. Its tree holds the same parsers, rules, policies, report presets, trust and exemptions an agent reads — those at the root are the defaults, what the library — plus packs (packs/<name>.yaml: a named list of rules, the set for one type of server), plus one file per agent holding its declaration (§8), the packs it runs, the rules it adds by name, and its overrides: the level it runs at, per pack a state or a level, and per rule a level, explicit numbers, a policy, or off. A rule is never copied, only listed: a machine runs the union of its packs and the rules it names, once each, and receives only what those rules need. That is the unit of attribution, and what a paid or curated set of rules reaches is decided here, by name (§6, the controller decides what an agent may have). Parsers, policies and presets are shared, never overridden: a machine that needs a different regex needs a different rule. The controller assembles each agent's configuration from defaults and overrides, validates the result exactly as the agent would, and refuses the revision whole if any agent's assembly fails. What an agent holds is what was assembled for it, written to files on the agent too, so shieldlist-agent -t and a person read the same thing there. An enrolled agent's rule language — parsers, rules, policies, presets, trust — comes from the controller alone; its agent.yaml, secrets, hooks and local exemptions stay its own and add to what it receives.
• Published exemption sources are refreshed, never typed. Search engines and CDNs publish their address ranges — Googlebot, Bingbot, Cloudflare — and an operator exempts them by naming a shipped source, not by pasting ranges. The controller fetches a source on command or on a schedule (opt-in outbound, principle 5), materialises it into a generated file under exemptions.d/, marked as such and never hand-edited, records when it was refreshed and what changed, and distributes it like any other exemption. A refresh is never in the path of a decision; a source that cannot be fetched keeps its last good file and says so. A standalone agent runs the same refresh from the CLI.
• The API is the surface; the UI and the CLI are its clients. Every operation the web interface offers is an API call an operator token can make — creating, changing and arming rules included, since a config edit through the API is the same file write the UI performs — so a script, a portal or an automated operator drives the platform without a browser. Inspection is one lookup: is this address banned, where, by which rule, since when, until when — fleet-wide for the operator, owner-scoped for everyone else.
• The UI and the CLI's config commands are a remote control for the files. They do not write to a database beside them; they edit the files themselves — canonical formatting, stable ordering, comments preserved, minimal diffs — and the controller validates and reloads. Whoever wants simple uses the interface; whoever wants git, review or automation edits the files directly, and each sees the other's changes. A concurrent on-disk change refuses the UI write rather than overwriting it. Confirming a discovery proposal (§5.1) is the same mechanism: it writes the declaration into the files.
• Runtime decisions are not configuration. Bans, unbans, owner-scoped exemptions, amnesty outcomes and query-built lists are operational state, recorded in the store like any sanction — shield ban writes no file.
• CLI covers every operation, is scriptable, and emits JSON on demand.
• Web UI provides live situational awareness, investigation (search history; pivot on offender, range, domain, machine, parser; and reviewing what was not sanctioned — the attacks let through are a first-class query, not a test special case) and operations (arm rules, ban, unban, exempt, bulk actions over history, manage the report queue and the amnesty queue). Live views aggregate server-side; no client-side mirroring of the event stream.
• Full audit trail either way: a UI edit and a file deploy both land as a configuration revision, hashed and diffable — who changed what, when, why.

5.10 Observability and operations#

• Metrics: health, ingestion lag, detection latency, enforcement coverage, report queue depth, amnesty queue depth, local queue depth per agent, per-rule CPU cost.
• Alerting hooks (webhook, email, chat) on platform health and on security events, with severity routing.
• Backup and restore of the whole state; an agent rejoins and resynchronises without manual intervention.

5.11 Machine watch: the same pipeline, pointed at health#

The platform sits on every machine, reads every log and knows who owns what. Watching health is the same job as watching attacks, and it reuses the same primitives — events, parsers, rules, alert routing (§5.10). A monitoring feature that would need a second pipeline does not belong.

• A rule can alert instead of sanction (§5.3): relevant log errors — 500s rising on a site, deprecation warnings after an upgrade — are ordinary rules over parsed fields, with a severity, routed like any alert, and shipped as rule sets like everything else.
• Probes are a log_input. The agent requests the sites its machine hosts — or a peer agent does, for network realism — and records outcome and response time as events. Availability and latency alerts are ordinary rules over those events, and the payoff is the cross-reference: when a site stops answering, its last minutes of logs are already in the store, so bug and attack distinguish themselves in one query. Probes target the fleet's own hosted sites, never third parties.
• Basic system metrics — cpu, memory, disk, network, connection counts, and on hosting machines per user and per process, which maps to owners (§5.6): which site is burning the box is an ownership question. Sampled by the agent, kept as history, alertable by rules. Curves and thresholds, not a profiler: the ambition is the cross-reference with the logs, not the thousand charts.
• An anomaly rate — how unusual is this machine's or this domain's traffic right now — needs the learned baselines and lands with them (§5.3).

5.12 Bursts and mitigation: recognise the pattern, not the origin#

A flood is not recognised by where it comes from but by what it does. The agent already reads every line; mitigation is the same pipeline asked two more questions — is this burst normal? and what do the abnormal requests have in common? — and answered with ordinary rules (§5.2), which is what lets a counter-measure exist without a line of code.

A baseline, always on. Per target and per machine, the agent keeps cheap rolling counters at one-second resolution: pages and assets served, status mix, share of requests without a referer, new addresses per minute, user-agent diversity, and per address the regularity of its request intervals (a bot ticks, a person bursts a page and its assets then pauses). Smoothed by hour of day they are the machine's normal. A burst is a departure from it above an absolute floor. A burst is a fact about volume, never a verdict.

A burst is not an attack. A site on television is a burst of thousands of addresses each behaving like a person: a page, its assets, a referer from the second page on, sessions that progress, diverse current browsers, an audience shaped like the site's. A flood is the opposite on at least one axis: one or two URLs, no assets, no referer, regular intervals, a narrow set of user-agent strings, a rate no person reaches. The question is asked of the population, not of one visitor: a visitor who only loaded the home page is not a bot, and is not one either when the server answers so slowly that nobody gets to load anything else. Signals that saturation itself produces — missing assets, errors, slowness — are therefore weighed only on requests the server answered normally, and an attack is never declared on volume alone.

Profiling. Declared a burst, the agent compares its window with a reference window (the hour before, the same hour the day before) on every axis it has: paths, assets-per-page, referer, method, protocol, user-agent strings, and the transport fingerprints the log carries (TLS version and cipher, JA4, accept-language — §5.1 names what a richer log format must include). Each candidate signature is scored twice: recall on the burst and cost on the reference traffic — the share of normal visitors it would hit. The signature kept is the combination with the best recall under a cost ceiling; at equal cost, the one that spares the most human-looking traffic. A country or an autonomous system is never a signature by itself: origin concentration raises confidence in a behavioural signature, it never replaces one.

The counter-measure is data. The outcome is a temporary rule: monotone (§5.3 — its verdict is instant), dated, with a time to live, switched off by itself when the burst subsides and a quarantine has passed, applied on the machine first and carried to the controller, which correlates — the same signature on several machines is a confirmed attack — and distributes it like any rule (§5.9). A machine alone does the same from its own files. Two modes: propose (the signature, its recall, its cost, one act to arm it) and automatic, bounded by the same cost ceiling and by every check of §5.4 — exemptions and own infrastructure win over a mitigation rule as over anything else.

The episode leaves a list. Every address the mitigation convicted is written to an auto-generated list named for the episode (§5.8): reviewable, shareable, publishable, and the seed of the next signature library. The episode itself — baseline, burst, signatures weighed, the one armed, its cost as measured afterwards — is recorded like a sanction is, with its evidence.

What it needs from the platform. Assets in the log (a front that serves static files without logging them hides the best human signal — the agent notices their absence and says so), and a log format carrying the transport fingerprints. Without them the profiling works on paths, referer, rate and user-agent — enough for the common flood, blind to the careful one.

Honest limit. A flood of addresses each sending one ordinary request with nothing in common is indistinguishable from an audience by the logs alone. Against it the platform protects the service rather than convicts the visitor: the kernel guardrails and the front's own rate limits (§5.4, DDoS below the logs), caching, and the origin layer the operator decides on.

6. Architecture#

Three binaries.

shieldlist-agent — one per protected machine, which both reads and enforces. Tails its log inputs, queues locally, ships raw batches. Evaluates agent-placed rules. Receives sanctions, applies them to the local data plane, persists them, reports enforcement state back.

shieldlist-controller — the central service. Internally structured as separable stages so a large deployment can split them later, but shipped and deployed as one binary until measurement says otherwise: ingest (receive, deduplicate, parse, enrich, attribute, persist), rules (evaluate live and replayed), sanction (exemptions, escalation, policy), distribute (push to agents, originating agent first), report (queue and providers), api, ui.

shield — the CLI, a thin client of the controller API, published from the controller repository and installed with agent and controller alike, so the platform is drivable from any machine in the fleet. It authenticates with an operator credential, never with an agent's, and holds no state or privileged path of its own.

The public frontend (§5.7) starts as a mode of the controller; it only ever talks to the API, so splitting it out later is cheap.

Transport is authenticated and encrypted. Enrolment is one join token: the agent registers itself and receives its own rotatable credential, granting only: submit its own events, receive rules and sanctions, report enforcement state.

Three kinds of credential, none interchangeable. Agents hold their enrolment credential (above). Humans hold local accounts — password, optional TOTP — used by the web UI; no external identity provider is required, and one can be added later without displacing local accounts. Automation holds scoped API tokens — bounded to explicit permissions and, where relevant, to one owner's scope, revocable, expirable. Tokens are what the CLI and scripts use, and what lets a hosting provider's own portal act on behalf of a customer (§5.6) without a second login. Every token belongs to an account or an owner, so the audit trail always resolves to someone.

The protocol is an ordinary HTTP API over HTTP/2, with protobuf payloads and zstd compression: batched POST for events, and a long-lived response stream from controller to agent for rules and sanctions. It travels wherever HTTPS travels, needs no extra runtime, gives the priority path to the originating agent for free, and can be reimplemented by a third party with any HTTP client.

Agents dial out; the controller never dials an agent. One long-lived, agent-initiated connection carries everything, multiplexed: the event POSTs and the rule-and-sanction stream share it. Outbound-only is what lets an agent sit behind NAT and a deny-all inbound firewall, with zero listening ports on the protected machine. After any cut the agent reconnects with jittered backoff and delivery resumes from the recorded position (§5.1); while disconnected it queues locally and keeps enforcing (§5.4). The open stream doubles as liveness: an agent's last-seen is the age of its connection, with no separate heartbeat machinery. On the shared connection, control preempts data in both directions: a saturated link delays event upload, never a rule or a sanction. A sanction travels with its convicting lines — what the agent kept at the decision, bounded there (twenty lines of a kilobyte) and bounded again on receipt — so the controller can show what was blocked, not only that it was; the event stream proper (§7) is a later phase and a different volume.

Configuration follows the same path. On every connection the agent presents the configuration revision it holds and the controller sends the assembled configuration (§5.9) when it differs; on the open stream a new revision arrives as soon as it is loaded. Where an open stream cannot be held — a proxy that cuts idle connections — the agent falls back to asking at an interval: same messages, same answers, only the latency differs. Nothing depends on a stream having survived; reconnecting is a full resynchronisation of configuration, sanctions and enforcement state.

Both sides know both versions. Agent and controller exchange their versions on every connection, and each shows both — shield status on a machine, the agent list on the controller. The controller advertises the range of agent versions it serves and the agent the oldest controller it can work with; a pair outside either range is refused with a message that names both versions and says which side is behind — on both sides, never a silent mis-parse or a protocol error nobody can read. Inside the range, additive is the rule: the controller never sends a key an agent's version would refuse, which is why it assembles configuration per agent knowing that agent's version.

The default port is 17453, not 443. The security plane does not share a port with the traffic it watches, and the controller must be able to live on a machine that also serves ordinary HTTPS; a high port also sits outside every common service range while staying below the kernel's ephemeral range. The port is configurable, and a deployment whose egress allows only 443 can bind the transport there.

Address filtering is on by default. Every listener carries its own address filter, closed by default — the public frontend is the only listener open by construction. The operator surface answers localhost until told otherwise. The agent transport answers enrolled agents' addresses; an unknown address gets exactly one narrow, aggressively rate-limited path — presenting a join token or an agent credential. Success records the address (a known agent reconnecting from a new address updates its record instead of being locked out); failure is dropped, and is itself a signal (§5.3).

Every audience gets its own listener. Agent transport, operator API and UI, owner UI, public frontend and metrics bind independently — address and port each — so each face can go on the network it belongs to: fleet transport on a private address or a VPN, operator surface on an admin network, the public frontend on another machine entirely. The frontend holds no state and talks to the API with a scoped read-only credential (§5.7): the machine that absorbs public traffic can be lost without touching the store or the fleet. Separation is a knob, not a prerequisite: the default install runs every face on one machine. The controller's machine runs an agent like any other — the security plane protects itself.

HTTP/3 is an option, not the default. The transport is selectable so an agent on a lossy or high-latency link can use QUIC, where head-of-line blocking hurts a long-lived stream most. It stays optional because QUIC runs over UDP, which a fair number of customer firewalls drop outright; an agent that cannot establish HTTP/3 falls back to HTTP/2 without operator action.

It scales down as far as one binary. The agent runs without a controller: it loads parsers, rules, exemptions, trust and policies from local config files (§5.9 — same files, same format), evaluates, enforces and persists its sanctions entirely alone — a single-machine deployment with no central service at all. It keeps a bounded local history, so a replay works on the recent past even with no controller. Attaching it to a controller later is enrolment, not migration: the files do not change. An agent is therefore in one of two modes: configured locally, the default, or enrolled — taking its rules, exemptions, trust and policies from the controller and reporting its activity, sanctions and enforcement state back.

A live tail joins the present. An agent that starts reads from the end of each file it watches, not from a position saved before it stopped. It was stopped for a reason, the file may have rotated meanwhile, and trusting an offset nobody watched is how a day of ordinary traffic gets replayed as if it had all just happened — which, on rules that count events in a window, means banning people for it. A file that appears after the agent started is all present and is read whole.

Reading a log's past is therefore a deliberate act with its own command, never a side effect of starting up. A replay evaluates events at their own timestamps and applies a sanction only if it would still be in force now — window and ban duration, not window alone. That is what lets an operator turn up after an attack has begun and still stop the addresses behind it, and what distinguishes this from a tool that only ever asks whether an event is recent.

The agent's working state is RAM-first and configurable. The local history and the queue live in memory by default, bounded, with disk as an option for either or both. Sanctions always persist: a restart must never unban (principle 6). Read positions are deliberately not persisted — a live tail joins the present (below) — and not re-shipping what was already delivered is the ingest's deduplication (§5.1), never a saved offset's promise. And all of it is embedded — the agent never requires an external service on a machine it protects: no database daemon, no cache daemon. Durable state lives in a single embedded SQLite-class store; the optional on-disk history is plain append-only segments, compressed, expired by dropping whole segments; everything hot is in-process memory.

The process itself is capped. Agent and controller each run under a configured memory and CPU ceiling — self-imposed by the runtime, and doubled in the shipped systemd unit — and saturation has a defined shedding order, not an OOM kill. Under memory pressure the agent sheds its biggest, least critical state first: the local history ring shrinks, then the queue, then per-rule state within its caps — never the sanction store. Under CPU pressure it cheapens work in order: compression level first, then discovery and probes pause, and past that the agent falls behind on its reads — with the costliest rules named in its figures (§5.2) so the operator knows where the cycles went — never enforcement, which is the kernel's work anyway, and never a rule switched off by the agent itself. Every shed is a metric and an alert, never silent. And if the process dies despite all this, enforcement does not: the nftables sets and their timeouts live in the kernel, and a restarting agent reloads them from its store — the failure mode is a detection gap, never an unban (principle 6). One machine can also run agent and controller together, frontend included — the simple install for a small fleet.

Scaling path, downward included: an agent alone → agent and controller on one machine → one controller, a fleet of agents → split store from controller → sharded rule evaluation and multiple ingest fronts → federated instances exchanging lists.

6.1 Distribution and auto-update#

Everything is open source. Agents run on machines that are not upgraded in step with the controller, so the two binaries are released independently and each carries its own version.

• Auto-update is opt-in per agent, verifies the release before replacing anything, and can be pinned or frozen. A failed update leaves the running version in place, and enforcement survives the restart (principle 6).
• The controller never pushes code (principle 7). Agents pull their binaries from the release channel and verify them against a key that is not the controller's, so a compromised controller does not become fleet-wide remote code execution. Rules are data and are pushed; §5.2 is what makes that distinction hold.
• Releases are published in the repository itself, as usable binaries, and as Debian packages first among distributions. The controller is a static binary with an embedded state store by default (§7), so a container image is a packaging of the same binary, offered alongside for the deployment that prefers it; only an external event store engine, if the benchmark chooses one, adds a service beside it.
• Every release is multi-architecture. ARM is a first-class target, not an afterthought: agents run on whatever hardware a customer rents.
• Protocol compatibility is a hard requirement, independent of release timing. The wire format is versioned and additive: optional fields, field numbers never reused, and the controller advertises the range of agent versions it supports. Too old is refused explicitly and visibly, never silently mis-parsed. The compatibility matrix is tested in CI across supported version pairs.

Repository layout follows from this. Two repositories, shieldlist-controller and shieldlist-agent, each with its own tag stream and release feed, which is what independent auto-update consumes, and which lets anyone audit or package the agent alone. The protocol is not a shared Go module: its schema and documentation are published from the controller repository as a public API, and the agent is one implementation of it, as a third party's would be. That costs a little duplication and buys the thing that matters — a breaking protocol change becomes impossible to make by accident, because nothing is shared that could quietly absorb it.

7. Storage#

Three workloads that do not want the same home.

• Configuration is the files (§5.9). The controller holds only its loaded image and the revision history; there is no configuration database to administer, and nothing that can drift from what the operator versioned.
• Operational state: the agents' runtime records and credentials, offenders, sanctions, owner-scoped exemptions, reports, amnesty requests, lists, audit, configuration revisions. Transactional, and not small — at the §5.4 throughput target the offender and sanction tables grow continuously and are read on every enforcement sync. Pluggable, SQLite by default, upgradable to MySQL or PostgreSQL, so it can live on whichever machine the operator chooses. One interface, one portable schema; no engine-specific feature may leak into the rest of the code. Whether the high-churn part (offenders, sanctions) stays here or sits with the events is part of what the benchmark must answer.
• Events: append-only, never updated, enormous, read by analytical scans. At the §5.1 volume target held for the maximum retention, this reaches the order of billions of rows and a terabyte of raw text. Never SQLite.

7.1 Why the shape matters#

A row store writes each event's fields contiguously, then the next event. A query counting 404s over a month reads every row whole, including the user-agents and paths it does not need. A columnar store writes all values of one column together, so the same query reads two columns out of thirty.

The second effect matters more: when neighbouring values on disk are all of one kind and resemble each other — consecutive timestamps, a few dozen distinct status codes, a few thousand repeated addresses — compression becomes large. On log-shaped data 10–30× is ordinary, against ~1× for a row store plus the weight of its indexes, which at this volume outweigh the data.

The price of columnar is no transactions, no foreign keys, no real UPDATE/DELETE, and weaker single-row lookups. That is the exact profile of events, and the exact opposite of what configuration and sanctions need — hence separate engines rather than one compromise bad at both.

7.2 Candidates#

A. PostgreSQL only, partitioned by time with BRIN indexes. One engine, one backup, already known. Ingestion is fine; the cost is on reads. With the indexes needed to pivot on address, range and domain, on-disk size runs to roughly twice the raw text and year-wide scans take tens of minutes. A replay becomes a batch job rather than an interactive action, which weakens §5.3 considerably.

B. PostgreSQL for state + ClickHouse for events. Columnar SQL server, self-hosted, Apache 2.0, installed from its own APT repository rather than from Debian — a packaging and release-cadence matter, not a maturity signal, and the same reason PostgreSQL is commonly installed from PGDG. The same dataset lands an order of magnitude smaller than option A, with year-wide scans in seconds. Cost: a second daemon to install, tune, back up and monitor.

C. PostgreSQL for state + embedded columnar (DuckDB over Parquet). Columnar in-process: no daemon, no third-party repository, operationally as light as a library, reading standard Parquet files we lay out. Cost: cgo, a single-writer design, and far less proven under sustained concurrent server load.

A fourth option — PostgreSQL plus a self-written Parquet cold tier — is rejected: it means writing the roll-off, the file layout and the query execution (filters, aggregations, ordering), and the last never stops costing and competes directly with building the product. Writing an actual columnar engine from scratch is out of scope for the same reason: a larger project than Shieldlist itself.

7.3 How this gets decided#

By benchmark, not argument. Generate a realistic corpus — on the order of 100 million lines shaped like the real HTTP and mail logs — load it into A, B and C, and measure what the product actually does: replaying a rule over six months, pivoting on one offender, aggregating per domain over a year, sustained ingest throughput, on-disk size — and a flood day: a botnet of hundreds of thousands of distinct addresses, cycling most of them daily, holding tens of thousands of requests a minute against one machine, with ingest, evaluation, sanction throughput and the UI's aggregate queries measured while it runs. Decide on those numbers, on this hardware.

The event store sits behind an interface in every case, so the choice is reversible at bounded cost and a small deployment can fall back to A.

8. Data model#

Defined in the config files (§5.9):

• owner — reseller or client, parent link, granted powers, users.
• resource — an owned domain, machine or address range, linked to its owner. One kind of object for the three, because scope resolution treats them uniformly.
• agent — declaration: hostname (the agent's identity), log inputs, the level it runs at and its per-rule overrides (§5.9), propagation switches, who may lift its sanctions, who reports them (§5.4, §5.5).
• exemption source — a published list to exempt (Googlebot, Bingbot, Cloudflare, …): shipped by name or declared by the operator with its address and format, refresh schedule; its output is a generated exemptions file (§5.9). Only that file ever reaches an agent.
• parser — log category, prefilter, pattern, extracted fields, version.
• rule — definition, version, placement (agent/controller), propagation, report policy, state (on/test/off).
• policy — ladder or progression, duration start (evidence or application), decay, port scope per rung.
• exemption — scope, value, conditions. Operator-authored; owner-authored exemptions are operational state (§5.9).
• trust — origin (country, ASN or range), threshold multiplier, sanction bounds.

Operational state:

• agent runtime — credential, last seen and from where, enforcement state, running versions (its own and the controller's it last spoke to), the configuration revision it holds, measured per-rule cost.
• rule runtime — measured cost, replay results.
• revision — a loaded configuration: hash, source (file deploy or UI edit), author, diff.
• offender — scope + value, first and last seen, geo/ASN, counters, current and historical sanctions.
• trigger — a rule firing: rule, offender, evidence reference, severity, whether it was enforced.
• sanction — scope, value, type, ports, duration, origin (rule, evidence machines), state (active/expired/revoked), escalation counter.
• report — provider, offender, categories, evidence, lifecycle, response.
• amnesty_request — offender, scope, submitted evidence, policy applied, state (pending → granted | refused | expired), resolver.
• list / list_entry — name, visibility, source, per-entry provenance.
• audit — who (human or system), action, target, timestamp, justification.

Events:

• event — agent, log input, target domain, local address, ingest and event timestamps, raw line, offender, parsed fields. The raw line is retained so parsing can always be redone; domain, machine and address are what make ownership scoping enforceable in the query layer.

Hits and in-window rule state are not stored: they are recomputed from events, on the live path and in a replay alike. A trigger references its evidence events rather than copying them.

9. Security requirements#

• Untrusted inputs are: log lines, imported lists, federated feeds, and amnesty requests. All are parsed defensively, bounded in size, and never executed or interpolated into queries or commands.
• Rules are data and can never become code (§5.2). This is what makes pushing them to agents acceptable.
• Own infrastructure is permanently exempt from sanction.
• Owner isolation (§5.6) is part of the threat model: opening the platform to customers turns a display concern into an access-control boundary. Cross-owner leakage is a security defect.
• An agent is authenticated, not trusted. It runs on a machine the operator may not control, so its evidence is data, not authority: every sanction records its provenance — the rule and the machines whose evidence produced it — and one action invalidates, after the fact, every decision built on a given agent's evidence over a chosen time span. Agents are identified by hostname; identity is the hostname, authority is the credential.
• The agent runs least-privilege: the capabilities that nftables and log reading require, never a full root shell service. The shipped systemd unit is hardened and the binary must operate under it.
• Discovered log inputs are validated. The agent never follows a target that resolves outside the allowed log roots — on shared hosting, a customer's symlink must not turn the agent into a file exfiltrator — and identity is re-verified on every rotation.
• Personal data is a data class, not an afterthought. Addresses are personal data. Retention is configurable per class — events, sanctions, audit — with a floor for legal retention duties and a ceiling for data minimisation, and expiry past the ceiling deletes. Publishing a list and sending a report are publications of personal data: explicit operator decisions, never defaults.
• Secrets (provider keys, inter-component credentials, release signing keys) live outside the config repository and rotate without downtime.
• The public frontend exposes only what is explicitly published and cannot reach the store directly.
• No feature may introduce a synchronous dependency on attacker-controlled data.

10. Delivery phases#

1. The autonomous agent — collection (tailing, rotation-proofing, discovery), parsers, the full rule language and evaluator, sanctions, policies, exemptions, trust, enforcement, the kernel guardrails, test, the alert outcome, config from local files, and a bounded local history so a replay works on the recent past. A complete single-machine product — the controller attaches to it later, not the reverse.
2. Control plane — enrolment and transport, versions both ways, configuration assembly (defaults, levels, overrides) and distribution, sanctions and enforcement state fed back by agents, fleet-wide lookup, ban, unban and exempt through the API and the CLI, published exemption sources, operator accounts and tokens, and the operations face of the web interface: a fleet manageable from one place before any raw log is centralised. Fleet propagation of sanctions and its switches land here too, since the state they need is already flowing.
3. Collect and retain — ingest, deduplication, enrichment, ownership attribution, the event store (after §7.3's benchmark), retention, fleet-wide search, CLI at parity. The ownership model lands here, in the schema, even though the interfaces that exploit it come later — adding it afterwards is what is expensive.
4. Fleet detection — controller-placed rules, replay over the fleet history, observed-versus-enforced comparison.
5. Report — the placeable sender on the controller, allowance shared across a fleet, submission after the fact; a standalone agent already carries it for its own sanctions (phase 1).
6. Interface — web UI: live view, investigation, bulk actions over history.
7. Lists and public surface — own lists and the rules that watch them, public frontend, feeds, sharing, guarded ingestion, amnesty workflow, scoped owner access.
8. Machine watch — probes on hosted sites, system metrics, shipped alert rule sets, the outage↔logs cross-reference (§5.11).
9. Smart detection — baselines, pattern mining, DDoS correlation, machine-assisted analysis.
10. Scale-out — component separation, sharding, federation.

Phase 1 alone is a usable product on one machine. Phases 1–6 are the complete, self-sufficient fleet product; 7–10 are what distinguish it. The control plane comes before ingestion because the store decision (§11) gates ingestion and nothing gates the control plane. The three-way parity of §1 is judged on the finished product: config and CLI reach each feature when it ships, and the UI catches up in phase 6.

11. Open decisions#

1. Event store engine — §7. The first releases are SQLite throughout (decided 2026-08-15): operational state and events alike, one embedded engine, no daemon, schema versioned and migrated in place by the controller. The scale-out engine is decided by benchmark (§7.3), which also answers where operational state belongs at that scale; the store sits behind an interface, so the move is a migration, not a rewrite.
2. Release signing — §6.1 fixes the requirement (a verification key that is not the controller's) but not the mechanism: key format, distribution and rotation. To be settled when distribution is actually on the table.
3. An XDP fast-drop stage. Enforcement is nftables, in the kernel, and that is not what this decision is about: a drop in prerouting is a kernel drop, with native per-element timeouts, ranges, atomic updates and sets that survive the agent. What XDP — an eBPF program at the driver — adds is a drop before the netfilter stack, an order of magnitude cheaper per packet, which matters on a flood measured in packets per second that loads the CPU without filling the uplink (the layer between "nftables suffices" and "defeated upstream", §5.4). Considered and set aside for the first releases: it would reimplement what nftables gives for free (expiry, ranges, persistence, atomicity) as BPF maps and timers, and native XDP depends on the driver, the kernel, BTF and capabilities — where the product must install on any VPS or shared host, nftables is everywhere. If it comes, it is an optional stage behind a capability probe, fed by the same sanction store, never a prerequisite, and it is measured first (§7.3's flood day). Observation through eBPF (sockets, SNI, HTTP by uprobes) belongs to the request-level question below, not to this one.
4. Request-level inspection (WAF). Whether the platform should replace the web server's WAF — examining and blocking single requests, per vhost, bodies included — rather than read the logs after the fact. Every vehicle (an engine embedded in the web server, a proxy tier) puts the platform in the request path, where principle 6 currently forbids it to fail. To be examined against the machinery it costs; until decided, nftables is the only enforcer and throttle and challenge stay out of scope.
5. Fleet firewall management. The agent already owns a table in every machine's nftables and the controller already delivers data to every agent; a static firewall policy — what each machine accepts and from where, by port and source, per machine or for the whole fleet — is the natural next thing to carry down that path. It would be written like everything else in the tree (a policy file, per-agent overrides, assembled and pushed as data, validated before it leaves), and applied by the agent into its own table beside the ban sets, never by rewriting the host's existing firewall. Recorded 2026-08-16 as a direction, not a commitment. Whatever form it takes, two constraints are settled now: the agent keeps the machine reachable by its operator whatever the policy says (management access and the controller's own stream are never closed by a pushed rule, and a policy that cuts the stream is rolled back, not kept), and a mistake at the controller reaches one machine before it reaches the fleet — a staged application that §6 does not impose on configuration today and would have to for this.