Forwarding FOIA Audit Logs to a SIEM with Append-Only Guarantees

Within Audit Logging Architecture, forwarding is the hop where a tamper-evident local chain becomes a monitored, independently held record — and the hop where a careless implementation quietly destroys the guarantees the chain was built to provide. This page shows how to move FOIA audit events from the local append-only store to a SIEM or data lake without losing an event, duplicating one, or breaking the hash chain, so the second copy strengthens non-repudiation rather than undermining it.

Scenario & Stakes

A records automation platform writes every lifecycle event to a local hash-chained store and forwards each one to the agency’s central SIEM, where security and compliance teams monitor deadline breaches, unusual redaction patterns, and disposition of held records. Two properties make this forwarding load-bearing. First, audit continuity under partition: when the network between the platform and the SIEM drops — a maintenance window, a broker outage, a cross-region blip — events must not be lost, because a gap in the SIEM copy is indistinguishable from an attacker deleting evidence. Second, non-repudiation to an independent store: the value of shipping to the SIEM is that it is administered separately, so an operator who could rewrite the local store cannot silently rewrite the SIEM’s copy too; the two chains become mutual witnesses.

The failure modes are specific and unforgiving. A forwarder that drops events on a crash leaves a hole. A forwarder that retries naively delivers the same event twice, and if the receiver treats each delivery as a new link, the chain forks. A forwarder that reorders events under concurrency presents a history that never happened. Each of these converts a defensible audit trail into a liability, which is why the design leans on the same idempotent event_id and hash-chain linkage defined in the structured JSON audit log schema. The mechanics here echo the durability patterns behind managing high-volume intake with Celery task queues, but the correctness bar is higher: an intake queue may drop a duplicate, whereas an audit forwarder must preserve every event exactly once in effect while tolerating at-least-once in transit.

It helps to be precise about why exactly-once delivery is impossible and therefore not the goal. In any system where the network can fail, a sender that transmits an event and then loses the connection cannot know whether the receiver committed it before the failure or not. It has two choices: re-send, risking a duplicate, or stay silent, risking a loss. For an audit trail a loss is catastrophic and a duplicate is merely inconvenient, so the only safe choice is to re-send — which makes at-least-once the floor, not a compromise. The engineering work is then to make duplicates harmless, which is exactly what an idempotent event_id at the receiver accomplishes. This inversion — accept duplicate transport to guarantee zero loss, then neutralize duplicates at commit — is the whole design in one sentence, and it is why the receiver, not the sender, is where deduplication belongs.

The independence of the SIEM copy is worth dwelling on because it is the property that makes forwarding worth the effort at all. A single hash-chained store proves that its own contents were not altered — but only to someone who trusts that store’s operators not to have rewritten the chain and its published head in one coordinated move. An insider with sufficient access to the primary store could, in principle, do exactly that. Shipping every event to a SIEM administered by a different team, under different credentials, in a different trust domain, means that erasing an inconvenient event undetectably now requires colluding across both boundaries simultaneously. The two chains become mutual witnesses, and the routine reconciliation of their heads is what turns “we believe the log is intact” into “we can demonstrate the two independent copies agree.” This is the practical meaning of NIST SP 800-53 AU-9’s requirement to protect audit information from the very administrators who operate the system.

Prerequisites

  • Python 3.11+ — for datetime.UTC and standard-library JSON, hashing, and SQLite journaling; no third-party runtime dependency is required for the core forwarder.
  • A durable local journal — a small append-only store (SQLite in WAL mode below, or an on-disk queue) that survives process restart, so an event is safe the instant it is written and before it is acknowledged by the SIEM.
  • A SIEM or collector with an idempotent ingest path — any endpoint that accepts newline-delimited JSON and can deduplicate on event_id, or a receiver you control that verifies the chain on ingest.
  • The event schema and chain fields — events already carry a stable event_id, prev_hash, and entry_hash from the Audit Logging Architecture writer.
  • Least-privilege transport credentials — the forwarder can read the journal and write to the SIEM, and nothing else; retention and disposition of the forwarded copy follow Records Retention Scheduling.

Implementation

The forwarder has three parts: a durable journal that persists each event before any delivery attempt, an at-least-once sender with idempotent delivery and exponential backoff, and a receiver-side chain verifier that rejects gaps and duplicates. The invariant is that an event is durable on disk the moment it is produced, so a crash between production and acknowledgement loses nothing — recovery simply re-sends every un-acknowledged event, and the idempotent event_id makes the re-send harmless.

At-least-once audit forwarding with receiver-side deduplication A left-to-right flow. A producer seals each event and writes it to a durable write-ahead journal before any network attempt. A forwarder delivers un-acknowledged events at least once with exponential backoff. The SIEM receiver deduplicates on the stable event id. A duplicate re-delivery, shown as a dashed retry arrow, and the normal delivery both reach the receiver's checks, which drop an already-seen event id as a no-op, reject any event whose prev_hash does not match the running head as a gap, and otherwise commit exactly one link. Transport is at-least-once but the committed effect is exactly once. Producer seals + chains event Durable journal WAL, append-only Forwarder at-least-once + backoff SIEM receiver dedup on event_id retry = duplicate Receiver checks event_id already seen → no-op prev_hash ≠ head → reject (gap) else commit exactly one link
The journal makes loss impossible; the receiver's dedup and chain check make at-least-once transport commit exactly once.
python 
"""
audit_forwarder.py
Durable, at-least-once forwarding of FOIA audit events to a SIEM with
append-only guarantees. Events survive crashes in a local journal, deliver
idempotently on a stable event_id, and are chain-verified on the receiver.
"""
from __future__ import annotations

import json
import sqlite3
import time
from datetime import datetime, UTC
from typing import Any, Callable, Iterable

GENESIS_HASH = "0" * 64


class DurableJournal:
    """Append-only local journal. An event is safe the instant it lands here,
    before any network attempt (NIST SP 800-53 AU-9: protect audit info; the
    journal is the crash-durable buffer that prevents audit loss)."""

    def __init__(self, path: str) -> None:
        self._conn = sqlite3.connect(path)
        self._conn.execute("PRAGMA journal_mode=WAL")  # durable across crashes
        self._conn.execute(
            """CREATE TABLE IF NOT EXISTS journal (
                   event_id   TEXT PRIMARY KEY,   -- idempotency key: no dup rows
                   seq        INTEGER,            -- append order for this request
                   body       TEXT NOT NULL,      -- serialized audit event
                   forwarded  INTEGER DEFAULT 0   -- 0 = pending, 1 = acked
               )"""
        )
        self._conn.commit()

    def record(self, event: dict[str, Any], seq: int) -> None:
        """Persist before forwarding. INSERT OR IGNORE keeps the write
        idempotent so a replayed event never creates a second journal row."""
        self._conn.execute(
            "INSERT OR IGNORE INTO journal(event_id, seq, body) VALUES (?, ?, ?)",
            (event["event_id"], seq, json.dumps(event, sort_keys=True, separators=(",", ":"))),
        )
        self._conn.commit()

    def pending(self) -> list[tuple[str, str]]:
        """Un-acknowledged events in append order — never reordered."""
        rows = self._conn.execute(
            "SELECT event_id, body FROM journal WHERE forwarded = 0 ORDER BY seq ASC"
        ).fetchall()
        return list(rows)

    def mark_forwarded(self, event_id: str) -> None:
        self._conn.execute("UPDATE journal SET forwarded = 1 WHERE event_id = ?", (event_id,))
        self._conn.commit()


def forward_pending(
    journal: DurableJournal,
    send: Callable[[str], bool],
    max_attempts: int = 6,
    base_delay: float = 0.5,
) -> int:
    """Deliver each pending event at-least-once with exponential backoff.

    Order is preserved (journal returns events by seq); delivery is idempotent
    because `send` transmits the stable event_id, so a retry after an
    ambiguous failure cannot fork the chain. Returns count acknowledged.
    """
    acked = 0
    for event_id, body in journal.pending():
        for attempt in range(max_attempts):
            try:
                if send(body):                 # receiver dedups on event_id
                    journal.mark_forwarded(event_id)
                    acked += 1
                    break
            except (OSError, TimeoutError):
                # Transient transport failure: back off and retry. The event
                # stays in the journal, so a crash here still loses nothing.
                pass
            time.sleep(base_delay * (2 ** attempt))  # 0.5s, 1s, 2s, ...
        else:
            # Exhausted retries: leave forwarded=0 so the next run resumes it.
            break
    return acked


def verify_on_receive(events: Iterable[dict[str, Any]],
                      known_head: str = GENESIS_HASH) -> tuple[bool, str]:
    """Receiver-side guard: reject duplicates and chain gaps on ingest.

    Deduplicates on event_id and confirms each event's prev_hash equals the
    running head, so an out-of-order or replayed delivery is caught before it
    is committed to the SIEM's append-only index.
    """
    seen: set[str] = set()
    head = known_head
    for event in events:
        eid = event["event_id"]
        if eid in seen:
            continue                           # idempotent: duplicate is a no-op
        if event["prev_hash"] != head:
            return (False, head)               # gap or reorder — alert, do not commit
        seen.add(eid)
        head = event["entry_hash"]
    return (True, head)


if __name__ == "__main__":
    journal = DurableJournal(":memory:")
    # Two chained events produced locally; entry_hash of the first is the
    # prev_hash of the second (chain linkage from the audit writer).
    e1 = {"event_id": "id-1", "prev_hash": GENESIS_HASH, "entry_hash": "aaaa",
          "event_type": "request_received", "ts": datetime.now(UTC).isoformat()}
    e2 = {"event_id": "id-2", "prev_hash": "aaaa", "entry_hash": "bbbb",
          "event_type": "record_delivered", "ts": datetime.now(UTC).isoformat()}
    journal.record(e1, seq=1)
    journal.record(e2, seq=2)
    journal.record(e1, seq=1)  # replayed produce: INSERT OR IGNORE => no dup

    delivered: list[dict[str, Any]] = []
    forward_pending(journal, send=lambda body: delivered.append(json.loads(body)) is None or True)
    ok, head = verify_on_receive(delivered)
    print(json.dumps({"acked": len(delivered), "chain_ok": ok, "head": head}))

The three pieces enforce exactly-once effect over at-least-once transport. The journal’s INSERT OR IGNORE on the event_id primary key means a replayed produce never doubles a row; forward_pending retries only un-acknowledged events in seq order, so ordering survives crashes; and verify_on_receive deduplicates and checks chain linkage so even if the network delivers an event twice, the SIEM commits it once and rejects anything that would leave a gap. A delivery that fails ambiguously — the SIEM received it but the acknowledgement was lost — is simply re-sent and harmlessly deduplicated.

Notice the ordering guarantees are layered, not incidental. The journal assigns a monotonic seq per request at record time and pending() returns rows strictly by that sequence, so the sender never presents events out of order for a given request. The receiver’s prev_hash check then provides a second, independent guarantee: even if some future refactor or a misconfigured parallel worker did present events out of order, the chain-linkage test would reject the mis-ordered event rather than commit a history that never happened. Defense in depth here is deliberate — the sender’s ordering is an optimization that keeps the common path clean, while the receiver’s verification is the correctness backstop that holds even when the optimization is violated. A system that relied on only one of the two would be a single bug away from a corrupt trail.

The backoff schedule deserves a word as well. Exponential backoff with a growing delay prevents a forwarder from hammering a struggling SIEM into a deeper outage, but the crucial property is that failure is safe to wait out: because every un-acknowledged event remains durably in the journal marked forwarded = 0, an exhausted retry loop simply leaves the work for the next scheduled run rather than dropping it. There is no timeout after which an event is abandoned. In production the loop is driven on an interval or triggered by new writes, and a dead_letter state is reserved only for events the SIEM actively rejects as malformed — never for events it merely failed to acknowledge, which are always retried indefinitely until they land.

Expected Output

Running the module forwards both events, absorbs the replayed produce without a duplicate, and verifies the chain intact:

json 
{"acked": 2, "chain_ok": true, "head": "bbbb"}

Two events acknowledged (not three, despite the replayed record call), chain_ok true because each prev_hash matched the running head, and the final head equal to the last event’s entry_hash. On the SIEM side, that head is what a reconciliation job compares against the local store’s head: equal heads prove the two independently administered copies agree, and any divergence is an alertable integrity event.

Common Pitfalls

  • Duplicate events on retry. Retrying after an ambiguous failure — the SIEM committed the event but the ack was lost — is unavoidable in an at-least-once system. The mistake is letting the receiver treat the re-delivery as a new link, which forks the chain. Deduplicate on the stable event_id at the receiver, as verify_on_receive does, so a re-send is a no-op rather than a second entry.
  • Buffer loss on crash. Holding events only in memory before forwarding means a crash between production and acknowledgement silently drops them, leaving a gap that looks identical to tampering. Persist every event to a crash-durable journal (WAL-mode SQLite or an on-disk queue) before the first delivery attempt, and only mark it forwarded after the SIEM acknowledges.
  • Chain gaps from partial forwarding. Marking an event forwarded before the acknowledgement arrives, or forwarding out of order under concurrency, produces a receiver-side gap where a prev_hash no longer matches the running head. Forward strictly in seq order, mark forwarded only on confirmed delivery, and let the receiver reject any event whose linkage does not match rather than committing it and papering over the hole.
  • Ordering under parallelism. Running multiple forwarders against one journal without coordination interleaves events so the receiver sees them out of chain order. Keep forwarding per request single-threaded, or partition the journal by request_id so each chain is delivered by exactly one worker in order.

FAQ

By separating transport semantics from commit semantics. The transport is at-least-once — the forwarder will re-send any event it did not get an acknowledgement for, so no event is ever lost. The commit is exactly-once in effect because the receiver deduplicates on the stable event_id and rejects any event whose prev_hash does not match its running head. A re-sent event therefore either matches an id it has already committed (a harmless no-op) or extends the chain cleanly; it can never create a second link for the same event.

What happens to audit continuity during a network partition?

Nothing is lost, because every event is durable in the local journal the instant it is produced, before any network attempt. During the partition the forwarder’s deliveries fail and back off, but the events wait in the journal marked un-acknowledged. When connectivity returns, forward_pending resumes from the oldest un-acknowledged event in order, and the receiver’s chain verification confirms no gap was introduced. The SIEM copy simply catches up, and a head-reconciliation job confirms the two stores agree once the backlog drains.

Why forward to a SIEM at all if the local store is already tamper-evident?

Because tamper-evidence and independence are different properties. The local hash chain proves the log was not altered if you can trust the copy you are verifying — but an operator with access to the local store could, in principle, rewrite the chain and the head together. Forwarding to a separately administered SIEM creates a second witness under a different administrative boundary, so altering history undetectably would require compromising both stores at once. The two heads reconciling against each other is what makes the guarantee resistant to an insider, which is precisely what NIST SP 800-53 AU-9 protection of audit information is aiming at.

← Back to all public records automation topics