Designing a Structured JSON Audit Log Schema for the FOIA Request Lifecycle #

Within Audit Logging Architecture, the event schema is the contract that decides whether a records request’s history is queryable and defensible years later or merely a pile of strings nobody can reconcile. This page specifies that schema field by field — what each field means, why it exists, and how a validator rejects any event that would poison the trail. A hash-chained store proves the log was not altered; a disciplined schema proves the log actually answers the questions an oversight body will ask.

Scenario & Compliance Stakes #

A state agency’s public records portal has been running for two years. An appeal lands: a requester alleges the agency withheld a document it should have released and blew the statutory deadline doing it. The agency’s defense lives entirely in its audit log. If every lifecycle event — receipt, classification, the exemption decision, the deadline computation, delivery — was recorded in one consistent JSON shape with a stable identifier, the analyst reconstructs the timeline in a single query and the agency demonstrates exactly what it decided and when. If instead each subsystem logged its own ad-hoc shape, with timestamps in local time and free-text action names, the analyst spends days stitching partial records together and the agency cannot prove its timeline at all.

The stakes are precisely the failure classes the parent architecture names. A schema that omits a stable event identifier makes deduplication impossible, so a retried delivery looks like two disclosures. A schema that records local timestamps introduces drift that corrupts the very timeline the log exists to establish. A schema with an unbounded free-form payload invites megabyte events that copy sensitive content into the log and blow out storage. Getting the field set right is therefore not stylistic — it is what makes the audit trail admissible. The vocabulary of event types and lifecycle states here is drawn from the same classification model as the FOIA Request Taxonomy Design, so an event’s event_type and a record’s classification tier speak the same language.

The discipline that separates a queryable schema from a useless one is captured by NIST SP 800-53 AU-3, which enumerates the content an audit record must carry: what type of event occurred, when, where, the source, the outcome, and the identity of any individuals associated with the event. A FOIA event schema is essentially AU-3 rendered into a records-request domain — event_type is the “what,” timestamp is the “when,” request_id and correlation_id locate the “where” within a specific request and operation, actor is the “who,” and statutory_citation plus payload capture the outcome and its legal basis. Designing the schema is therefore not an exercise in inventing fields but in mapping a compliance control onto a concrete record shape, and every field below traces to a clause of that control. A field with no such justification is noise that bloats storage; a missing field is a question the log cannot answer under review.

There is also a forward-compatibility dimension. Statutes change, new event types appear as the pipeline grows, and the schema must absorb both without breaking the years of history already written. The solution is additive evolution: new optional fields and new enum members may be added, but existing field meanings are never repurposed and existing enum values are never deleted. A verifier reading a two-year-old event must interpret it exactly as it was written, so the schema is versioned and every change is backward-compatible by construction — the same immutability principle that governs the individual records governs the shape they take.

Prerequisites #

• Python 3.11+ — for datetime.UTC, StrEnum, and dataclass(slots=True). The module below uses only the standard library.
• A shared event-type enum — the controlled vocabulary of lifecycle transitions, owned centrally so no subsystem invents its own action names.
• A UTC clock discipline — every host stamping events must record time in UTC; the validator rejects anything else.
• An append-only sink — the schema is designed to be serialized one object per line into the hash-chained store described in the Audit Logging Architecture; this page defines the record shape, not the storage.
• A statutory citation registry — a lookup mapping each decision type to the authority it is taken under, so statutory_citation is a controlled value and not a typo-prone string.

Implementation #

The module defines the event as a single focused schema and a validator that enforces every rule the trail depends on. Each field earns its place. event_id is a stable UUID assigned once at creation so a replayed or retried event carries the same identity and deduplicates cleanly. event_type is drawn from a closed enum so queries can filter on it exactly. request_id ties the event to its records request; correlation_id threads a multi-hop operation across services. actor names the authenticated principal — never a bare display name — so the event is non-repudiable. timestamp is UTC ISO-8601, always. prev_hash carries the chain linkage. statutory_citation records the authority the decision was taken under. payload holds a bounded map of decisions, never content.

"""
audit_schema.py
Field-by-field structured audit event schema for the FOIA request lifecycle.
Defines the record shape and a fail-closed validator; the hash-chained
append-only store consumes these events one JSON object per line.
"""
from __future__ import annotations

import json
import re
import uuid
from dataclasses import dataclass, field, asdict
from datetime import datetime, UTC
from enum import StrEnum
from typing import Any

GENESIS_HASH = "0" * 64
MAX_PAYLOAD_BYTES = 4096                       # bound events so logs stay lean
_HEX64 = re.compile(r"^[0-9a-f]{64}$")         # sha-256 hex digest shape
_CITATION = re.compile(r"^[0-9A-Za-z.§ ()-]{3,80}$")   # e.g. '5 U.S.C. 552(b)(6)'


class LifecycleEventType(StrEnum):
    """Closed vocabulary of auditable transitions (NIST SP 800-53 AU-2)."""
    REQUEST_RECEIVED = "request_received"
    REQUEST_PERFECTED = "request_perfected"
    REQUEST_CLASSIFIED = "request_classified"
    DEADLINE_COMPUTED = "deadline_computed"
    DEADLINE_TOLLED = "deadline_tolled"
    EXEMPTION_APPLIED = "exemption_applied"
    RECORD_REDACTED = "record_redacted"
    RECORD_DELIVERED = "record_delivered"
    REQUEST_DENIED = "request_denied"
    LEGAL_HOLD_APPLIED = "legal_hold_applied"


class AuditSchemaError(ValueError):
    """Raised when an event violates the schema. Never logged past the boundary."""


@dataclass(slots=True)
class LifecycleAuditEvent:
    """One structured audit event for a public records request lifecycle."""
    event_type: LifecycleEventType
    request_id: str
    actor: str
    statutory_citation: str
    payload: dict[str, Any] = field(default_factory=dict)
    correlation_id: str = ""
    event_id: str = field(default_factory=lambda: str(uuid.uuid4()))
    timestamp: str = field(default_factory=lambda: datetime.now(UTC).isoformat())
    prev_hash: str = GENESIS_HASH

    def to_json(self) -> str:
        """Canonical one-line serialization for the append-only sink.
        sort_keys keeps the digest stable regardless of field order."""
        return json.dumps(asdict(self) | {"event_type": self.event_type.value},
                          sort_keys=True, separators=(",", ":"))


def _require(condition: bool, message: str) -> None:
    if not condition:
        raise AuditSchemaError(message)


def validate_event(event: LifecycleAuditEvent) -> LifecycleAuditEvent:
    """Fail-closed validation of every field the audit trail depends on.

    A malformed event must be rejected at creation, never written, because a
    poisoned entry is far more expensive to remediate in an append-only,
    hash-chained store (NIST SP 800-53 AU-3: content of audit records).
    """
    # Stable identity: event_id and request_id must be present and well-formed.
    _require(isinstance(event.event_type, LifecycleEventType), "event_type must be from the enum")
    _require(bool(event.request_id) and len(event.request_id) <= 64, "request_id missing or oversized")
    try:
        uuid.UUID(event.event_id)  # reject non-UUID ids that break dedup
    except (ValueError, AttributeError) as exc:
        raise AuditSchemaError("event_id must be a UUID") from exc

    # Actor must be an attributable principal, not anonymous or blank (AU-10).
    _require(bool(event.actor) and event.actor.lower() != "anonymous", "actor must be attributable")

    # Timestamp must be timezone-aware UTC ISO-8601 — no local time, no drift.
    try:
        parsed = datetime.fromisoformat(event.timestamp)
    except ValueError as exc:
        raise AuditSchemaError("timestamp is not ISO-8601") from exc
    _require(parsed.tzinfo is not None and parsed.utcoffset() == UTC.utcoffset(None),
             "timestamp must be UTC and timezone-aware")

    # Statutory citation is a controlled, bounded value.
    _require(bool(_CITATION.match(event.statutory_citation)), "statutory_citation malformed")

    # Chain linkage must be a sha-256 hex digest.
    _require(bool(_HEX64.match(event.prev_hash)), "prev_hash must be a sha-256 hex digest")

    # Payload is bounded and must carry no obvious raw PII (defense in depth).
    encoded = json.dumps(event.payload, sort_keys=True).encode("utf-8")
    _require(len(encoded) <= MAX_PAYLOAD_BYTES, "payload exceeds size bound")
    _reject_raw_pii(event.payload)
    return event


_SSN = re.compile(r"\b\d{3}-\d{2}-\d{4}\b")


def _reject_raw_pii(payload: dict[str, Any]) -> None:
    """Reject payloads that inline recognizable PII. The audit log records
    decisions and references (which fields were redacted), never content."""
    for value in payload.values():
        if isinstance(value, str) and _SSN.search(value):
            raise AuditSchemaError("payload appears to contain raw PII; log references, not content")


if __name__ == "__main__":
    event = LifecycleAuditEvent(
        event_type=LifecycleEventType.EXEMPTION_APPLIED,
        request_id="REQ-2026-000481",
        actor="reviewer:jlee",
        statutory_citation="5 U.S.C. 552(b)(6)",
        correlation_id="c1f2e3d4-0000-4000-8000-000000000001",
        payload={"exemption": "b6-privacy", "redacted_fields": ["ssn", "dob"], "count": 2},
    )
    validate_event(event)
    print(event.to_json())

The validator is deliberately strict and fails closed: an event that cannot be validated is never written, because remediating a poisoned entry in an append-only, hash-chained store means annotating around it forever. Note what the payload example carries — redacted_fields: ["ssn", "dob"] names which fields were redacted and under which exemption, but the values themselves never appear. That discipline is what keeps the log from becoming a secondary disclosure surface, and it is enforced structurally by _reject_raw_pii rather than left to reviewer diligence.

Each validation rule maps to a specific failure it prevents. The uuid.UUID check on event_id guarantees the identifier is well-formed so downstream deduplication has a reliable key; a sequential integer or a timestamp-derived id would collide or drift and undermine at-least-once delivery. The actor check rejects blank or anonymous principals because a decision no one is attributed to is not non-repudiable — AU-10 requires that an action bind to an identity, and “anonymous redacted this record” is not a defense. The timezone assertion forces every timestamp to carry an explicit UTC offset, which is what lets events from a dozen hosts sort into one true order. The citation pattern keeps the legal basis a controlled value rather than a free-text field prone to typos that break later filtering. And the payload size bound plus PII screen are defense in depth: even a well-meaning author who tries to attach “just this once” the full text of a record is stopped at the boundary. The cumulative effect is that a valid event is, by construction, an admissible one.

Because the dataclass uses slots=True and the validator runs at construction time in practice, the cost of this rigor is negligible on the request path — a few regex matches and a JSON size check per event. That is a deliberate trade: the validation is cheap precisely so there is never an operational excuse to skip it, and skipping it is the one thing that would let a malformed event reach the immutable store.

Expected Output #

A single validated event serializes to one line. This is the shape a SIEM query filters and a chain verifier recomputes:

{"actor":"reviewer:jlee","correlation_id":"c1f2e3d4-0000-4000-8000-000000000001","event_id":"7b2e5c8a-1d4f-4a6b-9c30-2f8e11a0b7d1","event_type":"exemption_applied","payload":{"count":2,"exemption":"b6-privacy","redacted_fields":["ssn","dob"]},"prev_hash":"0000000000000000000000000000000000000000000000000000000000000000","request_id":"REQ-2026-000481","statutory_citation":"5 U.S.C. 552(b)(6)","timestamp":"2026-07-15T14:02:11.481920+00:00"}

Every field is present, the timestamp carries an explicit +00:00 offset, the citation is controlled, and the payload names decisions rather than content. Because keys are sorted, the line is byte-stable — two events built from the same inputs produce the identical string, which is what lets the downstream chain writer hash them reproducibly.

Byte-stability is easy to overlook and expensive to lose. If the serializer emitted keys in insertion order, then re-serializing the same event on a different code path — during verification, say, or after a round-trip through storage — could produce a different byte sequence and therefore a different hash, which a naive verifier would misread as tampering. Sorting keys removes that ambiguity entirely: the canonical form of an event is a pure function of its field values, independent of how the dict was built or which library version serialized it. That determinism is the quiet contract between this schema and the hash-chained store, and it is why sort_keys=True and a fixed separator tuple are not cosmetic choices but load-bearing ones. Any team that later swaps the encoder must preserve exactly this canonicalization, or every historical hash silently stops verifying.

Common Pitfalls #

• Unstable event IDs. Generating the event_id at write time instead of at event creation means a retried or replayed event gets a fresh id and looks like a distinct occurrence, so a single delivery appears twice in the trail. Assign the UUID once when the event object is constructed and carry it through every retry, so at-least-once delivery deduplicates on a stable key.
• Timezone drift. Recording local time, or naive timestamps with no offset, silently corrupts the timeline the log exists to establish — an event stamped at 11 p.m. Pacific sorts before one stamped at 1 a.m. Eastern that actually happened earlier. Stamp every event in UTC with a timezone-aware value and reject anything else at validation, as the module does.
• Unbounded payloads. Letting the payload grow without limit invites subsystems to dump entire records, request bodies, or stack traces into an event, bloating storage and copying content the log should never hold. Cap the serialized payload and keep it to a bounded map of decision fields; if a large artifact must be referenced, store its hash, not its bytes.
• PII in fields. The most damaging pitfall is treating the audit log as a convenient place to record what was in a document. The moment a raw SSN, address, or sealed detail lands in a payload, it propagates to the SIEM and every backup and becomes a disclosure the same Security Boundary Configuration that protects records must now also govern for the log. Record references and decisions; let structural validation reject content.

FAQ #

Why assign the event ID at creation instead of when the event is written? #

Because the identifier is what makes at-least-once delivery safe. A durable forwarder or a crash-recovery replay will re-emit an event, and if a fresh id is minted each time, the trail records two occurrences of one real event — which, for a delivery or a denial, is a materially false history. Assigning the UUID once when the event object is constructed, and carrying it through every retry, lets every downstream consumer deduplicate on a stable key so the same event collapses to one entry no matter how many times it is delivered.

Should the payload ever contain the content of the record being processed? #

No. The payload records decisions and references — which exemption applied, which fields were redacted, which deadline was computed, the hash of an input — never the sensitive content itself. Inlining record content turns the audit log into a second copy of the protected material that then spreads to the SIEM and every backup, expanding the disclosure surface the system exists to shrink. The validator enforces this structurally by bounding payload size and rejecting recognizable PII, so the guarantee does not depend on every author remembering it.

How does a fixed schema help years later under appeal? #

Consistency is what makes the trail queryable. When every subsystem emits the same field set with a controlled event_type and UTC timestamps, an analyst reconstructing a two-year-old timeline filters on request_id and reads the ordered story in one pass. A schema that drifts per service forces manual reconciliation of incompatible shapes and undermines the agency’s ability to prove what it decided and when — which is the entire point of keeping the log.

Related #

• Audit Logging Architecture — the parent design this schema feeds
• Forwarding FOIA Audit Logs to a SIEM with Append-Only Guarantees — durable delivery that deduplicates on the stable event id
• FOIA Request Taxonomy Design — the classification vocabulary the event types align to
• Security Boundary Configuration — the access separation that keeps sensitive content out of the log
• Core Architecture & Compliance Mapping — the parent domain this subsystem serves

← Back to all public records automation topics