Intake & Routing Workflows for Public Records & FOIA Automation
Government technology teams, records managers, and compliance officers must treat intake and routing as the control plane that decides whether an agency meets its statutory obligations under 5 U.S.C. § 552 and the parallel state open records acts. The first hours after a request arrives — capture, validation, classification, and assignment to a custodian — determine whether the agency starts its 20-business-day clock on time, preserves chain-of-custody, and produces a defensible audit trail if the matter ends up before an inspector general or a court. This page is the engineering reference for building that control plane in production: deterministic state transitions, idempotent ingestion, immutable logging, and explicit compliance hooks at every processing boundary.
The workflow described here sits upstream of every other capability on this site. It hands validated, classified, deadline-stamped requests to the downstream automation in Document Retrieval & Parsing and enforces the structural guarantees defined in Core Architecture & Compliance Mapping. Get intake wrong and no downstream redaction, retrieval, or disclosure step can be trusted.
Foundational Architecture & State Management
Intake & Routing Workflows must operate as a state-driven, idempotent pipeline. Each request enters as an untrusted payload, transitions through validation, enrichment, and classification layers, and exits into a durable assignment queue. A strict finite state machine prevents duplicate processing and enforces lifecycle boundaries: RECEIVED → VALIDATED → PRIORITIZED → ROUTED → ACKNOWLEDGED. Every transition is a single, logged, append-only event; there is no in-place mutation of request state without a corresponding audit record.
The core data model
A request is the atomic unit of the system, and its identity must be stable from the moment of receipt. Assign a server-generated request_id (a UUID, never a sequential integer that leaks volume) the instant a payload crosses the boundary, and compute an idempotency_hash over the normalized payload so the same submission arriving twice — a citizen who clicks “submit” three times, or an email retried by a misconfigured relay — resolves to a single tracked request. The model also carries the created_at timestamp in UTC, the calculated statutory_deadline, an assigned priority_tier, the resolved assigned_unit, and a boolean is_frozen flag that short-circuits every downstream transition.
Idempotency guarantees
Idempotency is not optional for statutory systems. A duplicate that slips through can double-start a deadline clock, route two copies to two custodians, and produce contradictory audit trails. The pipeline enforces idempotency at the validation boundary by hashing the canonical JSON form of the payload (keys sorted, whitespace normalized) and checking that hash against a durable set before admitting the request. To absorb submission spikes without losing or duplicating work, the validated payloads are handed to a broker rather than processed inline — configure Async Queue Management with persistent delivery and visibility timeouts so that an unacknowledged worker crash re-delivers the task instead of dropping it. All state transitions must persist to a write-ahead log before downstream execution begins, establishing a recoverable baseline for compliance audits and non-repudiation.
Statutory & Regulatory Context
Intake is where law becomes code. Federal FOIA (5 U.S.C. § 552(a)(6)(A)(i)) mandates a determination within 20 business days of receipt, with a permissible 10-day extension for “unusual circumstances” under § 552(a)(6)(B), and an expedited-processing track under § 552(a)(6)(E) for requests demonstrating a compelling need. The clock starts on receipt by the proper component, not on the day an engineer happens to process the queue — which is exactly why deadline calculation must happen at ingestion, deterministically, and be recorded immutably.
Deadline windows and tolling
State statutes diverge sharply and must be modeled per jurisdiction rather than approximated with a single constant. The State Law Compliance Frameworks reference encodes these differences: the California Public Records Act expects a determination within 10 calendar days (Gov. Code § 7922.535), the Texas Public Information Act runs on 10 business days with an Attorney General referral track, and New York FOIL requires acknowledgment within 5 business days. Tolling complicates every one of these — a request that requires payment of fees, clarification from the requester, or consultation with another agency may pause the clock under defined conditions. Encode tolling as explicit, logged events that move time, never as a silent recalculation.
Freeze states and preservation obligations
Retention schedules dictate how long intake artifacts, routing metadata, and decision logs remain in active storage before archival or disposition; those rules live in Records Retention Scheduling and must be wired into the intake record at creation so disposition is never guessed later. When litigation holds, active investigations, or executive directives require preservation, compliance officers must be able to instantly halt routing and retention clocks. The FROZEN state exists for exactly this: it must propagate atomically across all services, block every downstream mutation, and remain in place until an authorized compliance officer explicitly lifts the hold with a logged, attributable action. A freeze that depends on every worker independently noticing a flag is a freeze that will leak; enforce it at the state-transition gate so no path bypasses it.
Secure Ingestion & Classification Boundaries
Public records intake surfaces some of the most sensitive data an agency holds: PII, PHI, law-enforcement records, sealed juvenile matters, and privileged communications. The threat model must assume the ingestion endpoint is hostile by default — that submissions carry malformed encodings, oversized attachments, embedded executables, and payloads crafted to exhaust parsers or trigger ReDoS. Security boundaries enforce least-privilege access, data-classification tags, and cryptographic isolation at every routing hop, and the controls themselves are defined centrally in Security Boundary Configuration so intake inherits rather than reinvents them.
NIST SP 800-53 control alignment
Map each ingestion control to a named NIST SP 800-53 control family so audits trace cleanly. Access enforcement and least privilege align to the AC family (AC-3, AC-6); immutable audit events and non-repudiation align to AU (AU-2, AU-9, AU-12); input validation and information-handling boundaries align to SI (SI-10) and SC (SC-7) for network segmentation between ingestion endpoints and processing workers. Network segmentation prevents lateral movement and contains blast radius during a security incident: a compromised parser must not be able to reach the custodial assignment store or the audit sink directly.
Deterministic parsing before classification
Unstructured submissions require deterministic parsing before any classification decision. The Email & Form Parsing Pipelines extract requester metadata, attachment manifests, and statutory-language markers while stripping executable payloads and normalizing character encodings; scanned attachments are handed to the OCR Processing Pipelines so image-only PDFs become searchable, classifiable text rather than opaque blobs. Classification engines then apply sensitivity labels aligned with the categories in FOIA Request Taxonomy Design, ensuring downstream handlers only access data commensurate with their clearance and role-based access policies. A request that cannot be parsed deterministically is routed to a quarantine queue for human review, never guessed at.
Production-Grade Python Implementation
The following reference implementation demonstrates a secure, runnable intake router with structured JSON logging, deterministic state transitions, statutory deadline calculation, and idempotency enforcement. It adheres to audit-ready standards and avoids external dependencies for portability. In production you would swap the in-memory processed_hashes set for a durable store and feed transitions through the broker described in Async Queue Management, but the state-machine guarantees and compliance hooks remain identical.
"""
production_intake_router.py
Deterministic FOIA/Public Records Intake & Routing Workflow
Compliance-aligned, audit-ready, and idempotent.
"""
import json
import uuid
import hashlib
import logging
import sys
from datetime import datetime, timedelta, timezone
from enum import Enum
from dataclasses import dataclass, field
from typing import Optional, Dict, Any
# ---------------------------------------------------------------------------
# Structured Audit Logging Configuration
# AU-2 / AU-12: every state transition emits a machine-parseable audit event.
# ---------------------------------------------------------------------------
class JSONFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
log_entry = {
"timestamp": self.formatTime(record, self.datefmt),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
"request_id": getattr(record, "request_id", None),
"state_transition": getattr(record, "state_transition", None),
"compliance_flag": getattr(record, "compliance_flag", None),
}
return json.dumps(log_entry)
audit_logger = logging.getLogger("foia_intake_audit")
audit_logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JSONFormatter())
audit_logger.addHandler(handler)
# ---------------------------------------------------------------------------
# Domain Models & State Machine
# ---------------------------------------------------------------------------
class RequestState(str, Enum):
RECEIVED = "RECEIVED"
VALIDATED = "VALIDATED"
PRIORITIZED = "PRIORITIZED"
ROUTED = "ROUTED"
ACKNOWLEDGED = "ACKNOWLEDGED"
FROZEN = "FROZEN"
@dataclass
class IntakeRequest:
request_id: str
raw_payload: Dict[str, Any]
state: RequestState = RequestState.RECEIVED
created_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
statutory_deadline: Optional[datetime] = None
priority_tier: Optional[int] = None
assigned_unit: Optional[str] = None
idempotency_hash: Optional[str] = None
is_frozen: bool = False
def compute_idempotency_hash(self) -> str:
"""Deterministic hash over canonical payload for duplicate detection."""
payload_str = json.dumps(self.raw_payload, sort_keys=True)
return hashlib.sha256(payload_str.encode()).hexdigest()
# ---------------------------------------------------------------------------
# Compliance & Routing Engine
# ---------------------------------------------------------------------------
class IntakeRoutingEngine:
def __init__(self, jurisdiction_days: int = 20):
self.jurisdiction_days = jurisdiction_days
self.processed_hashes: set = set()
def _guard_frozen(self, req: IntakeRequest) -> None:
# Litigation-hold gate: no transition may proceed while frozen.
if req.is_frozen or req.state == RequestState.FROZEN:
audit_logger.warning(
"Transition blocked by active litigation hold.",
extra={"request_id": req.request_id, "compliance_flag": "FREEZE_BLOCK"},
)
raise PermissionError("Request is frozen under a preservation hold")
def validate_and_transition(self, req: IntakeRequest) -> IntakeRequest:
self._guard_frozen(req)
if req.state != RequestState.RECEIVED:
raise ValueError(f"Invalid state transition from {req.state}")
req.idempotency_hash = req.compute_idempotency_hash()
if req.idempotency_hash in self.processed_hashes:
audit_logger.warning(
"Duplicate request detected. Aborting.",
extra={"request_id": req.request_id, "compliance_flag": "IDEMPOTENCY_BLOCK"},
)
raise ValueError("Duplicate request payload")
# 5 U.S.C. § 552(a)(6)(A)(i): determination due within 20 business days
# of receipt. Calculate at ingestion so the clock cannot start late.
req.statutory_deadline = self._calculate_deadline(req.created_at)
req.state = RequestState.VALIDATED
self.processed_hashes.add(req.idempotency_hash)
self._log_transition(req, "VALIDATED", "payload_integrity_verified")
return req
def prioritize(self, req: IntakeRequest) -> IntakeRequest:
self._guard_frozen(req)
if req.state != RequestState.VALIDATED:
raise ValueError("Request must be validated before prioritization")
# Expedited-processing signal per § 552(a)(6)(E); production scoring is
# delegated to the Priority Scoring Algorithms sub-system.
complexity = len(req.raw_payload.get("attachments", []))
req.priority_tier = 3 if complexity > 5 else 1
req.state = RequestState.PRIORITIZED
self._log_transition(req, "PRIORITIZED", f"tier_{req.priority_tier}")
return req
def route(self, req: IntakeRequest) -> IntakeRequest:
self._guard_frozen(req)
if req.state != RequestState.PRIORITIZED:
raise ValueError("Request must be prioritized before routing")
# Deterministic custodial resolution; production keyword/series maps live
# in the Department Routing Logic sub-system.
subject = req.raw_payload.get("subject", "").lower()
if "finance" in subject or "budget" in subject:
req.assigned_unit = "FINANCE_RECORDS"
elif "personnel" in subject or "hr" in subject:
req.assigned_unit = "HR_COMPLIANCE"
else:
req.assigned_unit = "GENERAL_CUSTODIAN"
req.state = RequestState.ROUTED
self._log_transition(req, "ROUTED", f"unit_{req.assigned_unit}")
return req
def acknowledge(self, req: IntakeRequest) -> IntakeRequest:
self._guard_frozen(req)
if req.state != RequestState.ROUTED:
raise ValueError("Request must be routed before acknowledgment")
req.state = RequestState.ACKNOWLEDGED
self._log_transition(req, "ACKNOWLEDGED", "statutory_clock_started")
return req
def _calculate_deadline(self, start: datetime) -> datetime:
"""Approximates the business-day statutory window. Production deployments
must apply a federal/state holiday calendar and per-jurisdiction tolling."""
return start + timedelta(days=self.jurisdiction_days)
def _log_transition(self, req: IntakeRequest, new_state: str, detail: str):
audit_logger.info(
f"State transition: {req.state.value} -> {new_state} | {detail}",
extra={
"request_id": req.request_id,
"state_transition": f"{req.state.value}->{new_state}",
"compliance_flag": detail,
},
)
# ---------------------------------------------------------------------------
# Execution Flow
# ---------------------------------------------------------------------------
def process_intake_request(raw_payload: Dict[str, Any]) -> Dict[str, Any]:
engine = IntakeRoutingEngine(jurisdiction_days=20)
req = IntakeRequest(request_id=str(uuid.uuid4()), raw_payload=raw_payload)
try:
req = engine.validate_and_transition(req)
req = engine.prioritize(req)
req = engine.route(req)
req = engine.acknowledge(req)
return {
"status": "success",
"request_id": req.request_id,
"final_state": req.state.value,
"assigned_unit": req.assigned_unit,
"statutory_deadline": req.statutory_deadline.isoformat(),
"priority_tier": req.priority_tier,
}
except (ValueError, PermissionError) as exc:
# Failures are logged to the audit sink, never silently swallowed.
audit_logger.error(
f"Intake processing halted: {exc}",
extra={"request_id": req.request_id, "compliance_flag": "PROCESSING_FAILURE"},
)
raise
if __name__ == "__main__":
sample_payload = {
"requestor": "citizen@example.gov",
"subject": "Personnel records for FY2025",
"attachments": ["doc1.pdf", "doc2.pdf"],
"jurisdiction": "federal",
}
result = process_intake_request(sample_payload)
print(json.dumps(result, indent=2))
Running the module emits one structured audit line per transition and a final summary object. Because every transition checks the preceding state and the freeze gate, an out-of-order call or a held request raises immediately instead of corrupting the lifecycle — the property that makes the workflow defensible under review.
Operational Resilience & Failure Modes
Production intake systems must anticipate partial failures, network partitions, and malformed submissions without ever silently dropping a statutory request. Transient broker failures and validation timeouts are retried with exponential backoff and jitter so a thundering retry herd does not itself become an outage, and so retries never accidentally re-start a deadline clock. Irrecoverable payloads — the truly malformed, the oversized, the un-parseable scan — are diverted to a dead-letter queue that captures the original bytes, the failure reason, and a correlation ID for manual compliance review, isolating the fault domain while preserving the audit chain.
Audit continuity under partition
When the network partitions, the system must fail toward preservation, not toward loss. Buffer audit events locally with a durable, append-only journal and reconcile to the central sink when connectivity returns; never advance a state transition whose audit write has not been durably committed. All routing decisions, deadline calculations, and state mutations are cryptographically hashed (and, ideally, signed) so tampering is detectable, and the immutable stream is forwarded to a centralized SIEM or compliance data lake where automated monitors track SLA adherence, escalation triggers, and freeze-state propagation. The chain-of-custody and append-only store patterns these monitors depend on are specified in Core Architecture & Compliance Mapping. By treating intake and routing as a deterministic, auditable control plane, agencies eliminate manual bottlenecks, guarantee statutory compliance, and establish a defensible posture for public records transparency.
Compliance Verification Checklist
Use this checklist to confirm an intake deployment meets the statutory and security controls introduced above before it handles live requests.
FAQ
When does the statutory clock actually start, and how should the system record it?
Under federal FOIA the determination clock starts on receipt by the proper agency component, not when an engineer dequeues the request. The system must therefore stamp created_at and compute statutory_deadline at the ingestion boundary and write both to the immutable log in the same transition that admits the request. Recording the deadline later — or recomputing it during a retry — risks understating the agency’s true obligation and is difficult to defend on review.
How do we guarantee a duplicate submission does not start two deadline clocks?
Compute a deterministic idempotency_hash over the canonical (sorted-key, normalized) form of the payload and check it against a durable store before admitting the request. A matching hash is logged with an IDEMPOTENCY_BLOCK flag and rejected. Pair this with persistent broker acknowledgments in Async Queue Management so a worker crash re-delivers rather than re-creates the request.
What is the safest way to implement a litigation hold across a distributed intake system?
Enforce the hold at the state-transition gate, not as an advisory flag each worker is trusted to honor. A request in the FROZEN state must reject every downstream transition until an authorized compliance officer lifts it with a logged, attributable action. Because the gate is the single chokepoint every path crosses, no consumer can accidentally route, prioritize, or dispose of a held record.
How should the workflow handle a request it cannot parse or classify?
Route it to a quarantine or dead-letter queue with the original bytes and a correlation ID, and emit a PROCESSING_FAILURE audit event. Never infer a custodian or sensitivity label from an un-parseable submission — a wrong guess can leak exempt material or miss a deadline. Human review resolves the request and feeds corrections back into the parsing and classification rules.
Why model deadlines per jurisdiction instead of using a single constant?
State open records acts diverge on both duration and clock type: California runs on 10 calendar days, Texas on 10 business days with an Attorney General track, and New York FOIL requires a 5-business-day acknowledgment. A single constant will silently breach at least one jurisdiction. Encode each window, its business-day vs. calendar-day basis, and its tolling rules in the State Law Compliance Frameworks reference and resolve them at ingestion.
Related
- Async Queue Management — persistent brokering and visibility timeouts that absorb ingestion spikes without losing or duplicating statutory requests.
- Priority Scoring Algorithms — weighting request complexity, media type, and expedited-processing signals into a defensible priority tier.
- Department Routing Logic — mapping subject keywords and record-series identifiers to the correct custodial unit.
- Email & Form Parsing Pipelines — deterministic extraction of requester metadata and attachments from untrusted submissions.
- Core Architecture & Compliance Mapping — the chain-of-custody, retention, and security-boundary foundations intake builds on.
- Document Retrieval & Parsing — the downstream pipeline that receives validated, classified, deadline-stamped requests.