Detecting Expedited-Processing Requests from Compelling-Need Signals

Within Priority Scoring Algorithms, one detection task carries outsized legal weight: recognizing, at intake, that a request may qualify for expedited processing before the ordinary queue swallows it. A statutorily substantiated expedite demand runs on a far tighter clock than the routine window, and an agency that fails to spot it hands the requester a clean procedural grievance. This page shows how to score compelling-need signals from bounded request text, hold the decision behind a human-review gate, and emit an audit trail that proves the determination was rule-based rather than arbitrary.

Problem Framing & Statutory Requirement

Federal FOIA obliges an agency to grant expedited processing when a requester demonstrates a compelling need, and to make that determination within ten calendar days of the request under 5 U.S.C. § 552(a)(6)(E). The statute defines compelling need narrowly: an imminent threat to the life or physical safety of an individual, or an urgency to inform the public about actual or alleged government activity where the requester is primarily engaged in disseminating information. Several agency regulations add a due-process category — where a delay would cause the loss of substantial legal rights. Those categories are the only signals a detector should reward; anything broader invites the two symmetric failures that follow.

The first failure is a missed detection. If the intake engine treats an expedite demand as a routine filing, the ten-day determination window elapses silently, and the agency cannot later claim it evaluated the request in good time. Worse, the omission is invisible until a requester challenges it, by which point the record shows no evaluation ever occurred — the hardest posture to defend before an oversight body. The second failure is over-detection. If the detector fires on the word “urgent” sitting in an email signature, on a legal disclaimer, or on any requester who simply asserts urgency without substantiation, the expedite lane floods, genuinely qualifying requests get starved, and the agency’s handling looks arbitrary — precisely the posture a reviewing body probes. A defensible detector therefore does two things at once: it extracts compelling-need signals from bounded, sanitized text so boilerplate cannot trick it, and it records exactly which weighted signals produced each recommendation, so the determination is reproducible. Critically, the detector never grants an expedite on its own; the statutory determination is a human act, and the engine only surfaces candidates for a records officer to confirm.

Prerequisites & Environment Setup

Python 3.11+ for dataclasses slots, datetime.UTC, and structured exception handling.
Standard library only for the core — re, html, hashlib, json, logging, datetime. Keeping the detector dependency-free shrinks the audit surface and lets it run inside any intake worker without supply-chain exposure.
A versioned signal-weight table loaded from configuration, never hardcoded, so a policy change to what counts as a compelling-need signal is a reviewable commit rather than a code edit. Deadline-specific categories that vary by jurisdiction belong with the rules in State Law Compliance Frameworks, not in this module.
Append-only audit access — the detector needs read access to the parsed request payload and write-only access to an audit sink, and nothing else. Enforce that boundary per Security Boundary Configuration so a compromised detector cannot mutate request bodies or grant an expedite.

The normalized request text and sender classification the detector reads are produced upstream during parsing; the detector consumes them, it does not re-derive them. Keeping that boundary clean matters for replay: if the detector recomputed the sender class from raw headers, a later change to that logic would silently alter historical scores, and the audit record would no longer reconstruct the decision that was actually made.

How the Detection Works

The detector is a stateless function: bounded request text and a sender classification enter, a compelling-need score plus a candidacy recommendation leave, and one audit line is emitted. When the score clears a configured threshold, the request is not expedited — it is routed to a human-review gate where a records officer makes the statutory call. Only the officer’s decision closes the loop into an expedite or a denial.

Step-by-Step Implementation

1. Extract bounded compelling-need signals

The two non-negotiable defenses at extraction are a hard length cap, so signatures and quoted reply chains never reach the matcher, and pre-compiled anchored patterns, so a hostile payload cannot trigger catastrophic backtracking. Each signal maps to a statutory category rather than a loose synonym, and the detector records the substantiating phrase it matched so a reviewer can see what the engine reacted to.

python

import re
import html
import hashlib
import logging
from dataclasses import dataclass, field

logger = logging.getLogger("intake.expedite.detect")

# Compile once at module load: pre-compiled, bounded patterns cannot recompile
# per call and cannot backtrack catastrophically on hostile request text.
SIGNAL_PATTERNS = {
    # 5 U.S.C. 552(a)(6)(E): imminent threat to life or physical safety.
    "imminent_threat": re.compile(
        r"\b(imminent\s+threat|life\s+or\s+safety|physical\s+safety|loss\s+of\s+life)\b",
        re.IGNORECASE),
    # Urgency to inform the public about government activity.
    "public_urgency": re.compile(
        r"\b(urgen\w+\s+to\s+inform|breaking|time[-\s]sensitive\s+public)\b",
        re.IGNORECASE),
    # Agency-recognized due-process category: loss of substantial legal rights.
    "due_process": re.compile(
        r"\b(loss\s+of\s+(?:substantial\s+)?(?:due[-\s]process|legal)\s+rights)\b",
        re.IGNORECASE),
}
MAX_EVALUATION_LENGTH = 1500  # bounds regex input; ignores footers and quoted history


@dataclass(frozen=True, slots=True)
class CompellingNeedSignals:
    correlation_id: str
    text_hash: str                       # SHA-256, stable across worker restarts
    matched_categories: list[str] = field(default_factory=list)
    substantiating_phrases: list[str] = field(default_factory=list)
    sender_is_disseminator: bool = False


def extract_signals(raw_text: str, sender_class: str,
                    correlation_id: str) -> CompellingNeedSignals:
    """Extract bounded compelling-need signals from normalized request text.

    Stateless and idempotent so the same input always yields the same hash and
    signals — a precondition for the reproducible record 5 U.S.C. 552(a)(6)(E)
    determinations must rest on.
    """
    try:
        # Truncate BEFORE unescaping so the matcher never reads signature boilerplate.
        cleaned = html.unescape(raw_text[:MAX_EVALUATION_LENGTH])
    except (TypeError, ValueError) as exc:
        logger.warning(json.dumps({"event": "decode_failed",
                                   "correlation_id": correlation_id, "err": str(exc)}))
        cleaned = ""

    matched, phrases = [], []
    for category, pattern in SIGNAL_PATTERNS.items():
        found = pattern.search(cleaned)
        if found:
            matched.append(category)
            phrases.append(found.group(0).lower())

    return CompellingNeedSignals(
        correlation_id=correlation_id,
        # SHA-256, not built-in hash(): hash() is per-process salted and NOT audit-safe.
        text_hash=hashlib.sha256(cleaned.encode("utf-8")).hexdigest(),
        matched_categories=matched,
        substantiating_phrases=phrases,
        # "primarily engaged in disseminating information" is a statutory element
        # of the public-urgency category; supplied by upstream sender classification.
        sender_is_disseminator=(sender_class in {"press", "advocacy", "scholarly"}),
    )

2. Score the signals against a versioned weight table

Scoring turns the matched categories into a bounded number and a candidacy recommendation. Weights load from configuration so a policy change is a reviewable event, the public-urgency category only counts when the sender is a genuine disseminator (a statutory element), and the score is clamped so no single request can monopolize the queue.

python

import os
import json

# Externalized weights — a change is a version-controlled, reviewable commit.
WEIGHT_IMMINENT_THREAT = int(os.getenv("WEIGHT_IMMINENT_THREAT", "60"))
WEIGHT_DUE_PROCESS     = int(os.getenv("WEIGHT_DUE_PROCESS", "35"))
WEIGHT_PUBLIC_URGENCY  = int(os.getenv("WEIGHT_PUBLIC_URGENCY", "30"))
CANDIDATE_THRESHOLD    = int(os.getenv("EXPEDITE_CANDIDATE_THRESHOLD", "50"))
WEIGHTS_VERSION        = os.getenv("EXPEDITE_WEIGHTS_VERSION", "expedite-2026-07")


def score_compelling_need(signals: CompellingNeedSignals) -> dict:
    """Return a bounded score and a candidacy recommendation (never a grant).

    The detector only recommends; the 5 U.S.C. 552(a)(6)(E) determination is a
    human act. Every weight applied is recorded so a reviewer can reconstruct why
    a request was surfaced as an expedite candidate.
    """
    score = 0
    applied: list[str] = []
    try:
        if "imminent_threat" in signals.matched_categories:
            score += WEIGHT_IMMINENT_THREAT
            applied.append("imminent_threat")
        if "due_process" in signals.matched_categories:
            score += WEIGHT_DUE_PROCESS
            applied.append("due_process")
        # Public-urgency only scores for a genuine disseminator (statutory element).
        if "public_urgency" in signals.matched_categories and signals.sender_is_disseminator:
            score += WEIGHT_PUBLIC_URGENCY
            applied.append("public_urgency:disseminator")
    except Exception:
        # Never silently drop a possible expedite: log and re-raise to the DLQ.
        logger.exception(json.dumps({"event": "scoring_error",
                                     "correlation_id": signals.correlation_id}))
        raise

    final = min(100, score)  # bounded by contract
    is_candidate = final >= CANDIDATE_THRESHOLD

    result = {
        "event": "expedite_candidate_scored",
        "correlation_id": signals.correlation_id,
        "text_hash": signals.text_hash,
        "score": final,
        "threshold": CANDIDATE_THRESHOLD,
        "is_expedite_candidate": is_candidate,
        "applied_weights": applied,
        "matched_categories": signals.matched_categories,
        "weights_version": WEIGHTS_VERSION,
    }
    # Append-only audit line (NIST SP 800-53 AU-9: protect audit information).
    logger.info(json.dumps(result))
    return result

3. Route candidates to the human-review gate and log the determination

A candidate is never expedited automatically. It routes to a records officer who applies the statutory test, and the officer’s determination — grant or deny, with a reason and a deadline — is written as its own append-only line. The paired detection and determination records are what demonstrate the process was rule-based end to end.

python

from datetime import datetime, timedelta, UTC


def record_officer_determination(scored: dict, granted: bool, officer_id: str,
                                 rationale: str) -> dict:
    """Persist the human determination that closes an expedite candidacy.

    Records both the recommendation the officer saw and the decision made, so a
    reviewing body can confirm the 5 U.S.C. 552(a)(6)(E) call was made by a
    person within the ten-calendar-day window, not by the engine.
    """
    decided_at = datetime.now(UTC)
    # The statutory clock for the determination itself is 10 calendar days.
    determination_due = decided_at + timedelta(days=10)

    record = {
        "event": "expedite_determination",
        "correlation_id": scored["correlation_id"],
        "text_hash": scored["text_hash"],
        "detector_score": scored["score"],
        "granted": granted,
        "reason_code": "compelling_need_confirmed" if granted else "compelling_need_not_shown",
        "officer_id": officer_id,
        "rationale": rationale[:500],           # bounded free text, audit-preserved
        "determination_due": determination_due.isoformat(),
        "decided_at": decided_at.isoformat(),
        "weights_version": scored["weights_version"],
    }
    logger.info(json.dumps(record))
    return record

A granted determination hands the request to Deadline Tracking & Escalation, which starts the ten-day clock, while both the detection and determination lines flow to the durable store described in Audit Logging Architecture. For the general urgency-scoring kernel that this detector specializes, see Implementing Dynamic Priority Scoring for Urgent Municipal Requests.

Validation & Verification

Treat the detector like any safety-critical pure function and prove its invariants rather than trusting inspection:

Determinism. Extracting and scoring the same text twice must produce an identical text_hash and score. A divergence means something non-deterministic — a stray hash(), a wall-clock input — leaked into the path.
Threshold correctness. A request whose only signal is public urgency from a non-disseminator must stay below the candidate threshold, while a substantiated imminent-threat phrase must clear it. Encode both as parametric cases so a regression names the exact category.
No auto-grant. Assert that no code path in the detector sets granted; only record_officer_determination can, and only with an officer_id. This is the test that proves the statutory determination stays a human act.
Audit completeness. Capture the logger in tests and assert every scored request emits exactly one expedite_candidate_scored line carrying weights_version and text_hash, and that a determination emits its own paired line.

python

def test_detector_recommends_but_never_grants(caplog):
    with caplog.at_level(logging.INFO, logger="intake.expedite.detect"):
        sig = extract_signals("There is an imminent threat to life or safety.",
                              "press", "req-2026-08812")
        scored = score_compelling_need(sig)
    assert scored["is_expedite_candidate"] is True    # 60 >= 50
    assert "granted" not in scored                     # detector never grants
    line = json.loads(caplog.records[-1].message)
    assert line["weights_version"] == scored["weights_version"]

Troubleshooting & Edge Cases

Boilerplate false positives. Feeding the whole email body lets a legal disclaimer or a mailing-list footer trigger “urgent” or “safety.” Fix: truncate to MAX_EVALUATION_LENGTH and strip quoted reply chains before matching, so the detector only reads the requester’s actual ask.
Asserted urgency without substantiation. A requester writes “this is extremely urgent” with no qualifying category. Fix: score only the statutory categories, and let the human-review gate — not the keyword — decide, so bare assertions never inflate the candidate rate.
Non-disseminator public-urgency claims. A commercial requester invokes urgency to inform the public but is not primarily engaged in dissemination. Fix: gate the public-urgency weight on the upstream sender classification, as the code does, so the statutory element is enforced before the weight applies.
Determination window vs. response window. The ten-day expedite determination clock is distinct from the underlying response deadline. Fix: start a separate timer for the determination in Deadline Tracking & Escalation, and never conflate the two in the audit record.
Litigation-hold conflict. A request under an active hold must not be auto-surfaced or auto-closed by its score. Fix: treat an active hold as an override that pins the request to a manual lane regardless of the detector’s recommendation.

Compliance Verification Checklist

Compelling-need signals map only to statutory categories under 5 U.S.C. § 552(a)(6)(E) (imminent threat, urgency to inform the public, loss of substantial rights), not loose synonyms.
Request text is length-bounded and unescaped before matching, and patterns are pre-compiled so hostile text cannot cause catastrophic backtracking.
The public-urgency weight applies only when the sender is a genuine disseminator, enforcing the statutory element before scoring.
The detector only recommends; no code path grants an expedite, and the determination requires an officer_id.
Every scored request emits exactly one append-only JSON line carrying text_hash, score, applied_weights, and weights_version (NIST SP 800-53 AU-9).
The officer’s determination is recorded as its own paired line with a reason code and the ten-day determination deadline.
Signal weights load from a versioned configuration source so a policy change is a reviewable commit, not a code edit.
An active litigation or records hold overrides the recommendation and forces a manual lane.

FAQ

Should the engine ever grant expedited processing automatically?

No. A compelling-need determination under 5 U.S.C. § 552(a)(6)(E) is a statutory judgment that must be made by a person and defended as non-arbitrary. The detector’s role is strictly to surface candidates: it scores the substantiating signals, records exactly which weights fired, and routes anything above the threshold to a records officer. The officer’s grant or denial is logged as its own event with a reason code and a deadline. That split keeps throughput high on the obvious cases while ensuring the legal call — and the accountability for it — stays with a human.

How do you stop the detector from over-flagging requests that merely say "urgent"?

Two controls. First, the detector scores only the statutory categories — imminent threat, substantiated urgency to inform the public, and loss of substantial rights — so a bare assertion of urgency with no qualifying phrase never accumulates score. Second, the public-urgency category only counts when the upstream sender classification marks the requester as a genuine disseminator, enforcing a statutory element before the weight applies. If the candidate rate still climbs, the threshold is tuned through versioned configuration and the change is visible in the audit log, never buried in a code edit.

Where does the ten-day determination clock come from, and how is it tracked?

The ten-calendar-day window to decide an expedite request is set by 5 U.S.C. § 552(a)(6)(E) and is separate from the underlying response deadline. When an officer opens a candidate, record_officer_determination stamps a determination_due timestamp ten days out and hands it to deadline tracking, which runs the timer and escalates as it approaches. Keeping the determination clock distinct from the response clock in the audit record is what lets an agency show it evaluated the expedite request in time even if the substantive response is still in progress.

Priority Scoring Algorithms — the parent scoring stage this detector specializes for expedite candidacy.
Implementing Dynamic Priority Scoring for Urgent Municipal Requests — the general urgency-scoring kernel behind this detector.
Deadline Tracking & Escalation — starts and escalates the ten-day determination clock on a granted expedite.
Audit Logging Architecture — the append-only store that preserves the detection and determination records.
State Law Compliance Frameworks — where jurisdiction-specific expedite categories and windows are defined.

← Back to all public records automation topics

Detecting Expedited-Processing Requests from Compelling-Need Signals #

Problem Framing & Statutory Requirement #

Prerequisites & Environment Setup #

How the Detection Works #

Step-by-Step Implementation #

1. Extract bounded compelling-need signals #

2. Score the signals against a versioned weight table #

3. Route candidates to the human-review gate and log the determination #

Validation & Verification #

Troubleshooting & Edge Cases #

Compliance Verification Checklist #

FAQ #

Related #

Detecting Expedited-Processing Requests from Compelling-Need Signals

Problem Framing & Statutory Requirement

Prerequisites & Environment Setup

How the Detection Works

Step-by-Step Implementation

1. Extract bounded compelling-need signals

2. Score the signals against a versioned weight table

3. Route candidates to the human-review gate and log the determination

Validation & Verification

Troubleshooting & Edge Cases

Compliance Verification Checklist

FAQ

Related