PII Redaction Pipelines for Public Records Disclosure

Within Document Retrieval & Parsing, redaction is the last transformation a responsive record undergoes before it leaves the agency — and the one with the least tolerance for error. A public records program does not get to redo a disclosure: once a release copy reaches a requester, any personal identifier that survived under the black boxes is public, forever, and no retraction recovers it. Redaction is therefore not a drawing task where a reviewer paints rectangles over sensitive text; it is a regulated pipeline that must detect protected personal information, decide under an explicit statutory authority whether to withhold it, physically destroy the underlying content on a rendered copy, gate the result behind human sign-off, and emit an audit record that later proves — in an administrative appeal or in litigation — exactly what was withheld and why. This guide builds that pipeline as a deterministic, audit-first system rather than a manual annotation step.

Problem Framing & Statutory Requirement

The disclosure obligation and the withholding authority pull in opposite directions, and a redaction pipeline exists to hold the line between them. FOIA compels release of responsive records, but it permits an agency to withhold personal information under two personal-privacy exemptions: 5 U.S.C. § 552(b)(6), for personnel, medical, and similar files whose disclosure would be a clearly unwarranted invasion of personal privacy, and § 552(b)(7)©, for law-enforcement records whose disclosure could reasonably be expected to constitute an unwarranted invasion of personal privacy. Social Security numbers, dates of birth, home addresses, personal phone numbers, financial account numbers, and third-party names in investigative files are the recurring categories. Deciding which exemption applies to which span is a legal judgment; the pipeline’s job is to surface every candidate span, attach the citation an officer selected, and then make the withholding real on the page.

Two failure modes bound the whole design. Under-redaction — a black box drawn over text that still exists in the file’s content stream — is the catastrophic one, because the “redacted” PDF a requester downloads still yields the covered text on copy-paste, in a text-extraction script, or by deleting the annotation layer. Agencies have disclosed Social Security numbers and informant names exactly this way, and the resulting privacy breach is unrecoverable and legally actionable. Over-redaction is the quieter failure: withholding more than an exemption authorizes converts a lawful release into an unlawful one, invites a successful appeal, and erodes the public’s statutory right of access. A defensible pipeline treats both as first-class risks — it must remove covered content beyond recovery and record a specific authority for every withholding so the scope can be defended line by line.

Because the release copy is a legal artifact, the segregability requirement in 5 U.S.C. § 552(b) governs the output: any reasonably segregable non-exempt portion must be released, so the pipeline redacts spans, not whole pages, and preserves the surrounding responsive text intact. The redaction decisions themselves feed the privilege log and the withheld-material index the agency must be able to produce on demand, which is why every step is bound to the same audit discipline enforced across Document Retrieval & Parsing.

Prerequisites

  • Python 3.11 or later, for structured logging, dataclasses, and modern typing across the orchestration layer.
  • A PII detection engine. The reference implementation uses presidio_analyzer for named-entity and pattern recognition, extended with custom recognizers for government identifiers. Detection is a recall problem first — a missed SSN is a breach — so the pipeline is tuned to over-surface candidates and let the reviewer prune.
  • A PDF engine that physically removes content, not one that only draws over it. The reference uses pymupdf (imported as fitz) for its apply_redactions primitive, which deletes the underlying text and image objects inside a redaction rectangle rather than layering a shape on top.
  • A verified text layer. Born-digital PDFs expose selectable text directly; image-only scans have none and must pass through OCR Processing Pipelines first, because you cannot redact by entity what the detector cannot read.
  • A write-once audit sink — structured JSON to stdout in development, forwarded to an append-only store or SIEM in production through the discipline described in Audit Logging Architecture — so every redaction decision is reconstructable years later.
  • Least-privilege execution, so the redaction worker can read only the record under review and write only the release copy and the redaction log.

Architecture Overview

The pipeline is a linear sequence of gates, and the ordering is load-bearing: detection must precede policy, policy must precede an irreversible burn-in, and no release copy exists until a human has signed off. The burn-in stage is the pivot — everything upstream is a proposal on a working copy, and everything downstream operates on flattened pixels from which the withheld content has already been destroyed.

Exemption-aware PII redaction pipeline Five stages run left to right: detect entities such as SSN, name, and date of birth; map each candidate span to a FOIA privacy exemption code (b)(6) or (b)(7)(C); burn in and rasterize to physically remove content; route to reviewer sign-off; and emit the disclosure copy as a single flattened layer. Every stage writes a dashed line down to an append-only redaction audit log spanning the width of the diagram, recording what was withheld and why. Detect entities SSN · name · DOB Map exemptions (b)(6) · (b)(7)(C) Burn-in / rasterize Reviewer sign-off Disclosure copy single flat layer Append-only redaction audit log what was withheld · statutory authority · reviewer · hash

The upstream detector and policy mapper are deliberately non-destructive: they annotate a working copy with candidate spans and proposed authorities, and they are safe to re-run and re-tune. The moment of no return is the burn-in, which flattens the page to pixels and deletes the covered content from the underlying stream. Placing human sign-off between the reversible proposal and the irreversible output is what keeps a false positive from silently over-withholding and a false negative from silently leaking — the reviewer sees exactly the spans the detector proposed, mapped to the authority an officer will have to defend.

Step-by-Step Implementation

1. Detect candidate PII spans

Detection runs first and is tuned for recall. The analyzer returns every span that matches a built-in recognizer (person names, phone numbers, email addresses) or a custom government-identifier recognizer (SSN, case numbers, badge numbers), each with a confidence score and character offsets into the page text. The single most important rule at this stage is that a missed entity is a privacy breach, so borderline matches are surfaced rather than dropped — the reviewer prunes over-inclusion, but no automated step can recover an entity the detector never proposed. Custom recognizers for SSNs and agency-specific identifiers are covered in depth in Redacting SSNs and PII from FOIA PDFs with Microsoft Presidio.

python 
"""
redaction_pipeline.py
Exemption-aware PII redaction for FOIA / public-records disclosure.
Detection -> exemption policy -> irreversible burn-in -> reviewer gate -> release copy + audit log.
"""

import hashlib
import json
import logging
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any, Dict, List, Optional

# --- Structured JSON audit logging -----------------------------------------
# Every withholding decision must be reconstructable in an appeal or in
# litigation: who withheld what span, under which statutory authority, when.
class JSONAuditFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname,
            "event": record.getMessage(),
            "request_id": getattr(record, "request_id", "N/A"),
            "exemption": getattr(record, "exemption", "N/A"),
            "span_hash": getattr(record, "span_hash", "N/A"),
        }
        return json.dumps(payload)

audit_logger = logging.getLogger("redaction_audit")
audit_logger.setLevel(logging.INFO)
_handler = logging.StreamHandler()
_handler.setFormatter(JSONAuditFormatter())
audit_logger.addHandler(_handler)


@dataclass
class Candidate:
    entity_type: str          # SSN, PERSON, PHONE_NUMBER, CASE_NUMBER, ...
    text: str                 # the matched substring (never logged in the clear)
    start: int                # character offset into the page text layer
    end: int
    score: float              # detector confidence, 0.0 - 1.0
    exemption: Optional[str] = None   # assigned by the policy mapper


def detect_candidates(page_text: str, analyzer: Any) -> List[Candidate]:
    """Surface every PII candidate. Tuned for recall: a missed SSN is a breach."""
    results = analyzer.analyze(text=page_text, language="en")
    candidates: List[Candidate] = []
    for r in results:
        candidates.append(
            Candidate(
                entity_type=r.entity_type,
                text=page_text[r.start:r.end],
                start=r.start,
                end=r.end,
                score=r.score,
            )
        )
    return candidates

2. Map each span to an exemption authority

Detection tells you what a span is; policy decides whether and under which authority it is withheld. A machine cannot make the ultimate legal call, but it can apply the agency’s standing policy to propose a citation — Social Security numbers and dates of birth map to the personal-privacy exemptions, third-party names in an investigative file map to the law-enforcement privacy exemption — and flag anything ambiguous for a human. Encoding this as an explicit map, rather than burying it in conditionals, is what lets the withheld-material index cite a specific subsection for every box. The taxonomy that assigns records to exemption categories is designed in FOIA Request Taxonomy Design, and the model logic that resolves § 552(b)(6) versus § 552(b)(7)© is detailed in Classifying FOIA Privacy Exemptions (b)(6) and (b)(7)© in Python.

python 
# Standing agency policy: entity type -> personal-privacy exemption authority.
# 5 U.S.C. § 552(b)(6): personnel/medical/similar files, clearly unwarranted
#   invasion of personal privacy.
# 5 U.S.C. § 552(b)(7)(C): law-enforcement records, unwarranted invasion of
#   personal privacy of third parties.
EXEMPTION_POLICY = {
    "US_SSN": "(b)(6)",
    "DATE_TIME": "(b)(6)",          # dates of birth in personnel files
    "PHONE_NUMBER": "(b)(6)",
    "PERSON": "(b)(7)(C)",          # third-party names in investigative records
    "CASE_NUMBER": "(b)(7)(C)",
}
REVIEW_THRESHOLD = 0.85             # below this, force human adjudication


def map_exemptions(candidates: List[Candidate], request_id: str) -> List[Candidate]:
    """Attach a proposed statutory authority; low-confidence spans go to review."""
    for c in candidates:
        c.exemption = EXEMPTION_POLICY.get(c.entity_type)
        # Fail toward review, never toward silent release of an unmapped entity.
        if c.exemption is None or c.score < REVIEW_THRESHOLD:
            c.exemption = "REVIEW"
        span_hash = hashlib.sha256(c.text.encode("utf-8")).hexdigest()[:16]
        audit_logger.info(
            "candidate_mapped",
            extra={"request_id": request_id, "exemption": c.exemption,
                   "span_hash": span_hash},
        )
    return candidates

3. Apply redactions and burn in the release copy

This is the irreversible stage, and it must physically remove content. Marking a redaction rectangle is not enough on its own — the covered text and image objects have to be deleted from the page’s content stream, and the page then rasterized so no vector text or hidden object survives beneath the black fill. Any span still tagged REVIEW blocks the burn-in entirely; the pipeline refuses to produce a release copy while an unadjudicated candidate remains. The mechanics of destroying the underlying layers and verifying nothing survives are the subject of Flattening PDF Layers for Irreversible Redaction.

python 
@dataclass
class RedactionResult:
    request_id: str
    release_pixels: bytes = b""
    withheld_index: List[Dict[str, Any]] = field(default_factory=list)
    status: str = "PENDING"           # PENDING | BLOCKED | RELEASED
    error: Optional[str] = None


def apply_and_burn_in(page: Any, candidates: List[Candidate],
                      request_id: str) -> RedactionResult:
    """Physically remove covered content, then rasterize to a flat layer."""
    result = RedactionResult(request_id=request_id)

    # A single unadjudicated span must block the entire release: never burn in
    # a copy while a candidate's authority is still unresolved.
    pending = [c for c in candidates if c.exemption == "REVIEW"]
    if pending:
        result.status = "BLOCKED"
        result.error = f"{len(pending)} span(s) awaiting reviewer adjudication"
        audit_logger.warning("burn_in_blocked",
                             extra={"request_id": request_id})
        return result

    try:
        for c in candidates:
            rects = page.search_for(c.text)     # locate span -> page coordinates
            for rect in rects:
                # Add a redaction annotation whose apply step deletes the
                # underlying text/image objects, not merely covers them.
                page.add_redact_annot(rect, fill=(0, 0, 0))
                result.withheld_index.append({
                    "entity_type": c.entity_type,
                    "exemption": c.exemption,
                    "bbox": [rect.x0, rect.y0, rect.x1, rect.y1],
                })
        # apply_redactions() removes the covered content from the stream.
        page.apply_redactions()
        # Rasterize: flatten to pixels so no vector text survives under a box.
        pix = page.get_pixmap(dpi=200)
        result.release_pixels = pix.tobytes("png")
        result.status = "PENDING"       # still requires reviewer sign-off below
        return result
    except Exception as exc:
        # Fail closed: a burn-in error must never yield a partially redacted copy.
        result.status = "BLOCKED"
        result.error = str(exc)
        audit_logger.error("burn_in_failed", extra={"request_id": request_id})
        return result

4. Gate on reviewer sign-off, then emit the copy and log

The flattened copy is a candidate, not a release. A named reviewer must approve the withheld-material index — confirming both that nothing over-broad was withheld and that every box carries a defensible authority — before the file is marked releasable. Sign-off is itself an audited event: it records who approved the disclosure, against which content hash, at what time. This closes the chain that an appeal will later walk.

python 
def finalize_release(result: RedactionResult, reviewer_id: str) -> RedactionResult:
    """Human sign-off is mandatory before a copy becomes releasable."""
    if result.status != "PENDING" or not result.release_pixels:
        # Do not release a blocked or empty result under any circumstances.
        result.status = "BLOCKED"
        return result

    release_hash = hashlib.sha256(result.release_pixels).hexdigest()
    for entry in result.withheld_index:
        audit_logger.info(
            "withholding_signed_off",
            extra={"request_id": result.request_id,
                   "exemption": entry["exemption"],
                   "span_hash": release_hash[:16]},
        )
    audit_logger.info(
        f"release_approved reviewer={reviewer_id} sha256={release_hash}",
        extra={"request_id": result.request_id},
    )
    result.status = "RELEASED"
    return result

Validation & Verification

Redaction is a compliance control, so its correctness is asserted, not assumed. Three checks belong in every test suite and every production run:

  • No covered text survives. After burn-in, extract the text of the release copy and assert that none of the withheld strings — and, more strictly, no character inside any redaction bounding box — is recoverable. Because the copy is rasterized, a text-extraction pass over a correctly redacted page returns nothing under the boxes; if it returns a withheld SSN, the burn-in silently failed and the copy must never ship. This is the assertion that catches the under-redaction breach before a requester does.
  • Segregability is preserved. Assert that non-exempt text outside the redaction rectangles remains present and legible in the output. A release that redacts a whole page where a span would suffice fails segregability under § 552(b) and is an over-redaction defect.
  • Withheld-material index completeness. Every redaction box in the output must have a matching index entry carrying its entity type, statutory authority, and coordinates, and every entry must map to a real box. A box with no cited authority — or a citation with no box — is a defect that would leave a withholding indefensible on appeal.

The withheld-material index is not a by-product; it is the deliverable that lets the agency produce a privilege log and answer, span by span, how each withholding was decided. Written to write-once storage through Audit Logging Architecture, it turns a set of black boxes into a defensible record.

Troubleshooting & Edge Cases

  • OCR misses on the text layer. If a scanned record’s text layer is noisy, the detector never sees an entity the human eye can read, and it passes through unredacted. Diagnosis: entities visible on the rendered page with no corresponding candidate span. Fix: gate redaction on an OCR confidence floor, route low-confidence pages to full manual review, and never redact a scan whose text layer was not verified through OCR Processing Pipelines.
  • False negatives on formatting variants. An SSN written 123 45 6789 or split across a line break evades a recognizer tuned for 123-45-6789. Fix: normalize whitespace and hyphenation before detection, add pattern variants to the custom recognizer, and keep the recall bias so variants surface as candidates rather than slipping through.
  • Partial-word and substring matches. Searching for a short withheld string can match an innocuous substring elsewhere on the page, producing an over-redaction. Fix: anchor span search to the detector’s character offsets and map those to coordinates directly, rather than re-searching the page for the raw string.
  • Metadata and hidden-object leakage. The visible page can be perfectly redacted while the same name persists in document metadata, an attached file, or an off-canvas annotation. Fix: strip metadata, attachments, and annotations as part of burn-in and verify the output carries none of the withheld terms anywhere in the file, not just under the boxes.
  • A drawn box mistaken for a redaction. A reviewer places an opaque rectangle annotation and believes the record is redacted, but the text underneath is intact and recoverable. Fix: treat annotation-only “redaction” as a defect the pipeline rejects — only apply_redactions followed by rasterization counts, and the verification step proves it. The FOIA Request Taxonomy Design authority feed keeps the exemption codes on those boxes consistent across the program.

Compliance Verification Checklist

FAQ

Why isn't drawing a black box over text a valid redaction?

Because the text is still there. A filled rectangle drawn on a PDF page is an annotation layered on top of the content stream; the covered characters remain in the file and come straight back on copy-paste, in a text-extraction script, or when the annotation is deleted. Valid redaction physically removes the underlying text and image objects and then rasterizes the page so nothing survives beneath the box. The pipeline treats an annotation-only “redaction” as a defect and proves removal with a text-extraction assertion over the output.

How does the pipeline choose between Exemption (b)(6) and (b)(7)(C)?

It does not make the ultimate legal call — a human does — but it proposes an authority from standing agency policy so every box starts with a defensible citation. Personal identifiers in personnel, medical, and similar files map to the clearly-unwarranted-invasion standard of § 552(b)(6); third-party personal information compiled for law-enforcement purposes maps to § 552(b)(7)©, which applies a broader reasonable-expectation standard. Ambiguous or low-confidence spans are forced to review rather than assigned automatically, and the resolution logic is detailed in the exemption-classification guide.

What stops the pipeline from over-redacting?

Two mechanisms. First, redaction operates on spans mapped to coordinates, not whole pages, so reasonably segregable non-exempt text stays in the release copy — the verification step asserts that surrounding text remains present. Second, a named reviewer must approve the withheld-material index before release, which is the checkpoint where an over-broad withholding is caught and pulled back. Over-redaction converts a lawful release into an appealable one, so it is treated as a real defect, not a safe default.

How do we prove after the fact what was withheld and why?

Through the withheld-material index and the audit log. Every redaction box produces an index entry carrying its entity type, statutory authority, and page coordinates; the reviewer sign-off is logged against the release copy’s content hash; and both are written to write-once storage. When an appeal or a lawsuit asks how a specific withholding was decided, the agency reconstructs the exact span, citation, reviewer, and timestamp rather than relying on recollection — which is what makes the redaction defensible.

← Back to all public records automation topics