Incremental Repository Sync with ETags and Conditional Requests #

Within Repository Sync Protocols, incremental sync with ETags and conditional requests is the technique that lets a worker ask a records repository “has anything changed since I last looked?” and get told no in a single cheap round trip — without re-downloading a byte. This page builds that conditional-request loop against an HTTP source: sending If-None-Match and If-Modified-Since, short-circuiting on 304 Not Modified, capturing the fresh ETag on a 200, persisting a durable cursor, and audit-logging every delta so the chain of custody survives the transfer.

Scenario and Compliance Stakes #

A records office pulls responsive documents from an agency content repository that fronts its collections behind an HTTP API. The collection is large and mostly static — permit files, council minutes, inspection reports that were finalized years ago — but a handful change each day as amendments land and holds lift. Re-listing and re-downloading the whole collection on every sync pass is not just slow; on a metered egress link it is expensive, and under the response clock it is reckless. The federal Freedom of Information Act obliges the agency to a substantive response within 20 business days (5 U.S.C. § 552(a)(6)(A)(i)), so a sync layer that spends hours re-fetching unchanged records is burning deadline it cannot spare.

Conditional requests solve the cost problem, but they also touch a compliance nerve, because what you skip is as consequential as what you fetch. Miss a record that actually changed and the production is incomplete. Re-ingest an unchanged record and re-run its redactions and the disclosure is double-counted. Lose the link between a fetched payload and the validator the server used to gate it, and you cannot prove months later which snapshot a released document was drawn from. The controls follow directly: the sync must be incremental, driven by the server’s own ETag/Last-Modified validators rather than a guessed timestamp; it must be resumable, backed by a durable cursor that a crash cannot corrupt; and every fetched delta is a transfer of custody written to an append-only audit record under NIST SP 800-53 AU-2 and AU-12. Where the source repository predates HTTP validators entirely, the normalization that reconstructs a change signal for it is covered in Syncing legacy document management systems with modern REST APIs; this page assumes the source speaks conditional HTTP.

Prerequisites #

• Python 3.11+ for dataclasses, pathlib, and the logging module used for structured JSON audit output.
• httpx 0.27+ (or requests 2.31+) for HTTP with explicit conditional headers and strict TLS verification enabled by default.
• A durable cursor store — this page uses SQLite so the last ETag, Last-Modified, and pagination position survive a crash; a WORM-backed table or key-value store serves equally.
• Append-only audit storage — WORM object storage or a SIEM-forwarding handler — so per-delta audit lines satisfy NIST SP 800-53 AU-9 and cannot be rewritten after the fact.
• Read-scoped credentials held in a secrets manager, scoped by Security Boundary Configuration so the worker can never widen its own access mid-run.
• A downstream sink that deduplicates on a content-derived key, so a replayed page after a crash is absorbed rather than double-counted — the idempotency discipline detailed in the Repository Sync Protocols parent stage.

Implementation #

The design keeps two pieces of durable state per partition: the last validator the server gave us (ETag and Last-Modified) and the pagination position. On each pass the worker replays those validators as conditional headers. A 304 means the server has certified nothing changed — the worker does zero work and leaves the cursor exactly where it was. A 200 carries the changed page and a fresh ETag; the worker hands each record downstream, audit-logs the delta, and only then advances the durable cursor. Persisting after handoff is what makes a crash replay the page rather than skip it.

import hashlib
import json
import logging
import sqlite3
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Callable

import httpx

# Append-only structured audit log. In production this handler forwards to WORM
# storage / a SIEM (NIST SP 800-53 AU-9) so custody lines cannot be altered after
# the fact. Every fetched delta is a transfer of custody and must be attributable.
logging.basicConfig(level=logging.INFO, format="%(message)s")
log = logging.getLogger("etag_sync")


@dataclass(frozen=True)
class SyncCursor:
    """Durable per-partition state: the server's own validators plus our position.
    Frozen so the value a page was fetched under cannot drift before it is persisted."""
    partition: str
    etag: str | None          # server ETag validator from the last successful 200
    last_modified: str | None # server Last-Modified validator from the last 200
    position: str             # opaque pagination cursor into the collection


class DurableCursorStore:
    """SQLite-backed cursor + ETag cache. Durability is the point: a crashed sync
    must resume from the exact validator it last committed, and an auditor relies on
    that persisted ETag to prove which server snapshot a disclosure was drawn from."""

    def __init__(self, db_path: Path):
        self.conn = sqlite3.connect(str(db_path))
        self.conn.execute(
            "CREATE TABLE IF NOT EXISTS sync_cursor ("
            "partition TEXT PRIMARY KEY, etag TEXT, last_modified TEXT, position TEXT)"
        )
        self.conn.commit()

    def load(self, partition: str) -> SyncCursor:
        row = self.conn.execute(
            "SELECT etag, last_modified, position FROM sync_cursor WHERE partition = ?",
            (partition,),
        ).fetchone()
        if row is None:
            return SyncCursor(partition, None, None, "0")   # first run: no validators yet
        return SyncCursor(partition, row[0], row[1], row[2])

    def save(self, cursor: SyncCursor) -> None:
        # Committed durably only AFTER a page is handed off downstream, so a crash
        # replays the page rather than skipping records (a chain-of-custody gap).
        self.conn.execute(
            "INSERT INTO sync_cursor (partition, etag, last_modified, position) "
            "VALUES (?, ?, ?, ?) ON CONFLICT(partition) DO UPDATE SET "
            "etag=excluded.etag, last_modified=excluded.last_modified, "
            "position=excluded.position",
            (cursor.partition, cursor.etag, cursor.last_modified, cursor.position),
        )
        self.conn.commit()


def _audit(event: str, partition: str, **fields: Any) -> None:
    """One append-only JSON custody line per sync decision (AU-2 / AU-12)."""
    log.info(json.dumps({
        "event": event,
        "partition": partition,
        "ts": datetime.now(timezone.utc).isoformat(),
        **fields,
    }, sort_keys=True))


def sync_partition(client: httpx.Client, base_url: str, partition: str,
                   store: DurableCursorStore,
                   handoff: Callable[[dict[str, Any]], None]) -> int:
    """One conditional sync pass for a partition. Returns the count of new deltas.
    A 304 short-circuits with zero downstream work; a 200 advances the cursor."""
    cursor = store.load(partition)
    headers: dict[str, str] = {"Accept": "application/json"}
    # Replay the server's own validators. If-None-Match is authoritative; the date
    # is a weaker fallback the server may honor when it has no strong ETag.
    if cursor.etag:
        headers["If-None-Match"] = cursor.etag
    if cursor.last_modified:
        headers["If-Modified-Since"] = cursor.last_modified

    try:
        resp = client.get(f"{base_url.rstrip('/')}/records",
                          headers=headers,
                          params={"cursor": cursor.position, "limit": 250},
                          timeout=30.0)
    except httpx.HTTPError as exc:               # transport failure: surface for retry
        _audit("sync_transport_error", partition, error=str(exc))
        raise

    if resp.status_code == 304:
        # Not Modified: the server certifies our snapshot is current. No bytes moved,
        # no transfer of custody, so we emit a no-op line and leave the cursor intact.
        _audit("sync_not_modified", partition, request_etag=cursor.etag)
        return 0

    resp.raise_for_status()                      # 5xx/4xx surface to the caller's retry
    body = resp.json()
    records = body.get("records", [])

    delta = 0
    for raw in records:
        # Hash the canonical serialization so the same record always fingerprints the
        # same way — the chain-of-custody anchor a downstream dedup sink keys on.
        canonical = json.dumps(raw, sort_keys=True, separators=(",", ":")).encode("utf-8")
        payload_sha = hashlib.sha256(canonical).hexdigest()
        handoff(raw)                             # deliver downstream BEFORE cursor advance
        _audit("sync_delta", partition, doc_id=raw.get("id"),
               modified_at=raw.get("modified_at"), payload_sha256=payload_sha)
        delta += 1

    # Advance durable state only after the whole page is handed off. Capture the
    # fresh validators the server returned so the next pass can short-circuit.
    store.save(SyncCursor(
        partition=partition,
        etag=resp.headers.get("ETag", cursor.etag),
        last_modified=resp.headers.get("Last-Modified", cursor.last_modified),
        position=body.get("next_cursor", cursor.position),
    ))
    _audit("sync_page_committed", partition, delta=delta,
           new_etag=resp.headers.get("ETag"))
    return delta


if __name__ == "__main__":
    store = DurableCursorStore(Path("sync_state.db"))
    with httpx.Client(verify=True) as client:   # strict TLS: SC-8/SC-13 in transit
        moved = sync_partition(client, "https://records.example.gov/api",
                               "permits", store, handoff=lambda r: None)
        print(f"permits: {moved} deltas this pass")

The header and status semantics are worth stating plainly, because each is a deliberate choice with a compliance consequence:

Expected Output and Verification #

A pass over an unchanged partition emits a single no-op line and moves nothing; a pass that finds changes emits one delta line per record and one commit line:

{"event": "sync_not_modified", "partition": "permits", "request_etag": "\"a1b2c3\"", "ts": "2026-07-12T14:02:11.418+00:00"}
{"event": "sync_delta", "doc_id": "PRR-2026-0457", "modified_at": "2026-07-11T09:14:00Z", "partition": "permits", "payload_sha256": "5a3d...b8", "ts": "2026-07-12T14:07:04.118+00:00"}
{"event": "sync_page_committed", "delta": 1, "new_etag": "\"d4e5f6\"", "partition": "permits", "ts": "2026-07-12T14:07:04.140+00:00"}

Verify three invariants before trusting a run. First, the short-circuit fires: an unchanged partition must return 0 and emit exactly one sync_not_modified line, proving no bytes were pulled:

def test_304_short_circuits(monkeypatch):
    store = DurableCursorStore(Path(":memory:"))
    store.save(SyncCursor("permits", '"a1b2c3"', None, "0"))

    class FakeResp:
        status_code = 304
        headers: dict[str, str] = {}
        def raise_for_status(self): ...
    class FakeClient:
        def get(self, *a, **k): return FakeResp()

    moved = sync_partition(FakeClient(), "https://x", "permits", store,
                           handoff=lambda r: (_ for _ in ()).throw(AssertionError()))
    assert moved == 0            # handoff must never run on a 304

Second, the cursor advances only forward and only on a 200: after a 304 the stored ETag is unchanged, and after a 200 it equals the server’s new validator — so re-running immediately re-short-circuits. Third, transfer accounting closes: the count of sync_delta events over a window must equal the downstream ingested count for the same window, and no payload_sha256 may appear under two different doc_ids. A nightly reconciliation that flags any mismatch for compliance review closes the loop, and pinning the source repository’s clock via NTP keeps If-Modified-Since from misfiring on skew.

Common Pitfalls #

• Trusting If-Modified-Since over If-None-Match. A date validator has one-second granularity and depends on aligned clocks, so a record edited twice within the same second, or a server whose clock drifts, will report not modified when it changed — a silently incomplete production. Always send the ETag when the server offers one and treat the date only as a fallback; where both are present, the ETag comparison is authoritative.
• Advancing the cursor before the page is handed off. Persisting the new ETag and position before the records reach the downstream sink means a crash mid-page loses every record in it permanently. Persist durable state only after handoff, and rely on the downstream content-keyed dedup to absorb the replay a crash-before-commit produces — replay-safe beats fast.
• Ignoring weak ETags. Some repositories emit weak validators (W/"...") that guarantee semantic, not byte-for-byte, equivalence. If your custody model requires exact bytes, do not treat a weak-ETag 304 as proof the payload is identical; fall back to a content hash comparison on the fetched body, and record which validator class the decision used.
• Letting a 304 clear state on error paths. A misrouted 304 from a proxy or an error page returned with a 304-like status must not be interpreted as “unchanged” and used to skip a real fetch. Verify the response actually came from the records endpoint over verified TLS before treating a short-circuit as authoritative, so a spoofed or cached intermediary can never suppress a genuine delta.

Compliance Verification Checklist #

• Every pass replays the server’s own ETag via If-None-Match, with If-Modified-Since only as a fallback, never as the sole change signal.
• A 304 Not Modified performs zero downstream work, emits one no-op audit line, and leaves the durable cursor untouched.
• A 200 OK captures the fresh ETag and Last-Modified, and the durable cursor advances only after the whole page is handed off.
• Cursor state is persisted to a durable store that survives a crash, so a resumed run short-circuits from the exact validator it last committed.
• Each fetched delta hashes its canonical payload and emits exactly one append-only JSON audit line (NIST SP 800-53 AU-2 / AU-12) forwarded to WORM/SIEM storage (AU-9).
• Weak ETags are recognized and, where exact-byte custody is required, confirmed with a content hash rather than trusted outright.
• All repository traffic runs over strictly verified TLS (SC-8 / SC-13); credentials come from a secrets manager under the least-privilege identity from Security Boundary Configuration.
• A nightly reconciliation confirms sync_delta counts match downstream ingested counts and that no payload hash appears under two document ids.

FAQ #

Related #

• Repository Sync Protocols — the parent ingestion boundary this conditional loop plugs into
• Syncing legacy document management systems with modern REST APIs — reconstructing a change signal when the source has no ETags
• Async Batch Processing — draining synced deltas under bounded concurrency
• Audit Logging Architecture — where per-delta custody lines are retained and made tamper-evident
• Security Boundary Configuration — the least-privilege identity the sync worker runs under

← Back to Repository Sync Protocols