cleanup

Data lifecycle cleanup: tombstone receipts, consumption checks, and the cleanup() and compact() entry points.

The on-disk state of downstream outputs is the consumption ledger: every decision here is recomputed from disk (plus per-file receipts), so sequential notebook runs, driver scripts, and parallel cluster jobs reach the same conclusions from the same evidence.

An output with retention ‘until_consumed’ may be deleted iff every consumer’s output passes its completeness check. Deletion leaves a tombstone receipt (.consumed.json) beside the would-be output path so skip-if-exists logic can honor the deliberate deletion without re-ingesting.

Classes

DataLock

Exclusive lock file under the data root for destructive operations.

Functions

is_orchestrated(→ bool)	True when running under an orchestrator (e.g. Snakemake).
receipt_path(→ pathlib.Path)	Path of the tombstone receipt for an output file or directory.
write_receipt(→ pathlib.Path)	Atomically write a tombstone receipt beside an output path.
read_receipt(→ dict | None)	Return the receipt dict for an output path, or None if absent/corrupt.
discard_receipt(→ None)	Remove the receipt for an output path (tolerant of it being absent).
is_output_complete(→ bool)	True when a recipe output physically exists and passes its checks.
output_conceptually_exists(→ bool)	True when the output exists on disk OR a valid receipt stands in.
receipt_justifies_skip(→ bool)	True when a tombstone receipt justifies skipping regeneration.
cleanup(→ pandas.DataFrame)	Reclaim consumed intermediate outputs upstream of a terminal recipe.
cleanup_consumed_inputs(→ pandas.DataFrame)	Reclaim the consumed direct inputs of one recipe for one admin unit.
compact([recipes, admin_ids, delete, dry_run, ...])	Garbage-collect the data buckets: report and optionally delete.

Module Contents

openplaces.io.cleanup.is_orchestrated() → bool

True when running under an orchestrator (e.g. Snakemake).

Orchestrated runs must produce the physical output file, so receipt-based skips are voided.

openplaces.io.cleanup.receipt_path(output_path) → pathlib.Path

Path of the tombstone receipt for an output file or directory.

openplaces.io.cleanup.write_receipt(output_path, receipt: dict) → pathlib.Path

Atomically write a tombstone receipt beside an output path.

openplaces.io.cleanup.read_receipt(output_path) → dict | None

Return the receipt dict for an output path, or None if absent/corrupt.

openplaces.io.cleanup.discard_receipt(output_path) → None

Remove the receipt for an output path (tolerant of it being absent).

openplaces.io.cleanup.is_output_complete(recipe, admin_id, required_partitions=None) → bool

True when a recipe output physically exists and passes its checks.

• plain parquet: exists AND the footer is readable AND registry-known columns pass the (schema-only) dtype check

• aggregated/partitioned parquet with a coverage footer: additionally, the recorded coverage is a superset of required_partitions

• enrich evidence: coverage covers all sub-admin units, or the COVERAGE_ALL sentinel

Parameters:

• recipe (str or dict) – Recipe ID or loaded recipe dictionary.

• admin_id (str or AdminId or None) – Admin unit of the output.

• required_partitions (iterable of str, optional) – Partition or sub-admin IDs that the output’s coverage footer must include (ignored when the footer is absent or COVERAGE_ALL).

openplaces.io.cleanup.output_conceptually_exists(recipe, admin_id) → bool

True when the output exists on disk OR a valid receipt stands in.

The receipt cascade rule (design section 4.3): a consumer that was itself cleaned up still counts as existing, so receipts of its inputs stay valid and nothing is needlessly re-ingested.

openplaces.io.cleanup.receipt_justifies_skip(recipe, admin_id, orchestrated=None) → bool

True when a tombstone receipt justifies skipping regeneration.

Requires (design section 4.3): retention.cleanup.honor_receipts enabled; not running under an orchestrator; a readable receipt; every recorded consumer output conceptually exists (physically, or via its own receipt); and the consumer set recomputed from the current recipe tree contains no consumer absent from the receipt (a recipe added after the deletion voids the skip).

class openplaces.io.cleanup.DataLock(admin_id=None, timeout_s=10.0, stale_after_s=3600.0)

Exclusive lock file under the data root for destructive operations.

A county-scoped lock (.openplaces.<admin_id>.lock) lets cleanups of different counties run concurrently; the global lock (.openplaces.lock) serializes data-root-wide operations like compact(). Stale locks (older than stale_after_s) are taken over.

openplaces.io.cleanup.cleanup(recipe, admin_ids=None, stages=None, include_images=False, aggressive=False, dry_run=True, verbose=True, force=False) → pandas.DataFrame

Reclaim consumed intermediate outputs upstream of a terminal recipe.

Walks the dependency DAG rooted at recipe, finds outputs with retention ‘until_consumed’ whose consumers are all complete (section 4.2 of the design), deletes them, and writes tombstone receipts. Every decision is recomputed from disk.

Parameters:

• recipe (str or dict) – Terminal recipe the DAG is rooted at (e.g. ‘US_footprint-cheer-2026’).

• admin_ids (str or list, optional) – Admin units in scope; None scans the terminal recipe’s outputs on disk.

• stages (tuple of str, optional) – Only consider upstream recipes of these stages (e.g. (‘ingest’,) to only reclaim the cache).

• include_images (bool) – Image caches are only deleted with this explicit opt-in (or retention.cleanup.include_images in the config); they are always listed in the report.

• aggressive (bool) – Additionally treat ‘core’ outputs as until_consumed for this call (kept only until the curated outputs exist). Enrich evidence stays ‘keep’ regardless: its input images may be gone.

• dry_run (bool) – Default True: only report. Pass dry_run=False to delete.

• verbose (bool) – Print a short summary.

• force (bool) – Skip the cluster-queue guard.

Returns:

Columns: path, bucket, class, action, size_mb, recipe_id, admin_id, blocked_by.

Return type:

pd.DataFrame

openplaces.io.cleanup.cleanup_consumed_inputs(recipe, admin_id, include_images=False, verbose=False) → pandas.DataFrame

Reclaim the consumed direct inputs of one recipe for one admin unit.

Backs the stage entrypoints’ cleanup=’consumed’ hook: after a stage finishes an admin unit, each of its direct inputs is deleted iff every consumer in the recipe tree is complete. Safe when called early — consumers with no output yet block deletion (e.g. the NSI parquet survives the footprint-spine hook until the parcel spine also exists). No-op when retention.cleanup.enabled is false.

Parameters:

• recipe (str or dict) – The stage recipe whose inputs to consider.

• admin_id (str or AdminId) – The admin unit just finished.

• include_images (bool) – Opt-in for image-cache deletion (or retention.cleanup.include_images in the config).

• verbose (bool) – Print a one-line summary when something was reclaimed.

openplaces.io.cleanup.compact(buckets=('cache', 'heap', 'external', 'core', 'out'), recipes=None, admin_ids=None, delete=(), dry_run=True, orphan_min_age_days=14, min_size_mb=0.0, include_shared=False, force=False) → pandas.DataFrame

Garbage-collect the data buckets: report and optionally delete.

Existent-first scan: every file on disk in the requested buckets is parsed for its recipe and admin unit and matched against the active recipe tree, then classified:

• ‘final’: expected output with retention ‘keep’ (report only)

• ‘intermediate/needed’: until_consumed with incomplete consumers

• ‘intermediate/consumed’: until_consumed, all consumers complete; deleted when ‘consumed’ is in delete

• ‘heap’: anything under the heap; deleted when ‘heap’ is in delete

• ‘orphan’: matches no recipe in the current tree; deleted only when ‘orphans’ is in delete (guards below)

• ‘receipt/stale’: receipt whose recorded consumers all vanished; pruned automatically when any deletion is enabled

Orphan-GC guards (all mandatory): compact aborts destructive runs when any recipe fails to parse; refuses orphan deletion when implausibly few recipes loaded; and never touches files matching retention.cleanup.exclude_patterns.

Parameters:

• buckets (tuple of str) – Buckets to scan.

• recipes (list of str, optional) – Restrict the recipe tree to these recipe IDs (None = all).

• admin_ids (list, optional) – Restrict to files under these admin units.

• delete (tuple of str) – Subset of {‘consumed’, ‘heap’, ‘orphans’}; empty = report only.

• dry_run (bool) – Must be False IN ADDITION to a non-empty delete for any deletion (two explicit acts).

• orphan_min_age_days (int) – A file younger than this is never classified as orphan.

• min_size_mb (float) – Drop report rows smaller than this.

• include_shared (bool) – Allow orphan deletion in shared buckets (external, share, raw).

• force (bool) – Skip the cluster-queue guard.

Returns:

Columns: path, bucket, class, action, size_mb, recipe_id, admin_id, blocked_by.

Return type:

pd.DataFrame