recipe#

Functions to handle recipes for data ingestion and harmonization

Read and validate recipes, find recipes, build derivatives, get output paths etc.

Classes#

DepEdge

One dependency edge: a recipe consuming another recipe's output.

Functions#

get_recipe(*args, **kwargs)

Load recipe (.yaml, .csv or .xlsx)

get_recipe_dict(filepath, *args, **kwargs)

Read a recipe .yaml file as a dictionary, cast it to schema

get_recipe_by_id(recipe_id, **kwargs)

Shortcut to get recipe_id by its parts

get_recipe_id(→ str)

Return the canonical recipe ID of a loaded recipe.

get_recipe_retention(→ str)

Resolve the retention class of a recipe's output.

build_table_recipe(→ dict)

Merge a primary recipe with an additional_layers spec.

get_table_recipe(→ dict)

Return the merged recipe for a secondary layer identified by entity.

find_recipe_id(admin_id, entity_or_dataset[, ...])

Find a recipe ID by admin_id and entity/dataset identifier.

iter_entity_sources(→ frozenset)

Return the (entity_type, source_id) pairs across all entity recipes.

provenance_suffixes(→ tuple[tuple[str, str], Ellipsis])

Provenance suffix -> source key, auto-generated from existing recipes.

split_provenance_suffix(→ tuple[str, str | None])

Split a trailing provenance suffix off name; return (base, source).

resolve_attribute_name(→ str)

Resolve a possibly provenance-suffixed column to its registry attribute.

source_id_from_recipe_id(→ str)

Extract the source id from a recipe id.

find_admin_recipe_id(admin_id, admin_level[, silent])

Find the ID of an administrative data ingestion recipe

find_entity_recipe_id(admin_id, entity_type[, stage, ...])

Find the most suitable entity recipe.

get_recipe_dependencies(→ list[DepEdge])

Extract upstream recipe references from a recipe.

get_layers(→ list[str])

Return the layer names available for a recipe's 'additional_layers'.

get_output_path(recipe[, admin_id, partition_id, geo, ...])

Return the path where recipe output is written.

get_save_admin_level(recipe[, operation_keys])

Return the admin level at which output files are split.

get_process_admin_level(recipe)

Return the admin level at which data is chunked for processing.

get_download_admin_level(recipe)

Return the admin level at which downloads are partitioned.

get_partition_ids(recipe)

Return the list of valid partition ID strings for a recipe.

Module Contents#

openplaces.recipe.get_recipe(*args, **kwargs)#

Load recipe (.yaml, .csv or .xlsx)

Parameters:

  • args (tuple) – Arguments for openplaces.path.recipe_path

  • kwargs (dict) – Keywords arguments. Those in openplaces.path.OpenPlacesReference and openplaces.path.recipe_path will be used to find the path, the remainder is passed to the reading functions: - yaml.safe_load() - pd.read_csv() - pd.read_excel()

openplaces.recipe.get_recipe_dict(filepath, *args, **kwargs)#

Read a recipe .yaml file as a dictionary, cast it to schema

Parameters:

  • filepath (pathlib.Path) – Filepath to .yaml file

  • args (list) – Passed on from get_recipe

  • kwargs (dict) – Passed on from get_recipe

openplaces.recipe.get_recipe_by_id(recipe_id, **kwargs)#

Shortcut to get recipe_id by its parts

Assumes syntax: {admin_id}_{entity}_{filename}.{extension}

admin_id or filename can be missing

(Datasets for non-entities aren’t yet supported)

Parameters:

  • recipe_id (str) – Identifier or a recipe

  • kwargs (dict) – Keyword arguments will be passed on to get_recipe()

openplaces.recipe.get_recipe_id(recipe: str | dict) → str#

Return the canonical recipe ID of a loaded recipe.

The ID is the recipe file’s stem, recorded by get_recipe_dict at load time. For recipe dicts constructed without a file (e.g. in tests), the ID is rebuilt from admin_id and entity/dataset; filename suffixes cannot be recovered in that case.

Parameters:

recipe (str or dict) – Recipe ID string (returned unchanged, minus a .yaml extension) or a loaded recipe dictionary.

openplaces.recipe.get_recipe_retention(recipe: str | dict) → str#

Resolve the retention class of a recipe’s output.

Combines the output bucket’s default (STANDARD_DIRS), configuration overrides, and the recipe’s own save_to.retention via retention_for().

Parameters:

recipe (str or dict) – Recipe ID or loaded recipe dictionary.

openplaces.recipe.build_table_recipe(primary_recipe: dict, layer_spec: dict) → dict#

Merge a primary recipe with an additional_layers spec.

Per-table keys (entity, layer, columns, index config, etc.) are taken from layer_spec when present, otherwise removed so that primary-only values do not bleed into the secondary table. process_by is inherited from the primary unless layer_spec sets it explicitly (use ‘process_by: null’ in the YAML to disable chunking for a specific additional table).

Parameters:

  • primary_recipe (dict) – Loaded primary recipe dictionary.

  • layer_spec (dict) – One entry from the primary recipe’s ‘additional_layers’ list.

Returns:

Merged recipe dict for the layer.

Return type:

dict

openplaces.recipe.get_table_recipe(recipe: str | dict, layer: str) → dict#

Return the merged recipe for a secondary layer identified by entity.

Parameters:

  • recipe (str or dict) – Primary recipe (ID string or loaded dict).

  • layer (str) – Entity type (e.g. ‘property’) or full entity string (e.g. ‘property-massgis-2025’) of the additional layer.

Returns:

Merged recipe dict for the requested layer.

Return type:

dict

Raises:

KeyError – If no additional_layers entry matching layer is found.

openplaces.recipe.find_recipe_id(admin_id, entity_or_dataset, filename=None, silent=False)#

Find a recipe ID by admin_id and entity/dataset identifier.

Parameters:

  • admin_id (str) – Administrative unit identifier.

  • entity_or_dataset (str) – Entity or dataset identifier string, may contain glob wildcards (e.g. ‘parcel--’, ‘admin-census-2021’).

  • filename (str, optional) – Filename stem to match within the recipe directory. When None (default), matches any .yaml file in the entity directory. A .yaml extension is appended automatically if absent.

  • silent (bool) – If True, suppress the message printed when multiple recipes are found.

openplaces.recipe.iter_entity_sources() → frozenset#

Return the (entity_type, source_id) pairs across all entity recipes.

Scans the bundled recipes directory once (cached) and parses each recipe filename for its entity token. Files whose entity token does not parse are skipped. A bare entity token (e.g. footprint) followed by a {theme}-{source}-{version} remainder — an entity+dataset enrich recipe such as US_footprint_built-n-stories-brails-2026 — falls back to the dataset’s own source id. Used to auto-generate the provenance suffix vocabulary so adding a new source needs no hardcoded list edits.

openplaces.recipe.provenance_suffixes() → tuple[tuple[str, str], Ellipsis]#

Provenance suffix -> source key, auto-generated from existing recipes.

For every (entity_type, source) pair known to the recipes, generate the column suffixes the harmonizer can produce: _{entity}_{source} and the bare _{source} fallback (e.g. _building_nsi and _nsi; _footprint_fema and _fema). Parcels are interchangeable, so they map by the entity-only _parcel. Returned longest-first so a specific suffix wins over its bare fallback. No hardcoded list — adding a source recipe extends this automatically.

openplaces.recipe.split_provenance_suffix(name: str) → tuple[str, str | None]#

Split a trailing provenance suffix off name; return (base, source).

openplaces.recipe.resolve_attribute_name(column: str) → str#

Resolve a possibly provenance-suffixed column to its registry attribute.

An exact registry entry always wins, so genuinely distinct attributes whose names merely end in a source-like token (n_footprints_per_parcel, priority_on_parcel, parcel_id_local) resolve to themselves. Only unregistered names fall back to stripping a provenance suffix (improvement_value_parcel -> improvement_value); a name that is neither registered nor suffixed is returned unchanged.

openplaces.recipe.source_id_from_recipe_id(recipe_id: str) → str#

Extract the source id from a recipe id.

A recipe id is {admin_id}_{entity_or_theme}-{source}-{version}[...]; takes the last _-delimited token, then the second --delimited field within it (e.g. 'US_building-nsi-2022' -> 'nsi'). Falls back to the whole token when it has no - (an un-versioned or otherwise irregular recipe id).

openplaces.recipe.find_admin_recipe_id(admin_id, admin_level, silent=False)#

Find the ID of an administrative data ingestion recipe

Parameters:

  • admin_id (str) – Administrative unit identifier

  • admin_level (int) – Administrative level for which a recipe is sought.

  • silent (bool) – If True, suppress the message printed when multiple recipes are found.

openplaces.recipe.find_entity_recipe_id(admin_id, entity_type, stage: str | None = None, source_id: str | None = None, filename: str | None = None, silent: bool = False)#

Find the most suitable entity recipe.

Recipes follow the pipeline order ingest, harmonize, enrich, curate unless stage is specified. Within a stage, prefer the requested source, the most specific applicable administrative scope, and the latest version.

class openplaces.recipe.DepEdge#

Bases: NamedTuple

One dependency edge: a recipe consuming another recipe’s output.

recipe_id#

ID of the consuming recipe the edge was extracted from.

Type:

str

upstream_recipe_id#

ID of the consumed recipe; None when auto-discovery could not resolve a concrete recipe (see resolved).

Type:

str or None

kind#

Reference style: the recipe key the edge came from (‘entity_recipe’, ‘image_recipe’, ‘recipe_id’, ‘admin_recipe_id’, ‘tile_recipe_id’, ‘footprint_recipe_id’, …) or ‘auto_discover’.

Type:

str

step#

Pipeline step name (or top-level recipe section) where the reference was found.

Type:

str or None

resolved#

False when an auto-discovered reference could not be resolved to a concrete recipe. Consumers of the dependency graph must treat unresolved edges as “may consume anything” (fail safe).

Type:

bool

openplaces.recipe.get_recipe_dependencies(recipe, admin_id=None) → list[DepEdge]#

Extract upstream recipe references from a recipe.

Edge sources (all present in committed recipes today):

  • top-level ‘entity_recipe’ (curate/enrich -> harmonized spine) and ‘image_recipe’ (enrich -> image ingest); enrich recipes without an explicit ‘entity_recipe’ resolve their spine dynamically, mirroring the enricher

  • any key matching the suffix ‘recipe_id’ anywhere in the recipe (‘recipe_id’ in pipeline sources and steps, ‘admin_recipe_id’, ‘download_by.tile_recipe_id’, ‘footprint_recipe_id’, merge_enrichments ‘recipes’ entries, …); keys under a ‘*crosswalk’ block and ‘remap_id’ are excluded (value crosswalks, not data dependencies)

  • pipeline steps or source entries with ‘auto_discover’ or a bare ‘entity_type’, resolved per admin unit the same way the pipeline resolves them at run time

Parameters:

  • recipe (str or dict) – Recipe ID or loaded recipe dictionary.

  • admin_id (str or AdminId, optional) – Admin unit to resolve auto-discovered references for. When None, auto-discovered references are returned as unresolved edges.

Returns:

Unresolved auto-discovery is returned as an edge with upstream_recipe_id=None and resolved=False (fail safe: the caller must assume such a recipe may consume anything it protects).

Return type:

list of DepEdge

openplaces.recipe.get_layers(recipe: str | dict) → list[str]#

Return the layer names available for a recipe’s ‘additional_layers’.

These are the values accepted by the layer argument of ‘get_entities’ and ‘get_output_path’.

Parameters:

recipe (str or dict) – Recipe dict or recipe ID string.

Returns:

Entity type strings (e.g. ‘property’, ‘transaction’) for each entry in ‘additional_layers’.

Return type:

list of str

openplaces.recipe.get_output_path(recipe, admin_id=None, partition_id=None, geo=False, layer=None, entity_recipe_id=None)#

Return the path where recipe output is written.

Mirrors Ingester._get_output_path without instantiating an Ingester. The output root is determined by ‘save_to’: ‘data_dir’ in the recipe (default: ‘cache’), which must name a directory registered in STANDARD_DIRS.

Parameters:

  • recipe (str or dict) – Recipe identifier (as accepted by get_recipe_by_id) or a pre-loaded recipe dict.

  • admin_id (str or AdminId, optional) – Administrative unit for which to resolve the output path. Pass None for recipes not split by admin unit.

  • partition_id (str, optional) – Partition value appended to the filename stem, e.g. ‘US-NC-BS_footprint-obm-2025_032012.parquet’ for a tile partition with id ‘032012’. Pass None (default) to obtain the final, merged output path.

  • geo (bool, optional) – If True, return the path to the companion ‘_geo.parquet’ file instead of the attribute parquet file.

  • layer (str, optional) – Entity type (e.g. ‘property’) or full entity string (e.g. ‘property-massgis-2025’) of a secondary layer defined in additional_layers. If given, the path for that layer is returned instead of the primary entity’s path.

  • entity_recipe_id (str or dict, optional) – Concrete entity recipe to enrich when recipe has stage ‘enrich’.

Returns:

Resolved output path for the recipe data file.

Return type:

pathlib.Path

openplaces.recipe.get_save_admin_level(recipe, operation_keys=('download_by', 'process_by', 'save_to'))#

Return the admin level at which output files are split.

When save_to: admin_level is explicitly set it defines the output granularity directly — process_by or download_by may be finer (aggregation) or coarser than this level. When save_to: admin_level is absent the level is the maximum found across the given operation keys, falling back to the recipe’s own admin ID depth.

Parameters:

  • recipe (dict) – Loaded recipe dictionary.

  • operation_keys (tuple of str) – Recipe section keys to inspect for ‘admin_level’. ‘save_to’ is included by default since save_to: admin_level controls output granularity. Override when calling from other recipe runners.

Returns:

Admin level for output files (0 = no admin split).

Return type:

int

openplaces.recipe.get_process_admin_level(recipe)#

Return the admin level at which data is chunked for processing.

openplaces.recipe.get_download_admin_level(recipe)#

Return the admin level at which downloads are partitioned.

openplaces.recipe.get_partition_ids(recipe)#

Return the list of valid partition ID strings for a recipe.

Returns [None] for recipes without a ‘download_by’: ‘partition’ key.

Parameters:

recipe (dict) – Loaded recipe dictionary.

Return type:

list of str or list of None

Raises:

  • ValueError – If ‘download_by’: ‘partition’ is ‘year’ or ‘year_month’ but ‘first’/’last’ are not defined.

  • NotImplementedError – If ‘download_by’: ‘partition’ names an unrecognised partition type.