ingester#

Orchestrates download, unzip, and processing of a recipe into output parquet files.

Submodules#

Classes#

Ingester

Smart data ingester for openplaces ingestion recipes.

Functions#

ingest(→ None)

Instantiate and run ingestion for recipe.

Package Contents#

class openplaces.io.ingester.Ingester(recipe: str | dict = None, admin_ids: str | list | None = None, partition_ids: str | list | None = None, timer: openplaces.timing.Timer | None = None, verbose: bool = False)#

Smart data ingester for openplaces ingestion recipes.

Handles downloads, unzipping, loading, and preprocessing.

ingest(reprocess=False, redownload=False, keep_unzipped=False, target_recipe_id: str | None = None)#

Run the full data ingestion

Parameters:

  • reprocess (bool) – If True, re-runs the data ingestion from the original file even if the output data already exists.

  • redownload (bool) – If True, re-downloads the original data file even if it already exists. Also sets reprocess to True. For image recipes, missing imagery is always fetched on first ingest; redownload only re-fetches images that already exist on disk (otherwise cached images are reused).

  • keep_unzipped (bool) – If True, keeps unzipped files in ‘heap’ folder after the download partition has been processed.

  • target_recipe_id (str, optional) – For image recipes only: recipe ID of the harmonized entity to photograph (e.g. 'US_building-nsi-2022'). Overrides entity_recipe in the image recipe YAML.

show_ingested_geometries(**kwargs)#

Plot the last ingested layer for visual inspection.

Delegates to openplaces.viz.maps.show_ingested_geometries(). See that function for the full list of keyword arguments.

show_random_entity()#

Plot a random entity from the last ingested admin unit with its attributes.

Delegates to openplaces.viz.maps.show_random_entity().

sample_layer(n=5)#

Return a transposed sample of the principal entity DataFrame.

Parameters:

n (int) – Number of rows to sample.

Returns:

Transposed sample of the principal entity table.

Return type:

pd.DataFrame

sample_additional_layer(n=5)#

Return a transposed sample of the first additional layer.

Parameters:

n (int) – Number of rows to sample.

Returns:

Transposed sample of the additional layer table.

Return type:

pd.DataFrame

get_ingested_partition_ids(admin_id) → set[str]#

Return partition ids already ingested for admin_id.

Combines two signals so skipping works whether or not the per-partition files are kept: partition ids whose per-partition output file is on disk, plus those recorded in the aggregated file’s footer coverage (openplaces:partitions, written by aggregate_partitions).

openplaces.io.ingester.ingest(recipe: str | dict, admin_ids: str | list | None = None, partition_ids: str | list | None = None, reprocess: bool = False, redownload: bool = False, keep_unzipped: bool = False, verbose: bool = False, years: int | list[int] | None = None) → None#

Instantiate and run ingestion for recipe.

Convenience wrapper around Ingester(recipe, ...).ingest().

Parameters:

  • recipe (str or dict) – Recipe ID string or loaded recipe dict.

  • admin_ids (str, list, or None) – Admin IDs to process (passed to the Ingester constructor).

  • partition_ids (str, list, or None) – Partition IDs to process (passed to the Ingester constructor).

  • reprocess (bool) – If True, re-run even if output already exists.

  • redownload (bool) – If True, re-download even if source file already exists. Also sets reprocess to True.

  • keep_unzipped (bool) – If True, keep unzipped files in the heap folder after processing.

  • verbose (bool) – Print progress messages.

  • years (int, list of int, or None) – Four-digit calendar years to process. Registry recipes receive this filter directly. For standard recipes partitioned by year or year-month, it is converted to partition_ids.