diagnostics#

System diagnostics: recipe availability, geographic coverage, disk usage, etc.

Functions#

find_recipes(→ pandas.DataFrame)

Find all recipes for a given entity type.

map_recipe_coverage(, ax, verbose, matplotlib.pyplot.Axes])

Map geographic coverage of recipes for a given entity type.

profile_disk_usage(→ pandas.DataFrame)

Profile disk usage of the openplaces data directories.

list_image_caches(→ pandas.DataFrame)

List downloaded image caches in the external data directory.

Module Contents#

openplaces.diagnostics.find_recipes(entity_type: str, stage: str | None = None) → pandas.DataFrame#

Find all recipes for a given entity type.

Scans the recipes directory and returns a table of available recipes.

Parameters:

  • entity_type (str) – Entity type to search for (e.g. 'building', 'parcel').

  • stage (str or None) – If given, only return recipes whose stage field matches (e.g. 'ingest', 'harmonize'). Recipes without an explicit stage field are treated as 'ingest'.

Returns:

Columns: admin_id, stage, entity_type, source_id, version, n_companion_files. Sorted by admin_id then source_id. Global recipes have an empty string for admin_id.

Return type:

pd.DataFrame

openplaces.diagnostics.map_recipe_coverage(entity_type: str, stage: str | None = None, figsize: tuple = (14, 7), ax: matplotlib.pyplot.Axes | None = None, verbose: bool = False) → tuple[matplotlib.pyplot.Figure, matplotlib.pyplot.Axes]#

Map geographic coverage of recipes for a given entity type.

Plots admin geometries fetched via get_admin(), layering smaller admin units on top of larger ones. Requires admin boundary data (GADM) to be ingested.

Colors encode both source (hue) and admin level (lightness): global recipes appear in a pastel shade, country-level recipes are slightly darker, state-level darker still, and so on.

Parameters:

  • entity_type (str) – Entity type to map (e.g. 'building').

  • stage (str or None) – If given, only map recipes whose stage field matches (e.g. 'ingest', 'harmonize').

  • figsize (tuple) – Figure size (width, height) in inches. Ignored when ax is given.

  • ax (matplotlib.axes.Axes or None) – Axes to plot into. A new figure is created when None.

  • verbose (bool) – Print timing for each stage.

Returns:

(fig, ax)

Return type:

tuple[plt.Figure, plt.Axes]

openplaces.diagnostics.profile_disk_usage(roots: dict[str, pathlib.Path | str] | None = None, min_size_mb: float = 1.0) → pandas.DataFrame#

Profile disk usage of the openplaces data directories.

Walks each root directory once and aggregates file sizes by admin unit and dataset, using the on-disk layout {root}/{admin levels…}/_all/{entity or dataset path…}/{files}.

Parameters:

  • roots (dict or None) – Mapping of label to directory to scan. Defaults to the configured core, external, heap, cache, and out directories that exist.

  • min_size_mb (float) – Drop groups smaller than this size.

Returns:

Columns: root, admin_id, dataset, n_files, size_mb. Sorted by size, descending. Files that do not follow the standard layout are aggregated with the path relative to the root as dataset.

Return type:

pd.DataFrame

openplaces.diagnostics.list_image_caches() → pandas.DataFrame#

List downloaded image caches in the external data directory.

Image caches are directories of the form {external}/{admin path}/_all/image/{source}/{version}, written by the image ingestion recipes (e.g. image-googlesatellite-z20).

Returns:

One row per cache: admin_id, source, version, n_files, size_mb, path. Sorted by size, descending.

Return type:

pd.DataFrame