registry_ingester#

RegistryIngester — orchestrates Avenu WebForms registry crawling.

Reads a recipe with a scraper key, iterates MA towns (admin4), looks up each town’s registry base URL from the town_to_registry.csv crosswalk, crawls monthly date partitions via AvenuAdapter, and saves parquet output via the standard openplaces path/save machinery.

Classes#

RegistryIngester

Crawl registry transaction records and save as parquet.

Module Contents#

class openplaces.io.ingester.registry_ingester.RegistryIngester(recipe: str | dict, admin_ids: str | list | None = None, years: int | list | None = None, reprocess: bool = False, verbose: bool = False, partition_ids: str | list | None = None)#

Crawl registry transaction records and save as parquet.

Parameters:

  • recipe (str or dict) – Recipe ID string or loaded recipe dict. Must contain a scraper key with Avenu WebForms configuration.

  • admin_ids (str, list, or None) – Admin4 IDs (MA towns) to process. If None, processes all towns found in the town_to_registry crosswalk.

  • years (int, list of int, or None) – Four-digit calendar year(s) (e.g. 2024) to crawl, in the order given. If neither years nor partition_ids is provided, crawls the largest available range from scraper.first_year through the current year, newest first.

  • partition_ids (str, list of str, or None) – Specific year-months (YYYYMM, e.g. 202401) to crawl.

  • reprocess (bool) – If True, re-crawl even when a checkpoint marks a partition done.

  • verbose (bool) – Print progress messages.

ingest() → None#

Run the full crawl (synchronous entry point).

Runs in a dedicated thread so that Playwright’s subprocess transport works correctly even when called from Jupyter (whose SelectorEventLoop does not support create_subprocess_exec on Windows).