US-WI_transaction-widor-2026_scraper#

Browser-driven scraper for the Wisconsin Real Estate Transfer Return (RETR) public dataset.

The Wisconsin Department of Revenue publishes monthly RETR extracts through its Tax Account Portal (TAP), a JavaScript single-page app behind a terms-of-use gate at https://tap.revenue.wi.gov/mta/. The portal generates each month’s CSV on the fly (the download URL is a one-off token), so the file cannot be fetched with a static URL. This module drives a real browser (via Playwright) to:

  1. open the portal and accept the terms of use if prompted,

  2. navigate to the requested year,

  3. switch the report format to CSV (“CSV Report” button),

  4. click the requested month to trigger generation of the CSV, and

  5. capture the resulting download and save it to a target path.

The portal serves one file per year-month named RETRHistoricalReport{YYYYMM}.csv.

Called by openplaces.io.ingester.Ingester._run_download_scraper when a recipe’s source sets download_url_scraper: US-WI_transaction-widor-2026_scraper (the browser-driven peer of download_url_source). The ingester loads this module by file path and calls its standard fetch entrypoint.

Because the portal’s DOM is not part of this codebase and may change, every step uses resilient text/role locators and every selector is overridable through the recipe’s scraper_options block. On failure a screenshot and the page HTML are written next to the target path to aid debugging.

Functions#

fetch(→ pathlib.Path)

Download one month of Wisconsin RETR data to target_path.

Module Contents#

openplaces.io.scrapers.US-WI_transaction-widor-2026_scraper.fetch(partition_id: str, target_path: str | pathlib.Path, portal_url: str | None = None, *, headless: bool = False, timeout_s: int = 120, label: str | None = None, entry_link_text: str = '', accept_terms: bool = True, terms_texts: tuple[str, Ellipsis] = DEFAULT_TERMS_TEXTS, csv_report_text: str = DEFAULT_CSV_REPORT_TEXT, month_style: str = 'name', browser_channel: str | None = None, slow_mo_ms: int = 0, screenshot_on_error: bool = True, verbose: bool = False) → pathlib.Path#

Download one month of Wisconsin RETR data to target_path.

Parameters:

  • partition_id (str) – Year-month to download, as a six-digit YYYYMM string (e.g. '202601').

  • target_path (str or pathlib.Path) – Where to save the downloaded CSV. Parent directories are created.

  • portal_url (str, optional) – TAP portal URL. Defaults to the public RETR landing page.

  • headless (bool, default False) – Run the browser without a visible window. The portal uses bot detection, so a visible (headed) browser is more reliable on first use; flip to True once a run is known to work in your environment.

  • timeout_s (int, default 120) – Per-action timeout in seconds (navigation, waits, clicks, download).

  • label (str, optional) – Human-readable prefix for progress messages (the Ingester passes the recipe id). Defaults to “Wisconsin RETR”.

  • entry_link_text (str, default '') – Empty by default because the default portal_url lands directly on the RETR tool. Set to a link label (e.g. ‘Download Historical RETR Data’) only when starting from the TAP home page rather than the tool URL.

  • accept_terms (bool, default True) – Click the terms-of-use button if the gate is shown.

  • terms_texts (tuple of str) – Candidate labels for the terms-of-use button, tried in order.

  • csv_report_text (str, default 'CSV Report') – Label of the button that switches the report format to CSV.

  • month_style ({'name', 'abbr', 'number'}, default 'name') – How months are labelled on the portal: full name (‘January’), three-letter abbreviation (‘Jan’), or zero-padded number (‘01’).

  • browser_channel (str, optional) – Playwright browser channel (e.g. 'chrome' or 'msedge') to use an installed system browser instead of the bundled Chromium.

  • slow_mo_ms (int, default 0) – Slow each Playwright operation by this many milliseconds. Useful for debugging or to look less robotic to bot detection.

  • screenshot_on_error (bool, default True) – On failure, write {stem}_error.png and {stem}_error.html next to target_path.

  • verbose (bool, default False) – Print progress messages.

Returns:

Path to the downloaded CSV (equal to target_path), or None when the requested year/month is not published yet (the portal shows a “file has not yet been generated” modal, or the year is not listed). None signals the caller to skip the partition.

Return type:

pathlib.Path or None

Raises:

RuntimeError – If Playwright is not installed, or a step fails (e.g. the terms gate, CSV-format button, year, or month could not be located).