API, Dataset or Web Scraping? A Data Source Decision Guide

Choose the least-maintenance eligible route for a public-data decision: an official API, a licensed dataset, narrowly authorized collection, or a field-level hybrid. An API is often a good candidate, but it does not automatically win. Each route must pass the same decision, rights, field, coverage, freshness, provenance, quality, and operating-boundary gates.

Consider a distributor that sends a daily stock-exception report. Its inventory operator must review missing availability for approved SKUs in each market before the local replenishment cutoff. The report is useful only if a named operator acts on it. Validate that a buyer needs and will use that report before expanding the workflow; this article makes no revenue forecast.

1. Do not buy a proxy when a simpler source passes

Do not buy proxy capacity when an official API or licensed dataset passes every required gate. Also stop before collection if the source terms, licence, contract, or permission do not allow the proposed activity, if the task requires personal or sensitive data without an appropriate review, or if no owner can act on the result. A low-frequency decision may need only a manual check or existing report.

A proxy is a network component, not a permission grant. The IETF's Robots Exclusion Protocol says its rules are not access authorization. A permissive robots.txt file therefore does not replace source terms, a licence, a contract, or written permission. A refusal, challenge, or rate limit is a reason to stop and use an approved route, not to keep changing identities.

2. Use eight acceptance gates before choosing a source

Use PASS only with current evidence, FAIL for an observed failure, and UNKNOWN for missing or expired evidence. Both FAIL and UNKNOWN block acceptance. Choose “no source yet,” narrow the task, obtain approval, or use a different supplier before a team builds automation.

3. Keep freshness meaning and clocks separate

The worksheet records published_at, source_as_of_or_verified_at, and observed_at, as well as the market timezone and decision deadline. Define the freshness basis for each field: a supplier as-of time or a verified inventory snapshot can show currentness; a last-changed timestamp only says when a value changed, and an ambiguous page-modified time or recent poll is not equivalent. If the required basis is unknown, freshness remains UNKNOWN.

candidate_id,source_type,required_field,coverage_universe,freshness_basis,source_as_of_or_verified_at,observed_at,freshness_status
api-a,official_api,availability,"120 approved SKUs / 2 countries",verified inventory snapshot,unknown,2026-10-13T09:00:00Z,UNKNOWN

In this example, the API may still be useful, but it cannot pass a decision that requires verified currentness. A supplier cadence is evidence only if it applies to the chosen delivery path, field, and contract.

4. Work a source choice through a hypothetical project

Northstar is hypothetical. Its distributor operator needs a daily availability exception report for 120 authorized SKUs in each of two countries: 240 SKU-country units. The action is a same-day inventory review before 10:00 in each selected market’s business timezone (Europe/London for the UK and Europe/Berlin for Germany in this example). For a five-business-day permitted sample, an accepted route must produce a validated record by 09:30 local, retain a documented source as-of or verified-inventory time no more than eight hours old at the 10:00 decision deadline, and retain the owner and source reference. A daily contract cadence alone does not prove delivery before the cutoff.

The minimum viable hybrid is API for 220 units plus authorized collection for 20 approved gaps. It is not “collect everything as a backup.” The reconciliation key is sku + country + business_date; API priority applies only while that API unit remains eligible. An unknown freshness basis, expired permission, failed schema check, or missed cutoff puts the unit on hold rather than silently falling back outside an approved gap.

if the decision or scope approval is no longer current:
    decision = "hold"
elif api_unit passes all eight gates for this decision:
    source_of_truth = "official_api"
elif unit is an approved collection gap and collection_unit passes all eight gates:
    source_of_truth = "authorized_collection"
else:
    decision = "hold; do not infer availability"

All pass/fail results above are assumed for teaching; no supplier was tested. Five sample days can support a limited pilot decision, not a long-term reliability claim. If one required unit is missing, report the gap and hold its decision; do not label it out of stock or silently reduce the 240-unit denominator.

5. Download and complete the acceptance record

Download the bilingual source-selection worksheet. It contains no results or provider recommendations. Each row is one candidate-field-country unit, and every hard-gate status starts as UNKNOWN. It records deadline and clock basis, evidence references, source priority, reconciliation, review, and stop conditions. Do not paste credentials or personal data.

1. Set the market timezone, cutoff, decision owner, field, and coverage denominator.
2. Attach current evidence references for rights, scope, schema, coverage, provenance, and quality.
3. Run only a small permitted sample and fill the freshness basis and clocks separately.
4. Use accept, hold, or reject for each row. Accept only when all eight status columns are PASS and the decision owner/action/deadline plus review, source priority, reconciliation and stop-condition fields are complete. Mark an accepted multi-source plan as a hybrid in notes; that label does not override a failed row.

6. Choose an outcome that matches the evidence

7. Turn the remaining data gap into trial requirements

For the later economics of a permitted monitoring workflow, use the price-monitoring cost model. If the task also depends on what a region actually displays, use the regional QA test matrix to control locale and context. The SEC’s official filing API page illustrates why delivery path matters: its APIs update as filings disseminate, with possible processing delays, while bulk ZIP files are republished nightly at approximately 03:00 ET. That is an SEC filing example, not an inventory or IPHTML benchmark.