HTTP 200, Wrong Data: Validate Fields Before Business Alerts

An HTTP 200 and a well-typed record can still be unsafe for a business action. A selector can shift from the intended price to a nearby price, or a source can legitimately change while an old baseline remains. For an authorized proxy-backed collection workflow, treat those as data-quality issues: isolate affected records before they reach an alert, export or decision, retain evidence, and recover through a controlled parser change.

1. Define the contract around the business action

Start with the narrow decision a record could trigger: for example, a permitted product-availability alert for one market. Name the fields, their type, whether each is required, whether it may be null, whether zero is meaningful, and the source evidence needed to trust the field. “Optional” means omission is allowed; “nullable” means the field may be present with null. Neither means a missing required value can become zero.

JSON Schema’s object reference explains that properties are not required by default and additional properties are allowed by default; set the constraints you need deliberately. A required field can still be null unless its schema disallows it. This article uses those ideas as a workflow contract, not as a complete JSON Schema implementation.

2. Keep collection time separate from source time

Record retrieved_at when your workflow received the response, together with the authorized source URL or redacted locator, response fingerprint, parser version and contract version. Record source_updated_at only when the source provides it and its meaning is documented. A fresh retrieval does not prove that the source data is fresh; an unchanged value can still be current. If source time is unavailable, write UNKNOWN and hold any action whose deadline depends on freshness.

Keep the raw response or an access-controlled, redacted reference long enough to reproduce a disputed extraction under your retention policy. Do not put credentials, cookies, personal data or protected page content into ordinary alert logs.

3. Route every record to accept, quarantine, hold or stop

Validation is a gate before a downstream business action, not proof that the source claim is true. A value can have the right type yet come from the wrong element. Aggregate all field checks for one sample_key into one row status: ACCEPT_NEXT_STAGE only when every required field, source-evidence check and any required freshness check passes. Mark missing evidence as UNKNOWN and route the record to HOLD. STOP (access restriction) takes priority, then QUARANTINE (a known invalid or mismatched record), then HOLD. Allowed action statuses are NOT_SENT, HELD, BLOCKED, SENT_AFTER_APPROVAL and UNKNOWN. The table is a teaching example with invented record IDs and outcomes; it is not an IPHTML measurement or a complete validator.

Scrapy documents that an item pipeline can validate items and drop an item so later pipeline components no longer process it. That supports an isolation stage, not a claim that its sample is a sufficient contract: its illustrative price check uses truthiness, which would treat 0 as missing. Build an explicit zero and null rule for your own domain. See Scrapy’s Item Pipeline documentation.

4. Reproduce a plausible wrong-field failure offline

This entirely synthetic fixture uses no network and no real site. Both numbers are integers, so a type check passes after an extractor path drifts from the target offer to a recommended card. The fragment and output make the semantic evidence failure reviewable by hand.

<div id="target" data-price-minor="1299">Target offer</div>
<aside class="recommended" data-price-minor="999">Recommended item</aside>

expected locator: #target[data-price-minor]  → {"price_minor": 1299}
drifted locator:  .recommended[data-price-minor] → {"price_minor": 999}

Both outputs: integer type PASS
Drifted output: source locator and label mismatch → QUARANTINE

Record the selector or extractor path, source fragment or approved redacted evidence, parser version and contract version. A contract check that passes does not establish that the extracted claim is true or semantically correct.

5. Diagnose drift from source evidence, not status code alone

For every quarantined record, retain the contract-field name, reason, source locator, retrieval time, parser and contract version, and a redacted response fingerprint. Compare the intended field’s nearby label, unit, currency and stable page context with the captured evidence. This distinguishes missing markup, a type change, a source-level change and a selector that now selects a plausible but wrong element.

A public r/webscraping question asks about a scraper that keeps returning records after a selector or layout change, including values with a valid type but the wrong meaning and who should update a legitimate-change baseline. It establishes that question exists; the comments, counts and recommendations are not evidence for this workflow.

6. Recover one contract change at a time

1. Freeze the affected downstream action and scope the impacted parser version, source and retrieval window. Do not declare a proxy incident merely because the page contract changed.
2. Propose one parser or contract revision with a reason, a reviewed source sample and an owner. A legitimate source change needs a new baseline; copying an old baseline forward is not approval.
3. Run a small authorized canary through the new version, preserving the old version’s result and evidence. Review field-to-source alignment, null/zero treatment and any freshness limit.
4. Approve or reject the revision and record the contract version. A parser repair alone does not release a HOLD record: each record still needs its own source evidence, freshness condition and permission rule to pass. Reprocess only the approved scope of pending records that still meet those conditions.

Use the reliability scorecard for explicit job outcomes and denominators. Use the incident runbook when a broader production incident needs command and staged restoration. This guide owns the record-level action gate after a technically successful response.

7. Use the blank contract worksheet before scale

Download the bilingual field-contract worksheet. It contains field definitions and blank operational evidence columns. Treat its example rules as a starting point: approve the field contract for the task before use. Copy it per source and contract version; do not overwrite a prior approved baseline. It includes space for the observed value or UNKNOWN, owner, reviewer, review time and approval/baseline evidence. Missing operational evidence stays UNKNOWN, and a quarantined row never reaches the business-action queue just because the HTTP request succeeded.

If an API, licensed feed or direct authorized access meets the action’s evidence and freshness needs, keep that simpler route. For an authorized workflow that may need a rotating route, review rotating proxies and contact the team with a redacted source category, permitted market, contract fields and isolation requirements. Confirm current suitability, permissions and limits before changing a route. This guide makes no IPHTML data-quality, availability or recovery-time promise.

Sources checked October 22, 2026: JSON Schema object reference, Scrapy Item Pipeline documentation and the linked public question. The contract, decisions and examples are editorial teaching material.