Design a Representative Proxy Trial: A Workload Sampling Plan

A proxy trial should represent the decisions your team must make, not merely the pages that are easiest to fetch. Define workload strata, reserve coverage for business-critical cells, and compare the same permitted cell under the same conditions. A single aggregate score must not hide an untested critical region, variant, or page class.

This guide is for a technical buyer running an authorized monitoring workflow. It supplies a reproducible manual allocation and trial sheet. It does not estimate cost, certify a provider, or turn a quota sample into a statistical confidence claim.

1. Write the decision and workload population

Start with a concrete action: “The inventory operator reviews approved low-stock SKU exceptions in each named market before its local cutoff.” List the permitted source, required field, market/context, page class, validator, and deadline. If a low-frequency decision is answered by a manual check, existing report, API, or licensed dataset, do not build a proxy trial. Use only sources and workflows permitted by applicable terms, licence, contract, or permission; a proxy does not grant access rights.

2. Form strata that could change the decision

A stratum is a named group of comparable scheduled jobs. For the example below, the approved scope contains only US and DE product-detail jobs. Before the trial, classify each reference into exactly one cell: US or DE × standard (not low-stock) or low-stock. Use the approved reference, not a successful response, to classify state. Put an unknown state in an explicit unclassified bucket; do not drop it or infer it from the outcome. The four-cell example assumes none are unclassified. If your list contains unknown states or other markets, HOLD the trial: resolve them or add those cells and rebuild the allocation before running it. Record this scope check in the worksheet; an unresolved bucket cannot receive PASS.

NIST describes blocking as controlling nuisance factors within blocks and randomizing what remains. Statistics Canada describes stratification as dividing a population into groups for sampling. These are planning concepts here; this convenience allocation is not a probability design, provider ranking, or confidence interval.

3. Keep population weights separate from trial quotas

Population weights describe the normal authorized workload mix. Derive them from a dated scheduler inventory or agreed job forecast, counting each scheduled job once; record that source. If the mix is unknown, label the weights as planning assumptions, not measured traffic shares. Trial quotas describe what you will deliberately exercise. Use weights to allocate discretionary jobs. Give a critical cell its own evaluated-job minimum even if its production weight is small. This intentional oversampling can make the trial mix unlike production, so the unweighted trial success rate is not an unbiased estimate of overall production success. Report each cell beside any aggregate; this small operational trial supplies no statistical precision guarantee.

4. Reproduce a 50-job allocation

This distributor is hypothetical. In this example, every job has been classified, giving four non-overlapping cells: A, US standard availability (60% population weight); B, DE standard availability (25%); C, US low-stock variant (10%, critical); and D, DE low-stock variant (5%, critical). The plan is 50 distinct scheduled jobs per route. C and D each need eight evaluated matched jobs for coverage and at least seven valid results for acceptance. A documented failure counts as evaluated coverage; a missing log does not. In this illustration, each job gets one top-level request and no retry. A browser job can fetch several resources, so job counts do not specify total network requests or billable traffic. Repeated jobs can share target or network conditions; distinct IDs do not make their outcomes statistically independent.

per-route budget = 50; reserve C=8 and D=8 evaluated jobs; remaining = 34; allocate A:B at 60:25; A = round(34 × 60 / 85) = 24; B = 10; per-route quotas = A24, B10, C8, D8

The allocation is reproducible, not universal. It creates deliberate coverage for C and D; it does not produce a confidence level, forecast, or score for IPHTML or any other provider. The 22/24, 9/10 and 7/8 valid-result gates are illustrative tolerances chosen before the run, not recommended universal rates. Comparing two routes uses the same 50 sample IDs per route—100 top-level attempts under this one-attempt policy—only if the combined traffic fits source limits. The downloadable CSV contains the plan with blank observations and NOT_RUN outcomes.

5. Do not average away an untested critical cell

Record completion and task-validator result separately, alongside permission reference, allowed source/context, validator version, time window, and matched sample ID. Do not replace a missing critical cell with more easy-page attempts. In a valid A24/B10/C8/D8 allocation, if all eight evaluated D jobs are invalid, at most 42 valid results remain. A different, violated allocation A32/B10/C8/D0 can show 49 valid results out of 50 and still requires hold: D has zero evaluated coverage.

Coverage incomplete: evaluated jobs below A24/B10/C8/D8, or required log evidence missing.
Acceptance failed: valid results below A22/B9/C7/D7.
Comparison unclear: target, validator, context, allowed window or concurrency differs.
All checks pass: this trial scope passes; review cost and operating limits before a limited rollout.

Select within each cell before execution. For this example, define two permitted observation windows and split each quota equally: A12+12, B5+5, C4+4, D4+4. Build the eligible task-ID list for each cell/window, draw without replacement, then freeze the selected IDs. Save the selection tool/method, its seed if applicable, and the actual selected list; the saved list is the replay record. Do not handpick known-working pages. If a cell/window has too few eligible jobs, record the shortfall and revise the plan before execution instead of filling it with repeats. For a two-route comparison, use the same sample IDs and alternate or randomize route order within approved windows without exceeding source limits. Keep the client release, timeout, attempt budget and content validator fixed too. A repeat or retry is evidence about the same scheduled job, not a new sample; record it as a deviation from this one-attempt example. A different scheduled window creates a different job, but still does not guarantee statistical independence. If critical quotas do not fit the permitted budget, hold and rescope; do not delete the cell. For outcome definitions, retries, and denominators, use the reliability scorecard.

6. Use the trial sheet before sending requests

Download the bilingual representative-trial CSV. Its four rows are a clearly labelled hypothetical allocation, not IPHTML results. Replace placeholders only with permitted workload cells, preserve evaluated and valid minima, and leave actual observation fields blank until execution. Make one copy per route and fill the route ID, population basis, permission reference, selected-list reference, windows, exact delivery deadline including time zone, validator and decision owner first. The CSV summarizes cells; keep a separate job ledger with sample_id, stratum_id, scheduled_window, route_id, attempt_no, evaluated, valid, deadline_at, delivered_at, evidence_ref. Define “valid” before running: correct item and market, required availability/source fields, and delivery by the agreed deadline. A known failure has an evidence reference; unknown delivery is not silently marked evaluated.

1. List critical cells and evaluated/valid minima first.
2. Set population weights only for the remaining budget.
3. Predefine approved task-ID selection, seed, validator, matched conditions, deadline, and hold rule.
4. Reconcile the job ledger into each row. Mark coverage PASS only when the planned evaluated count and evidence are complete; mark acceptance PASS only when that row’s valid threshold is met. Record comparison PASS only when the saved conditions and sample IDs match. Otherwise use HOLD or FAIL with the reason. Leave NOT_RUN until execution; any NOT_RUN, unresolved scope check or condition deviation prevents approval.
5. Approve only the tested scope when every row and comparison condition passes. Do not commit a whole production workload from one small trial; use the cost model and a limited operating window before expansion.

7. Keep related trial decisions separate

Use the price-monitoring cost model for economics, the regional QA matrix for context controls, and the public-data source-selection guide to confirm that collection is allowed and necessary.