HTTP Headers for Web Scraping: Python Examples

HTTP headers describe a request and the response formats your scraper can handle. Start with an honest User-Agent and an Accept value that matches the content you need. Add language or session information only when the task requires it. Headers do not grant access to a website or guarantee that a request will succeed.

Which headers should a web scraper send?

User-Agent: Identify the application making the request. A stable identifier such as ResearchClient/1.0 makes your own traffic easier to diagnose. Changing this string does not turn an HTTP client into a browser.
Accept: Describe the response formats you can process: text/html for HTML or application/json for a JSON endpoint. It differs from Content-Type, which describes a message body you send or receive.
Accept-Language: Specify a language preference when collecting localized content. For example, en-US,en;q=0.9 requests US English first. The server decides whether and how to honor it.
Accept-Encoding: Advertise compression formats the client can decode. Let Requests manage this header unless you have a specific compatibility requirement; do not advertise a decoder you have not installed.
Referer: This describes a referring page when one exists. A direct request may have no referrer. Do not invent one simply to fill out a header list.
Cookie and Authorization: Use only your authorized session or API credentials. Keep them out of source code, shared logs and public examples; avoid copying an entire browser request into a scraper.

A Python Requests example

Install Requests with python -m pip install requests. This example requests HTML and leaves transport details such as Host and Content-Length to the library.

import requests

headers = {
    "User-Agent": "ResearchClient/1.0",
    "Accept": "text/html,application/xhtml+xml;q=0.9",
    "Accept-Language": "en-US,en;q=0.9",
}

with requests.Session() as session:
    response = session.get(
        "https://example.com/",
        headers=headers,
        timeout=(5, 20),
    )
    response.raise_for_status()
    print(response.status_code)
    print(response.headers.get("Content-Type", ""))

For your own endpoint, compare the received headers with what the client sent. Change one setting at a time so you can tell which change affected the result.

Diagnose the response before changing more headers

403 Forbidden: check access requirements and the response message. Replacing headers is not a general solution to denied access.
406 Not Acceptable: check whether the endpoint can provide a format allowed by your Accept preferences.
429 Too Many Requests: reduce request frequency and follow the server's Retry-After instruction when provided. Do not immediately retry in a tight loop.
Unexpected language or HTML instead of JSON: inspect redirects, the final URL and Content-Type before parsing the body.

How headers and proxies fit together

A proxy chooses a network route; headers describe the HTTP request. Configure and test them separately. Follow the Python proxy tutorial for client setup, then use the IPHTML integration guide for service-specific details. The web scraping solution page explains the available proxy workflow.

Do I need every header from a browser?

No. Begin with the endpoint's documented requirements and a small, understandable request. Copying browser headers wholesale can also copy expired session data or credentials. It makes debugging harder without reproducing the browser's behavior.

References: MDN HTTP headers, User-Agent, Accept, Accept-Encoding and 429 responses.