Python Web Crawler: How to Build a Scalable Crawler with asyncio

A Python web crawler is a program that downloads pages, extracts data and links, and schedules further crawling. Teams use crawlers for technical audits, price monitoring, and dataset preparation. In real workloads, speed is limited by website rules, network latency, browser memory, and URL deduplication quality. That is why data scraping starts with a controlled queue, not with the number of open tabs.

A scalable crawler separates scheduling, fetching, parsing, and storage. This design lets you replace an HTTP client with Playwright only for JavaScript pages without rewriting the rest of the system. The guide below provides a working asyncio foundation, polite crawling rules, and a method for finding the point where more workers stop improving throughput.

What Components Make a Scalable Web Crawler

A scalable web crawler consists of a frontier queue, workers, a fetcher layer, a parser, and storage. Each component has one responsibility, so a slow website, selector error, or failed database write does not stop the entire crawl.

The frontier stores normalized URLs and decides which host may be queried next. asyncio.Queue is enough for a small website. For several domains, keep a separate queue per host, its own concurrency limit, and the next permitted request time. This approach pairs well with visual automation workflows when browser actions must remain separate from the scheduler.

The fetcher receives a URL and returns the HTTP status, final address, headers, and response body. It starts with a less resource-intensive HTTP request. When content is rendered in a browser, login is required, or an empty HTML response lacks the target data, the job escalates to Playwright. This HTTP-first approach uses the browser as a precise fallback rather than a universal transport.

The parser should not open network connections. It receives a finished document and extracts the canonical URL, JSON-LD, target fields, and new links. A separate classifier marks the page as ok, blocked, not_found, parse_error, or retryable. For dynamic pages, account for headless browsing signals because a WAF evaluates session behavior as well as User-Agent.

Storage keeps crawl results and state. SQLite in WAL mode is usually enough for one process, while PostgreSQL is appropriate for several crawler instances. JSONL is useful for export but weak as the only store because, after a restart, it is hard to tell which URLs finished and which were still running.

This separation may feel excessive while a crawler processes twenty pages. At tens of thousands of URLs, it saves time: a stalled Playwright session does not block HTTP workers, and data schema changes do not require a frontier rewrite.

Store a structured job in the frontier instead of a plain URL string: url, depth, priority, attempt, ready_at, discovered_from, and a browser-escalation flag. Priority lets product pages or recent articles run before all pagination pages finish. discovered_from helps identify a crawl loop or a bad source link.

Normalize URLs before deduplication. The code below performs only basic normalization of scheme, host case, default ports, fragments, and an empty path. Trailing-slash and query-parameter rules depend on the target site. Remove the fragment after #, lowercase the host, remove default ports, and apply one consistent trailing-slash policy. Parameters such as utm_source can often be discarded, while page=2 identifies a different resource. An allowlist of meaningful parameters is safer than deleting everything after the question mark.

For large crawls, an in-memory seen set is insufficient. Across processes, reserve jobs atomically with a method such as FOR UPDATE SKIP LOCKED in PostgreSQL and use an expiring lease so unfinished work returns after a failure. A Bloom filter saves memory but allows false positives that can skip new URLs. When completeness matters more than maximum speed, store the normalized URL or its cryptographic hash in a table with a unique index.

How to Write a Web Crawler with asyncio and Playwright

Build an asyncio crawler around workers that stay alive for the whole crawl and a queue with controlled load. The teaching example below uses an unbounded queue, while the limit parameter caps how many URLs can be added.

A production system needs a bounded queue and a separate frontier coordinator that prevents worker deadlocks. Do not create one task per URL through an unbounded asyncio.gather(): a large sitemap quickly turns the coroutine list into unnecessary memory usage.

Prepare Python 3.11 or newer and install the dependencies. The example uses Beautiful Soup for HTML parsing and Playwright for pages where target content appears after JavaScript execution.

1. create a virtual environment with python -m venv .venv
2. activate it for your operating system
3. install packages with pip install aiohttp beautifulsoup4 playwright
4. download Chromium with playwright install chromium
5. add a test domain that permits automated crawling

The framework below demonstrates deduplication before queue insertion, task_done() inside finally, a worker limit, and one reused browser. It is an educational minimum, not a ready-made crawler for any site. The normalization function does not validate embedded credentials, DNS responses, or internal IP reachability.

Untrusted URLs require separate address validation, DNS controls, and network restrictions, including SSRF protection. The example does not implement robots.txt, request delays, retries, or persistent state. raise_for_status() sends all HTTP failures to the generic handler, so this teaching code does not distinguish 404, 429, and 503 or use Retry-After.

Implement the status classification and retry rules described later separately. Run the example only against your own or explicitly authorized test resource after checking its access rules.

import asyncio
from dataclasses import dataclass
from urllib.parse import urldefrag, urljoin, urlsplit, urlunsplit

import aiohttp
from bs4 import BeautifulSoup
from playwright.async_api import Error as PlaywrightError, async_playwright

@dataclass
class Job:
    url: str
    depth: int = 0

def normalize(base: str, href: str) -> str | None:
    absolute, _ = urldefrag(urljoin(base, href))
    try:
        parsed = urlsplit(absolute)
        if parsed.scheme.lower() not in {"http", "https"} or not parsed.hostname:
            return None
        if parsed.username is not None or parsed.password is not None:
            return None
        scheme = parsed.scheme.lower()
        host = parsed.hostname.lower()
        port = parsed.port
        if ":" in host:  # IPv6
            host = f"[{host}]"
        if port and not ((scheme == "http" and port == 80) or
                         (scheme == "https" and port == 443)):
            host = f"{host}:{port}"
        # Keep query parameters because some identify the resource.
        # Define trailing-slash and tracking-parameter policies separately.
        return urlunsplit((scheme, host, parsed.path or "/", parsed.query, ""))
    except ValueError:
        return None

async def crawl(start_url: str, limit: int = 200, workers: int = 8):
    start_url = normalize(start_url, start_url)
    if not start_url:
        raise ValueError("Invalid start URL")
    if limit < 1 or workers < 1:
        raise ValueError("limit and workers must be positive")

    queue: asyncio.Queue[Job] = asyncio.Queue()
    seen = {start_url}
    results = []
    start_host = urlsplit(start_url).netloc
    timeout = aiohttp.ClientTimeout(total=20)
    connector = aiohttp.TCPConnector(limit=workers, limit_per_host=2)
    browser_slots = asyncio.Semaphore(2)

    async with aiohttp.ClientSession(
        timeout=timeout,
        connector=connector,
        headers={"User-Agent": "ExampleAuditBot/1.0 contact@example.com"},
    ) as session, async_playwright() as pw:
        browser = await pw.chromium.launch(headless=True)
        try:
            context = await browser.new_context()
            try:
                async def fetch(url: str) -> tuple[str, str]:
                    # Validate each HTTP redirect before the next request.
                    final_url = url
                    for _ in range(6):
                        if urlsplit(final_url).netloc != start_host:
                            raise ValueError("URL outside the allowed host")
                        async with session.get(final_url, allow_redirects=False) as response:
                            if response.status in {301, 302, 303, 307, 308}:
                                location = response.headers.get("Location")
                                if not location:
                                    raise ValueError("Redirect without Location")
                                next_url = normalize(final_url, location)
                                if not next_url or urlsplit(next_url).netloc != start_host:
                                    raise ValueError("Redirect outside the allowed host")
                                final_url = next_url
                                continue
                            response.raise_for_status()
                            html = await response.text(errors="replace")
                            final_url = str(response.url)
                            break
                    else:
                        raise ValueError("Too many redirects")
                    if "data-requires-javascript" not in html:
                        return final_url, html

                    async with browser_slots:
                        page = await context.new_page()
                        # Prevent the browser from requesting other hosts.
                        async def restrict_route(route):
                            if urlsplit(route.request.url).netloc != start_host:
                                await route.abort()
                            else:
                                await route.continue_()
                        await page.route("**/*", restrict_route)
                        try:
                            async with asyncio.timeout(45):
                                await page.goto(final_url, wait_until="domcontentloaded", timeout=30_000)
                                if urlsplit(page.url).netloc != start_host:
                                    raise ValueError("Browser navigated to another host")
                                return page.url, await page.content()
                        finally:
                            await page.close()

                async def worker():
                    while True:
                        job = await queue.get()
                        try:
                            final_url, html = await fetch(job.url)
                            soup = BeautifulSoup(html, "html.parser")
                            results.append({
                                "url": final_url,
                                "title": soup.title.get_text(strip=True) if soup.title else None,
                                "depth": job.depth,
                            })
                            for tag in soup.select("a[href]"):
                                link = normalize(final_url, tag["href"])
                                if not link or urlsplit(link).netloc != start_host:
                                    continue
                                if link not in seen and len(seen) < limit:
                                    seen.add(link)
                                    await queue.put(Job(link, job.depth + 1))
                        except Exception as error:
                            # Teaching example: record the error and keep the worker alive.
                            results.append({"url": job.url, "error": type(error).__name__})
                        finally:
                            queue.task_done()

                tasks = [asyncio.create_task(worker()) for _ in range(workers)]
                try:
                    await queue.put(Job(start_url))
                    await queue.join()
                finally:
                    for task in tasks:
                        task.cancel()
                    await asyncio.gather(*tasks, return_exceptions=True)
            finally:
                await context.close()
        finally:
            await browser.close()
    return results

if __name__ == "__main__":
    data = asyncio.run(crawl("https://example.com"))
    print(data)

Use a few simple tests to check URL normalization. They cover default ports, IPv6, fragments, and query-parameter preservation:

assert normalize("https://EXAMPLE.com:443/a", "#section") == "https://example.com/a"
assert normalize("http://example.com", "/?page=2") == "http://example.com/?page=2"
assert normalize("http://[::1]:80/", "/path") == "http://[::1]/path"
assert normalize("https://example.com", "mailto:info@example.com") is None

These tests validate address transformation only, not network safety. Untrusted URLs also require DNS validation and network restrictions.

The key detail is the job lifecycle, not Playwright. Every queue.put() must be matched by one queue.task_done(). Otherwise, queue.join() waits forever. This failure often looks like a slow network even though the cause is an exception inside a worker.

All workers in the example share one BrowserContext, while a semaphore limits simultaneously open pages. limit_per_host=2 limits concurrent HTTP connections, not requests per second; control crawl rate separately. For public pages without authorization, this is often enough.

When sources require different cookies or proxies:

• create a separate context for each logical session
• do not launch a separate Chromium process for every URL
• close the context after the job group finishes

A context uses fewer resources than a separate browser process. In production, also manage shared-context health because a critical error or lost browser connection may require recreating it.

Plan for:

• a maximum job completion time
• correct coroutine cancellation
• resource cleanup
• external supervision and cleanup of Chromium child processes

The finally block closes pages and contexts during normal completion or coroutine cancellation, but it is not guaranteed to run if the Python process crashes.

The host checks and page.route() request interception in the example restrict some navigations and requests but do not guarantee full browser isolation.

Consider these limitations:

• not every network request necessarily passes through this handler
• Service Worker behavior can introduce exceptions
• DNS can resolve an allowed hostname to an internal IP address

For untrusted sources, use an isolated environment with network policies that block internal addresses and metadata services.

The limit parameter caps unique URLs added to the crawl, not successfully processed pages.

A bounded queue that the same workers both read from and fill can deadlock: every coroutine may wait for free space while none remains available to consume the next URL.

In a production implementation:

• delegate discovered-link insertion to a dedicated frontier coordinator
• apply backpressure between the frontier coordinator, fetcher, and writer

A production system also needs a robots.txt check before requests, per-host rate limits, delayed retries, persistent frontier storage, and a separate writer. Workers send finished records to a second bounded queue, and the writer inserts them into the database in batches. A batch of 100 records usually has less overhead than 100 individual transactions. If the database slows down, the result queue fills gradually and limits new fetching, preventing system overload.

Replace the data-requires-javascript flag with a real check in production. For example, escalate a page to the browser when a required CSS selector is missing, HTML is shorter than an expected minimum, or the response contains a challenge page. Similar pipelines are useful for automated OSINT collection, where sources use different formats and loading modes.

How to Handle Errors and Respect robots.txt

A reliable crawler distinguishes permanent errors from temporary ones and checks access rules before fetching a URL. The teaching code above does not yet implement robots.txt. A 404 normally should not be retried, while 429, 502, 503, and a network timeout require a delayed attempt with exponential backoff and jitter.

First download /robots.txt for every host and cache the result. Check permission for the actual User-Agent, honor Crawl-delay when provided, and discover sitemap links. The absence of a robots.txt prohibition does not override website terms, personal-data requirements, or access restrictions.

Define response handling explicitly:

1. finish the URL without retrying 400, 401, 403, and 404 unless the source policy says otherwise
2. read Retry-After for 429 and 503; it can contain seconds or an HTTP date
3. calculate the delay as min(cap, base * 2 ** attempt) + jitter
4. return the job to the frontier with a future ready time
5. mark the URL failed after exhausting the attempt limit

Do not keep a worker inside asyncio.sleep() for tens of seconds. When the next-attempt time is stored as ready_at in the frontier, the worker can process another host. Separate per-host queues are more effective here than simply raising workers.

Set two timeouts for Playwright:

• a shorter navigation timeout
• a longer timeout for the full job

If page.goto() stalls, the outer asyncio.timeout() cancels the operation after the deadline but does not close the page by itself. Always close the page or context in finally to release the pool slot. Otherwise, several dozen failed navigations can leave Chromium processes in memory.

Retry only idempotent operations:

• a normal GET is suitable for repetition
• clicking a button that creates an order or changes data is not crawling

Maintain a retry budget per host, not only per URL. If half of the requests returned 429 or 503 during the last five minutes, pause new work for that domain for a short cooldown. Otherwise, one thousand URLs with three retries each turn a temporary issue into three thousand unnecessary requests.

A circuit breaker can use three states:

1. closed: requests run normally
2. open: the frontier stops issuing work for the host after the error threshold is exceeded
3. half_open: one test request is allowed after the cooldown

A successful test returns the system to closed; another failure sends it back to open.

Store each failure reason separately:

• dns_error
• connect_timeout
• read_timeout
• http_429
• browser_timeout
• parse_error

These failures need different remedies. One failed counter cannot explain why only seven of ten thousand pages were collected.

How to Crawl Sitemaps, Structured Data, and Updated Pages

For a known website, start with its sitemap rather than the home page. A sitemap provides a controlled URL list, often includes lastmod, and avoids opening navigation pages solely to discover links.

Support sitemap indexes, regular XML files, and gzip. Set a maximum uncompressed size before extraction because a small archive can expand into gigabytes. Normalize URLs and store <lastmod> separately from the actual crawl time. The value may be stale or inaccurate, so do not use it as the only recrawl signal.

Organize the first pass as follows:

1. download robots.txt and find declared sitemaps
2. parse the sitemap index down to final XML files
3. filter URLs by allowed host and required path prefix
4. enqueue pages in the frontier with priority based on type and lastmod
5. fetch HTML over HTTP
6. send complex pages to Playwright only after checking escalation signals
7. store the content hash, ETag, Last-Modified, and next crawl time

Look for JSON-LD first. Structured data often contains a title, price, currency, availability, or publication date in a more stable format than CSS selectors. Then use microdata, Open Graph, and only afterward page selectors. This order reduces breakage after redesigns.

Do not apply the same recrawl interval to every URL. A frequently changing product page needs a shorter cycle than an archived article. A simple rule is to shorten the interval after a content-hash change and gradually lengthen it after several unchanged crawls. For dynamic page parsing, also store the selector version so a parser failure is distinguishable from genuinely missing data.

Use If-None-Match with ETag or If-Modified-Since for recrawl requests when the server supports these headers. A 304 Not Modified response saves traffic and parsing CPU. It also must not overwrite the last successful record with an empty body.

After a lease expires, a job can be issued again, so result writes must be idempotent. Use a unique URL key and an upsert operation to avoid duplicates.

How a Web Crawler Differs from a Web Scraper and Browser Automation

A web crawler discovers and schedules URLs, a web scraper extracts fields from pages, and browser automation reproduces actions in a user interface. One project can contain all three, but mixing them in one class makes testing and scaling harder.

For a server-rendered catalog, aiohttp and a parser are enough. For an SPA where the price arrives through XHR, check the network API first. Keep Playwright for cases that require rendering, cookies, a session, or page interaction. A browser needs more memory, CPU, and startup time than an HTTP client.

HTTP-first architecture has another advantage: parsing code does not care where the HTML came from. In tests, you can replace the fetcher with a local fixture, validate selectors without a network, and reproduce an error a month later.

How to Find Bottlenecks in Traffic, Memory, and CPU

The slowest resource defines the concurrency limit: website limits, connection pool, network bandwidth, browser RAM, parser CPU, or database write speed. Do not guess the worker count. Select it through a series of short measurements on an authorized test set.

Start with 1, 2, 4, and 8 workers. For each run, measure pages/s, p50 and p95 response times, error rate, RSS for the whole process tree, CPU, and queue size. Add the next concurrency level only when throughput rises meaningfully while errors and p95 remain acceptable.

For a conservative first estimate on one host, use concurrency ≈ target_requests_per_second × p95_latency_seconds. This is guidance rather than an exact calculation: p95 is the 95th-percentile latency, while estimating average simultaneous requests requires average latency.

If a site permits one request per second and p95 is three seconds, use three concurrent requests as an initial test point, not a guaranteed sufficient concurrency level. Limit request launch rate separately from concurrent connections and verify the outcome through measurements.

Write metrics to structured logs:

• queued and active for frontier state
• ok, retry, blocked, and parse_error for outcomes
• http_seconds, browser_seconds, and parse_seconds for stages
• rss_mb and cpu_percent for process resources
• records_written and write_seconds for storage

If the parser blocks the event loop for long periods, move CPU-intensive processing to ProcessPoolExecutor. Threads help only when the library releases the GIL. Measure it: process the same HTML set in the event loop, a thread pool, and a process pool.

If Chromium memory usage grows, limit simultaneous pages, block unnecessary images, fonts, and video, and reuse a context within a compatible session. Do not keep hundreds of pages open. When collecting data through proxies, count traffic separately because a browser may download resources absent from an HTTP-first workflow.

Start diagnosis by reading graphs, not by making assumptions about a library. Rising queued with stable active means the fetcher or host limits cannot keep up. A growing writer queue with an empty frontier points to the database. Gradually increasing RSS after every completed page may indicate unclosed resources, although caching or Chromium memory management can produce a similar pattern.

The table suggests a possible cause, but only a repeated test with one changed parameter can confirm it. If you increase workers, replace the parser, and block images simultaneously, the source of the improvement remains unknown.

After every test, store the configuration with the results: Python version, Playwright version, machine type, worker count, and URL set. Otherwise, numbers from two runs cannot be compared honestly.

How to Organize Continuous Crawling and Browser Sessions with Afina

Continuous crawling requires checkpoints, stable browser environments, and clear ownership for every host. Store the frontier and statuses in a database, and never place secrets, proxies, or session data in code or logs.

When some permitted sources require a persistent browser session, Afina profiles can serve as a separate infrastructure layer for browser tasks, proxies, and repeatable workflows. The crawler remains responsible for the queue, robots.txt, limits, and result storage. This material is provided for informational and educational purposes only.

Do not send an entire sitemap to the browser layer. Run the inexpensive HTTP pass first and route only URLs with a confirmed JavaScript or persistent-session requirement into a profile. This makes costs easier to predict and failures easier to diagnose.