Afina

Download app

AppleWindows
EN
BlogGuides and Tutorials

November 19, 2026

How to Avoid Blocking During Web Scraping in 2026

Controlled web scraping without overloading websites

Reducing blocking during web scraping requires controlling request frequency, retries, caching, and session state according to the target site's rules. Workflow automation is useful for repetitive browser operations, but it does not replace permission to access a resource or override its restrictions.

To reduce the risk of blocking, first check robots.txt, the site's terms of use, and the availability of an official API. Only then should you assess whether IP rotation is necessary. If no API exists, the scraper should limit concurrency, respect Retry-After, cache unchanged pages, and stop after an explicit refusal. For managed task execution through REST, you can separately explore the Afina API without confusing browser automation with a data collection service API.

A 429 status usually indicates that a limit has been exceeded, while 403 may mean insufficient permissions, unsuitable headers, a broken session, or an access policy. They should not be handled with the same retry logic. When a task requires signing in to an authorized account, separate browser profiles help prevent cookies and local storage from mixing between sessions.

Principles That Reduce Blocking

A stable scraper maintains a predictable load and stops automatically when access restrictions appear. Its basic loop includes a URL queue, a per-domain request limit, cache storage, a response log, and a retry mechanism. If one component is missing, errors multiply quickly: retries create additional load, while duplicates consume quota without providing value.

Define the process for a new source before scaling it:

  1. check the access rules and official API
  2. run a small sample with one concurrent request
  3. save statuses, headers, and a control fragment of each response
  4. configure exponential backoff with a randomized delay
  5. add caching and URL deduplication
  6. increase concurrency only after a stable control run

Request limits should follow the rules of the specific resource. They may apply to a domain, an individual API route, an IP address, an account, an API key, or a group of routes. If the exact boundaries are unknown, begin with minimum concurrency and adjust it based on server responses. After a 429 response, return the request to the queue until the time specified by Retry-After instead of launching it immediately. Backoff intervals such as 2, 4, 8, and 16 seconds with jitter are appropriate for network errors. Persistent 401, 403, or validation errors should not be retried without changing the conditions.

Caching reduces repeated requests. For a catalog, it may be enough to store a response under a key composed of the URL and parameters, with a lifetime based on how frequently the data changes. ETag with If-None-Match, or Last-Modified with If-Modified-Since, can return a 304 response without transferring the document again when the server supports conditional requests. A 304 response lets you reuse cached data, although the request itself may still count toward server limits.

Session consistency matters in authorized sign-in scenarios. Within one authenticated session, preserve a consistent cookie store and the connection parameters required by the application. When necessary, keep the User-Agent, language, time zone, and network exit stable. Rotating every parameter for every URL often looks less natural than calm, sequential activity.

How to Diagnose 403 and 429 Without Excessive Retries

Treat 429 as a signal to slow down. Read Retry-After, reduce concurrent requests to the affected domain, and place the URL back in the queue for the specified period. If the header is missing, use exponential backoff with an upper bound and a retry limit. Unlimited retries sharply increase server load.

A 403 response requires a different check: it indicates denied access, but not necessarily that the scraper has been blocked. The cause may be user permissions, network restrictions, server configuration, or an automated access policy. Compare the URL, method, authorization, cookies, headers, and response body with an authorized control request. Confirm that you are seeing a genuine denial page rather than a routing error or expired token. If the resource explicitly prohibits automated access or displays a challenge, stop the task and use the official API or arrange authorized access.

Diagnostic flow for HTTP 403 and 429

An error log should contain the time, domain, route, status, Retry-After, session identifier, number of previous attempts, and a hash of the response body. Do not store secrets, personal data, or complete cookies in logs. A hash can group identical blocking pages without retaining their contents.

Criteria for Comparing Data Collection Services

The services in this selection perform different roles. A scraping API accepts a URL and returns HTML or structured data. A proxy provider supplies network exits, but retry and parsing logic stays in your code. A managed pipeline handles development, maintenance, and dataset delivery.

CriterionWhat to Check Before Paying
request limitsconcurrency limits, behavior after exceeding them, and support for separate queues
delays and retrieswhich statuses are retried, the maximum number of attempts, and whether failed requests are billed
cachingresult storage, deduplication, TTL, and conditional requests
official APIsspecialized API endpoints for the required source and permitted usage formats
403 and 429availability of logs, target-site statuses, and credit-spend diagnostics
session consistencycookies, sticky sessions, geolocation, browser sessions, and state lifetime

The prices below were checked on October 30, 2026. Comparing only the monthly fee is insufficient: a standard request may cost one credit, while JavaScript, a premium proxy, or a difficult domain can multiply the cost.

Comparison of Service Roles and Entry Conditions

The choice depends on how much infrastructure the team is ready to maintain. The table does not rank services from best to worst. It shows different product models and the lowest entry point available on the review date.

ServiceMain ModelEntry ConditionBest For
ScrapingBeescraping APIfrom $19/monthfast API access with a managed browser
ScrapflyAPI and Cloud Browser1,000 free creditscombining HTTP and CDP sessions
Bright DataWeb Scraper API5,000 free recordsready-made scrapers and large batches
ScraperAPIAPI, crawler, pipelinefrom $49/monthsynchronous and asynchronous tasks
ZenRowsshared credit systemfree, then from $16/monthFetch, Batch, and Browser Sessions
Roundproxiesproxiesfrom $3.50 per GB with a discounta custom scraper and IP control
CrawlbaseAPI, crawler, storagepay as you go without a subscriptionone balance for multiple components
PromptCloudmanaged servicecustom quoterecurring enterprise data flows
Scrapingdogscraping APIfree, then from $40/monthspecialized API endpoints
OxylabsWeb API and proxiesfrom $30/monthscaling from an API to a proxy stack

Do not choose a plan solely by the number of available credits. First take 100 representative URLs, divide them into static, JavaScript, and authenticated pages, and calculate the actual consumption for each group. One million credits may mean one million simple pages or only forty thousand requests with JavaScript and a premium proxy.

ScrapingBee for API Requests With a Managed Browser

ScrapingBee dashboard for launching API requests

ScrapingBee combines an HTML API, JavaScript rendering, proxies, geotargeting, screenshots, and data extraction rules. Auto-Mode can select the least expensive working combination of rendering and proxies, while max_cost limits the cost of an individual request. This helps with sources of varying difficulty, where a fixed mode either costs too much or frequently returns an error.

The Hobby plan starts at $19 per month for 75,000 credits and 25 concurrent requests. The free trial includes 1,000 credits. A standard request starts at one credit, JavaScript costs five, a premium proxy without JavaScript costs ten, and combining both modes costs twenty-five.

The service suits developers who need one HTTP API endpoint instead of maintaining their own browser fleet. A built-in Afina integration has not been confirmed. A practical adjacent workflow is to let ScrapingBee collect data through its API while Afina runs authorized browser steps in separate profiles when a manual result check is required.

Scrapfly for APIs, CDP Sessions, and Structured Extraction

Scrapfly platform for managing web requests and browsers

Scrapfly provides a Web Scraping API, Cloud Browser through CDP, Screenshot API, Extraction API, and crawler under one key. Cloud Browser can connect to Playwright or Puppeteer as a remote Chromium instance. This mode is useful for pages with clicks, forms, and long sessions, while a regular API request is less expensive for static HTML.

After registration, 1,000 free credits are available without a card. Billing is credit-based, while custom contracts for workloads above 6 million monthly credits start at $1,200 per month. Before budgeting, check the multipliers for proxies, browser use, and extraction because the base credit price does not reveal the cost of a specific source.

Scrapfly fits teams that want to combine HTTP collection and managed CDP sessions. No direct integration with Afina has been confirmed. It is better to separate their roles: Scrapfly handles server-side data retrieval, while Afina retains a separate browser context for authorized operations that cannot be moved to an API.

Bright Data for Ready-Made Scrapers and Large Datasets

Bright Data catalog with web scraping tools

Bright Data offers a Web Scraper API with templates for popular sources, browser rendering, proxy management, batch jobs, and delivery in JSON or CSV. Charging for successfully delivered records simplifies calculations for structured API endpoints, but the billing unit becomes a record rather than the original HTTP request.

The free tier includes 5,000 records per month. Pay as you go for the Web Scraper API starts at $1.50 per 1,000 records, while Scale starts at $499 per month and includes 384,000 records. For tasks where one document produces many records, this may be more cost-effective than a credit model. For pages with few fields, calculate both options.

The service suits large catalogs, price monitoring, and recurring batch jobs. A built-in Afina integration has not been confirmed. An adjacent process can use independent tasks: Bright Data creates the dataset, and Afina is used only where the operator has a lawful basis to work with a browser session.

ScraperAPI for Synchronous and Asynchronous Collection

ScraperAPI interface with web request parameters

ScraperAPI supports synchronous requests, asynchronous batches, JavaScript rendering, premium proxies, custom sessions, and specialized API endpoints with structured data. Asynchronous mode is convenient for large URL lists: the client does not keep the connection open and retrieves the result after the task finishes.

The Hobby plan starts at $49 per month for 100,000 credits and 20 threads. A free seven-day trial provides 5,000 credits without a card. Request cost depends on the domain and parameters: a standard page consumes one credit, while difficult sources and anti-bot processing have their own multipliers. Set max_cost before a large run.

ScraperAPI suits teams that need an API, crawler, and DataPipeline in one dashboard. No dedicated Afina integration has been confirmed. If a separate browser stage is required, do not move cookies or tokens between systems unnecessarily: API jobs and an Afina profile have distinct session boundaries.

ZenRows for Credit-Based Pricing and Browser Sessions

ZenRows dashboard for configuring website data retrieval

ZenRows combines Fetch, Extract, Crawl, Batch, and Browser Sessions. All components use a shared credit balance. A standard successful page costs one credit, JavaScript rendering costs five, a premium proxy ten, and both options together twenty-five. Failed requests are not charged, while 404 and 410 count as valid responses.

The free plan includes 5,000 credits each month. The paid Build plan starts at $16 per month for 45,000 credits and 20 concurrent requests, while Launch costs $58 for 250,000 credits. Browser Sessions additionally consume five credits per minute plus traffic, so long interactive workflows need a separate estimate.

ZenRows is convenient for teams that want to move from basic page retrieval to batches and an interactive browser session without changing providers. No native Afina integration has been confirmed. In an adjacent workflow, ZenRows handles server-side data while Afina preserves the isolated browser-profile state for authorized manual operations.

Roundproxies for Managing the Network Exit

Roundproxies dashboard with scraping proxy plans

Roundproxies differs from a scraping API: it provides residential, datacenter, ISP, and mobile proxies but does not write a parser or manage the URL queue. Request-rate limits, progressive retry delays, caching, analysis of 403 and 429 responses, and response completeness checks remain in your application. The proxy solves only the network layer.

The residential package starts at $3.50 per GB with a first-purchase discount, while the regular price is $7 per GB. HTTP, HTTPS, and SOCKS5 are available, along with automatic rotation and sticky sessions. An authenticated session should retain its IP for the session lifetime instead of changing the exit between consecutive pages.

Roundproxies suits teams with their own scraper that need control over geolocation and sessions. A proxy can be added to an individual Afina profile as standard connection credentials. This is ordinary protocol-level compatibility, not a confirmed built-in integration between the products.

Crawlbase for One API and Crawler Balance

Crawlbase platform for launching a managed crawler

Crawlbase uses one balance for the Crawling API, Crawler, Smart AI Proxy, screenshots, and Cloud Storage. A standard successful request equals one credit, JavaScript doubles consumption, a screenshot costs two credits, and storing a page for one month consumes half a credit. Domain difficulty adds a multiplier.

Pay as you go has no subscription fee and uses tiered rates from $3 to $0.02 per 1,000 successful requests depending on volume. The Starter subscription begins at $99 per month for 200,000 credits. A new account receives 1,000 credits, and the balance may increase to 5,000 after completing setup steps.

Crawlbase suits teams that want to fund several components from one balance and set a strict spending limit. No direct integration with Afina has been confirmed. The API and browser profile should be orchestrated as two independent stages rather than presented as a built-in connector.

PromptCloud for Fully Managed Data Flows

PromptCloud page describing managed data pipelines

PromptCloud operates as a managed service rather than a self-service API for a single URL. The provider's team designs the crawler, maintains extraction logic, validates the schema, and delivers structured data through an API, S3, FTP, or another agreed channel. This reduces internal maintenance but provides less low-level control.

There are no fixed plans. Pricing is calculated individually based on the number of sources, record volume, update frequency, protection complexity, delivery format, and schema. This model is appropriate for recurring flows from ten or more diverse sources. A one-time collection of several URLs generally does not fit the service's positioning.

PromptCloud suits companies that need an SLA, human quality control, and a ready dataset. A built-in Afina integration has not been confirmed. Results can be delivered to corporate storage while authorized browser operations run separately, without a shared session between the platforms.

Scrapingdog for Specialized Scraping APIs

Scrapingdog dashboard with available APIs and credits

Scrapingdog provides a general Web Scraping API and specialized API endpoints for search, maps, e-commerce, and other common sources. Plans unlock all APIs and differ by credit volume and concurrency. Credits for failed, blocked, and timed-out requests are returned to the balance, which simplifies cost control.

The free tier includes 200 credits and one concurrent request. Lite starts at $40 per month for 200,000 credits and five concurrent requests, while Standard costs $90 for 1 million credits. A request with geolocation or JavaScript may consume more credits, so translate the monthly quota into the actual number of pages of your type.

Scrapingdog suits small products and teams that need ready API endpoints without their own proxy manager. No confirmed Afina integration exists. If a task includes both API collection and an authorized browser stage, pass only the required data between them rather than the complete cookie store.

Oxylabs for Web APIs and Proxies Under One Contract

Oxylabs catalog with web APIs and proxy products

Oxylabs combines Web API, Web Unblocker, residential, mobile, ISP, and datacenter proxies. Web API provides separate API endpoints for search and page retrieval, supports ready-made parsers, and delivers structured results. This lets a team purchase a managed API or keep extraction in its own code and use only proxies.

New users receive $5 in welcome credits. The Starter subscription begins at $30 per month. For the /Scrape API endpoint, rates start at $0.50 per 1,000 results on lower plans, while residential proxies begin at $6 per GB. Enterprise plans for large volumes start at $2,500 per month. Do not confuse the price of an API result with the cost of proxy traffic.

Oxylabs suits companies that want to scale from one API endpoint to separate proxy infrastructure. A built-in Afina integration has not been confirmed. Standard proxy credentials can be configured in a profile, while Web API remains an independent server-side collection channel.

How to Choose a Service for Your Team's Workload

For a small prototype, choose a service with free credits, transparent rules for JavaScript charges, and the ability to limit concurrent requests. At this stage, the goal is not maximum speed but determining the cost of processing one useful page and the share of retries.

A proxy provider may be more economical for a team with its own crawler and experience maintaining queues. The team then remains responsible for retry delays, caching, sessions, parsing, and monitoring. If development and infrastructure resources are limited, a scraping API can transfer part of that work to the provider. A managed service makes sense when the business needs an SLA-backed data flow rather than control over every request.

Run a short check before paying:

  1. calculate the number of unique URLs and required update frequency
  2. determine the share of pages requiring JavaScript and authorization
  3. estimate credit consumption on a control sample
  4. test concurrency, queue behavior, and handling after 429
  5. measure the share of complete and correctly parsed responses
  6. set a budget limit before scaling

The most important metric is not the monthly plan price but the cost of a valid record after retries, rendering, proxies, and parsing. This figure allows an objective comparison of different pricing models.

How to Use Afina in an Adjacent Workflow

Afina can serve as a separate browser layer for authorized tasks that require profiles, cookies, proxies, and a controlled manual or automated session. The scraping API operates through its own server-side key. Data moves between stages through a queue, file, or database rather than shared cookies.

This separation does not make Afina a built-in integration with every service mentioned. It only separates server-side collection from browser work so that a failure in one loop does not mix sessions in the other. This material is provided for informational and educational purposes only.

Before launching, check the target site's rules, the lawful basis for processing data, available APIs, and provider restrictions. A browser profile is not permission to collect data and does not override request limits.

Download

FAQ — Frequently Asked Questions

How can you avoid 429 during web scraping?

Reduce concurrency, follow Retry-After, and add exponential backoff with jitter. Retry a request only a limited number of times.

Why does a scraper receive 403?

The cause may be missing permissions, an expired session, an incorrect route, or an automated access policy. Check the response and resource rules before retrying automatically.

Which service suits a small team?

A small team may prefer an API with a free trial, transparent credits, and a budget limit. Choose only after testing a control sample of your own URLs.

What determines the total cost of scraping?

Cost depends on page volume, JavaScript rendering, proxy type, update frequency, retries, and parsing complexity. A plan's monthly price does not show the cost of a valid record.

Is IP rotation required for every request?

No. An authorized session often needs a stable IP together with one cookie store. Rotation should match the access model and comply with site rules.

Can caching be used during data collection?

Yes. Caching reduces repeated requests and load. Align the TTL with the source's actual update frequency.

How do data collection services work with Afina Browser?

No built-in integration has been confirmed for the APIs mentioned. They can run in a separate server-side loop while Afina is reserved for an authorized browser session.

What should be checked before scaling a scraper?

Check access rules, actual credit consumption, concurrency, reactions to 403 and 429, and data completeness. Then set budget and speed limits.

Related terms

Continue reading onWeb scraping automation — data processing | Afina Browser
Artem Vishnepolskyі

Artem Vyshnepolskyi is a drop hunting and Web3 automation specialist, active in the crypto industry since 2021 and a member of the Afina team. He focuses on systematic participation in testnets, campaigns, and retrodrop activities, with notable life-changing cases including Starknet, Movement, and Initia.

At Afina, he works as a Support Specialist, helping users implement automation solutions and adapt tools to their specific goals. With a humanities background and no formal technical education, Artem proves that effective Web3 automation is accessible even to non-technical users