How to protect a web scraping API from free credit abuse
Free scraping credits cost real bandwidth and slow paying developers down. See the arithmetic, the basic fixes that break and what to set up this week.
$4,800 of residential proxy bandwidth went out last month to accounts that will never pay you. That's a scenario, and every number in this post is an assumption to replace with your own.
Say you run a scraping API and new accounts get 10,000 free credits. A residential request with a rendered browser costs 25 credits. Each page pulls 2.5 MB through the proxy, and your supplier charges you $4 per GB.
10,000 credits ÷ 25 = 400 pages
400 pages × 2.5 MB = 1,000 MB, or 1 GB
1 GB × $4 = $4 per free accountFour dollars looks harmless. Then 15 people open 80 accounts each, and that's what web scraping API free credits abuse adds up to: 1,200 accounts, 1,200 GB and $4,800.
You hear about it from your biggest customer, who pays $299 a month to send competitor prices to a dozen retailers every morning: "Last night's run timed out and our clients got an empty feed. Is this us or you?"
The logs show why. From 3:20 to 4:13 a.m., 80 API keys you've never looked at held your browser pool, each at its limit of 5 concurrent requests. Your rate limiter stayed quiet because no key broke a rule. You have 60 browsers, and a page takes 6 seconds.
80 keys × 5 concurrent = 400 requests in flight
400 − 60 browsers = 340 waiting
60 ÷ 6 seconds = 10 pages a second
340 ÷ 10 + 6 = 40 seconds for one pageYour customer's client gives up at 30 seconds, and its retries went to the back of that same line. This illustrative chart shows the wait growing with free accounts. The line is that 30-second timeout.
One person's 80 accounts hold 32,000 pages (80 × 400), about 53 minutes of your whole pool at 10 a second. All 15 people come to 480,000 pages, so for more than 13 hours a month (480,000 ÷ 10 ÷ 3,600) your paying developers are standing behind someone who'll never buy.
Your 300 customers average $49, which brings in $14,700 a month. Their own traffic uses 950 GB, or $3,800. Covering the free accounts' $4,800 takes 98 of those customers (98 × $49 = $4,802), so about one customer in three is paying for people who won't.
And then there's what you signed. Two months ago the bandwidth chart looked like growth, so you committed to 12 months at 2,000 GB a month, $8,000. This month you used 2,150 GB, so the bill is $8,600 with 150 GB of overage. Say you shut down every one of those accounts today. You'd need 950 GB, or $3,800, and you'd still owe $8,000 for each of the 10 months left. That's $4,200 a month for bandwidth nobody uses, and $42,000 before the contract ends.
After proxies, $6,100 ($14,700 − $8,600) is left to pay for browsers, support and you. The $299 customer is comparing providers, and that's $3,588 a year if they go. So you look at the free credits, which is how almost all of your customers started. Cut them to 1,000 and an honest developer gets 40 pages, which may not show whether you can reach the one site they care about. The people running 80 accounts just open 800.
Why scraping APIs lose so much to free credits
In most products a free credit buys a sample. On a scraping API it buys finished output, the same thing paying customers buy, and 32,000 pages of retail listings are a dataset. The people who want that dataset are well equipped to open accounts in volume, because automating websites is what they do all day.
Concurrency, rate limits and credit balances are counted per key, while your browsers and your supplier's bill are counted across everyone. That's why the first fix people reach for, a lower limit per key, doesn't hold. A limit of 5 across 80 keys is a limit of 400, as it is on any API with a free tier.
The costs also carry on after the accounts go quiet. A retailer that gets 32,000 requests in an hour starts blocking the addresses they came from, and your paying customers use those addresses too. Say half your paying traffic, 475 GB, goes to sites like that, and success there falls from 96% to 80%. You only charge for successful pages, so the retries are yours: 475 GB × (96 ÷ 80) = 570 GB. That's 95 extra GB, or $380 a month for as long as the blocks last.
Here's the same illustrative scenario, from one night to the end of the contract.
The basic fixes, and how each one breaks here
Each of these catches something real. A unique-email rule compares strings, while Gmail ignores dots and plus-tagged mail reaches the same inbox, so one mailbox can look like many addresses. That's simply how email works.
| Basic fix | What it catches | What it misses | Who it wrongly blocks | What it costs to keep working |
|---|---|---|---|---|
| Email verification | Typos and dead addresses | Any second real inbox | Developers whose code lands in spam | Sending and deliverability work |
| Unique emails | The same address twice | Every different address | Almost nobody | Very little |
| Disposable-domain blocks | Known throwaway services | New domains and ordinary mailboxes | Developers using privacy forwarding | A list someone updates forever |
| Gmail dots and plus addressing | One Gmail inbox written several ways | A second real mailbox | People who sort their mail with plus addresses | Upkeep as mail services change |
| CAPTCHA at signup | Simple signup scripts | Accounts opened by hand | People using screen readers | Fees and lost signups |
| Phone or card requirement | Casual second accounts | Anyone with another number or card | Developers testing before they commit | A fee per text, fewer people trying your API |
| One signup per IP | Repeat signups on one connection | Any other network | Offices and coworking spaces | Exceptions and tickets |
| VPN blocks | Known VPN ranges at signup | Everything off your list | Developers on company VPNs | Another list to maintain |
| Rate and concurrency limits per key | One key overrunning | 80 keys under the limit | Real users in a busy hour | Tuning per endpoint |
| Usage caps per account | One account overspending | Each new account's fresh cap | Anyone mid-job at the cap | Deciding what a half-finished job returns |
| Smaller grants or expiring credits | Some cost per account | More accounts make up the difference | Every honest tester | Fewer people reach paying |
| Manual review | What a person can spot | Anything after hours | Whoever looks odd to a tired reviewer | 1,200 accounts × 1 minute = 20 hours |
IP rules are weaker here than on a consumer app, because your honest customers call you from cloud servers and shared offices. Blocking VPNs at signup mostly lands on developers at work. A CAPTCHA has its limits too, and your signup page attracts people who get past them for a living.
Stack the lot and a developer faces a puzzle, a phone number and a card before sending one request. What you'd really like to know is whether account 80 is the same person as accounts 1 to 79. Nothing in that table says, and a reviewer sees a farmed account and a real trial sending the same request.
What to set up this week
These cap the damage no matter who's behind the accounts, which rate limits alone won't do.
Give free traffic its own browsers. Set aside 10 of your 60 for free accounts, with their own queue. Paying customers keep 50, about 8 pages a second, and a flood of free jobs waits in its own line. Real testers wait longer during a flood.
Split residential out of the free grant. Keep 10,000 credits for datacenter requests and add a separate 40 residential pages. That's 100 MB, or $0.40. The same 1,200 accounts would cost $480 in residential bandwidth instead of $4,800, though some people will open more accounts to make up the difference.
Set one daily bandwidth budget for all free traffic. Say 10 GB, which is $40 a day and $1,200 over 30 days. One person with 100 accounts can still spend the day's budget before your real testers wake up.
Cap free requests per target site across the whole free pool, for example 500 an hour. One job can't put 32,000 requests on one retailer.
Forecast from paying traffic only. Take free GB out before you sign a commitment. Here that's 950 GB, a long way from 2,150.
Deciding which free credit claims go through
Every step above limits what a free claim can cost you. None of them helps you decide whether a given claim for 10,000 credits should go through at all.
AbuseZero exists for that moment. There's also a proprietary algorithm, with AI that checks the claim again. Creating a fresh account isn't enough to erase previous activity. With multi-layer AI risk analysis, AbuseZero examines 140,000+ signals, detects linked claims and flags abuse that single checks miss, and single checks are all that table has. AbuseZero recognizes repeat devices across new accounts, even when email addresses keep changing. It checks mailbox aliases, disposable domains and mail routing, and it sees beyond an IP address with VPN, hosting, network reputation, Tor and proxy intelligence, so several rows of that table stop being your team's job. Your side is one API call before the credits go out, with a clear allow or deny and rules you control. You don't have to build your own abuse prevention system.
Your application still runs the free pool, the budget and the per-key limits. Observe mode shows you what would be denied before anyone is blocked, so you can watch a week of claims before one developer is turned away. The quickstart walks you through your first benefit.
Frequently asked questions
How many free credits should a web scraping API give?
Enough for one developer to test the hardest site they care about. In this scenario that's 40 residential pages for $0.40. See setting limits that don't invite extra accounts.
Is every developer with two accounts abusing free credits?
No. People keep work and personal accounts, and freelancers keep one per client. Write down what you meant to give before you enforce anything.
Can I stop repeat claims without adding steps to signup?
Yes. Move the decision to the moment credits are granted. AbuseZero's evaluation returns allow or deny there, and an honest developer sees no extra step.
Keep your bandwidth for paying developers
Start with Builder
Pay as you go. Evaluate your first claim today.
See pricing
Builder, Pro, Scale and Enterprise.
Talk to us about Enterprise
High volumes and custom terms.