API4:2023 — Unrestricted Resource Consumption

Slide 10 · Bulk Scraping

No rate limit. No pagination cap. The dataset is yours.

How an automated client drains an entire dataset from an unprotected API.

The Setup

An API endpoint returns a list of records with optional pagination. Without a rate limit, a client can call it as fast as the network allows. Without a maximum page size, a single request can return millions of records. Without authentication on public endpoints, no credential is needed at all.

An attacker writes a simple loop:

# Pseudocode: bulk scrape with no rate limiting page = 1 while True: response = GET /api/users?page={page}&limit=1000 if response.records is empty: break save(response.records) # write to disk page += 1 # no sleep needed — the API doesn't enforce any delay # Result: all users extracted, one page per request, as fast as the server responds # On a fast API: millions of records in hours

Why there's no alarm

Each individual request is valid. It hits the same endpoint a real user would. It uses a valid (or no) token. The response is correct data. The only anomaly is the rate — but without a rate limit defined, the server has no baseline to compare against. From the server's perspective, it's just a busy day.

The three controls that stop this

Rate limiting (max requests per minute per client), maximum page size (cap the records-per-page parameter), and — for sensitive data — authentication. Any one of these adds meaningful friction. All three together make bulk scraping economically unfeasible.

← Back Real incident: Venmo →