Open-source scraper defence

Make abusive scraping pointless.

BotLocker is a reverse proxy that spots scrapers and lures them into an endless maze of generated pages. They burn their own time and bandwidth, while your real content and your real visitors are never touched.

Quick start See how it works
  • Zero harm to humans
  • Honours robots.txt
  • Runs anywhere: binary or nginx

Manifest

Why BotLocker exists

Bots take what we publish without consent, without credit, and without compensation. They burn bandwidth, inflate costs, and degrade the experience for the people we build for. We are tired of it, so BotLocker makes abusive scraping expensive and pointless.

Mission

Make abusive scraping costly and pointless, and protect the content creators have made.

Goals

  • Protect real users, who never see a trap
  • Protect good bots such as search engines
  • Impose cost on the bots that ignore the rules
  • Stay simple to deploy, at the edge or locally
  • Stay open and transparent

Principles

  • Trap only on clear signals, never on a hunch
  • Serve only harmless, generated filler
  • Allowlist verified crawlers first
  • Keep a kill switch and hard caps

Values

01

Consent over extraction

Our content is ours to offer, not to take.

02

Defense, not offense

We only waste the time of bots that ignore our rules. We never attack third parties.

03

Zero harm to humans

A misclassified visitor always has a way out, and is never served anything harmful.

04

Simplicity

Small, understandable parts beat clever ones.

05

Transparency

We honour robots.txt and document what we do, in plain words.

How it works

Every request takes the same path

BotLocker sits in front of your site. Each request is classified before it reaches your origin.

  1. 1

    Verified crawler?

    Search engines and other allowlisted good bots pass straight through.

  2. 2

    Honeypot?

    A request under the maze path is always trapped. No human reaches it.

  3. 3

    Score

    Weak signals add up: headers, user agent, behaviour, network, and more.

  4. 4

    Decide

    Low scores pass. Mid scores get a challenge. High scores from two or more signal families go to the maze.

The maze

Each maze page is generated from a keyed hash of its path, so the same path always gives the same page and nothing has to be stored. Every page is filler text with links deeper into the maze, so it never ends. Maze pages are marked noindex, and robots.txt disallows the path so compliant crawlers never enter it.

The challenge

  1. The browser gets a short challenge page, not your site.
  2. The page reports browser signals with a short-lived token.
  3. Clean browsers get a signed pass cookie, bound to their IP.
  4. Automation signals trap the client for the same period, even with a cookie.

Clients that don't run the page never get a pass.

Detection

Many weak signals, one decision

No single signal is trusted alone. A score crosses a threshold only when several independent signal families agree. That keeps one noisy check from trapping a real person.

Headers

Browser headers

Missing Accept-Language or Accept, and Chrome user agents without fetch metadata.

Client

Known scrapers

User agents from common scraping libraries, and empty user agents.

Behaviour

Crawl pattern

Request rate per window, and many page views with no CSS, JS, or image loads.

Honeypot

Hidden link

A link no human sees. Any request to the maze path is a trap, with no score needed.

Browser

Browser report

WebDriver automation, headless Chrome, and zero-size windows, reported from the challenge page.

Needs data

Network and TLS

Hosting ASNs and TLS fingerprints. These need a data source, so they stay silent until you add one.

The trap rule

Score at or above trap_at (default 6.0) from two or more families means the maze. Score at or above challenge_at (default 3.0) means a challenge. Below that, the visitor passes.

Dashboard

See what the bots are doing

A local dashboard shows every connection, every captured bot, and why it was captured. It also lets you change settings, and the proxy reloads within seconds.

Captured

Every captured client, with the rule that caught it and its evidence.

Live feed

The most recent decisions, with the action and the reason for each.

Settings

Edit thresholds and limits. Each save is validated, audited, and applied without a restart.

Deploy

Two ways to run it

Both run the same Rust detection core, so the rules are identical.

Standalone binary

A single Rust binary that proxies your site and serves the dashboard on loopback.

cd backend
cargo build --release -p botlocker
cp botlocker.example.toml botlocker.toml
export BOTLOCKER_SECRET="$(openssl rand -hex 32)"
./target/release/botlocker --config botlocker.toml

nginx with Docker

An nginx build with the module compiled in, started with Compose. The dashboard is optional.

cd nginx
cp .env.example .env          # set ORIGIN_URL
mkdir -p secrets && openssl rand -hex 32 > secrets/botlocker_secret
docker compose -f docker-compose-nginx.yaml up -d --build

Put your robots.txt in nginx/robots/ to serve it, and keep the Disallow: /maze/ line.

Honest limits

What BotLocker can't do

BotLocker raises the cost of scraping. It does not promise perfect detection.

Stealth browsers

A browser that hides automation, or a headless one with realistic fingerprints, can pass the browser report.

Forged reports

The challenge report comes from the browser, so a client can send one without running the page. It raises the cost but doesn't prove a real browser.

Shared IPs

Visitors behind one NAT or CDN share a rate limit and a pass. Real client IP handling behind a load balancer is not built yet.

Some signals need data

ASN and TLS fingerprint signals need a data source. The nginx build has no ASN or fingerprint data.

Built for people who publish.

Your content is yours to offer. BotLocker makes taking it without asking a bad deal.

Get started Read the docs