Everything you need to run Quarry yourself.

Overview

Quarry is a self-hosted web scraper and crawler built for AI agent context. Scrape pages to clean markdown, crawl entire sites, monitor for changes, and extract structured data. It runs on your own infrastructure, with zero third-party dependency.

Quick start

Clone the repo, set your Postgres password, and bring the stack up with Docker Compose.

git clone https://github.com/samuelorobosa/quarry
cd quarry
cp .env.example .env   # edit .env, set POSTGRES_PASSWORD at minimum
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Install script

Prefer a one-liner? The installer detects your OS, installs Docker if missing, clones the repo to /opt/quarry, generates a .env with random secrets, starts every service (including the Lightpanda browser worker for JS-rendered pages), and confirms the API is healthy before declaring success.

curl -fsSL https://raw.githubusercontent.com/samuelorobosa/quarry/main/install.sh | bash

POST /scrape

Fetch a single URL and return clean markdown.

curl -X POST http://localhost:3000/scrape \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com" }'

{
  "url": "https://example.com",
  "title": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in...",
  "scraped-at": "2026-06-20T10:00:00Z"
}

POST /crawl

Start an async multi-page crawl. Returns immediately with a job ID; the crawl runs in the background.

curl -X POST http://localhost:3000/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "max-depth": 3,
    "max-pages": 100,
    "include-patterns": ["/docs/*", "/blog/*"],
    "exclude-patterns": ["/legal/*"],
    "webhook-url": "https://your-app.com/webhooks/crawl-done"
  }'

Status lifecycle: queued → running → completed | failed. Page status values: scraped · not_found ·blocked · timeout · error.

curl http://localhost:3000/jobs/job_8f2a1c

POST /monitors

Save a crawl config as a recurring monitor. Re-crawls on schedule and fires a webhook only when content actually changes.

curl -X POST http://localhost:3000/monitors \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "frequency": "daily",
    "webhook-url": "https://your-app.com/webhooks/site-changed",
    "goal": "alert me when pricing changes"
  }'

frequency options: hourly · daily · weekly.goal is optional — when set, each diff is evaluated by your configured LLM, and only relevant changes fire the webhook.

GET    /monitors/:id   # status, frequency, last checked, last job
DELETE /monitors/:id   # stop and remove

POST /extract

Scrape a URL and extract structured fields using an LLM. RequiresLLM_PROVIDER and LLM_API_KEY to be configured. Fields not found on the page return null — the model will not hallucinate values.

curl -X POST http://localhost:3000/extract \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "schema": {
      "plan-name": "string",
      "monthly-price": "number",
      "features": "array of strings"
    }
  }'

Schema type hints: "string" · "number" · "boolean" · "array of strings".

GET /dashboard

Available at http://localhost:3000/dashboard once running.

GET /metrics

Prometheus metrics for everything the worker touches. Point your scraper at /metrics and go.

LLM providers

Only required for /extract and monitor goal filtering. Everything else works without it.

LLM_PROVIDER=openai
LLM_API_KEY=sk-...
LLM_MODEL=gpt-4o

LLM_PROVIDER=anthropic
LLM_API_KEY=sk-ant-...
LLM_MODEL=claude-sonnet-4-6

LLM_PROVIDER=openai
LLM_BASE_URL=http://localhost:11434/v1
LLM_MODEL=llama3.2
LLM_API_KEY=ollama

Environment variables

Services

All services run in Docker.

Enable the browser worker with: docker compose --profile browser up -d

Hardware sizing