CrawlOptions
Every field accepted by crawl() with type and default.
Required
| Option | Type | Description |
|---|---|---|
url | string | Seed URL where crawling starts |
Crawl control
| Option | Type | Default | Description |
|---|---|---|---|
depth | number | 1 | Max crawl depth from the seed |
maxPages | number | 20 | Max pages to discover (hard limit) |
scrape | boolean | false | Also scrape each discovered page |
delayMs | number | 1000 | Delay between requests (rate limiting) |
timeoutMs | number | - | Total crawl timeout |
URL filtering
| Option | Type | Default | Description |
|---|---|---|---|
includePatterns | string[] | [] | Regex - URL must match at least one |
excludePatterns | string[] | [] | Regex - URL must not match any |
Crawling is always same-domain (Reader does not follow external links). Use these patterns to further scope within a domain.
When scrape: true
These options only apply when scrape: true:
| Option | Type | Default | Description |
|---|---|---|---|
formats | Array<"markdown" | "html"> | ["markdown"] | Output formats |
scrapeConcurrency | number | 2 | Parallel scrapes during the crawl |
removeAds | boolean | true | Remove ad selectors |
removeBase64Images | boolean | true | Strip inline base64 images |
Proxy & misc
| Option | Type | Default | Description |
|---|---|---|---|
proxy | ProxyConfig | - | Single proxy for this crawl |
proxyTier | "standard" | "premium" | - | Pick a proxy from the configured pool |
userAgent | string | Chrome UA | Custom user agent |
verbose | boolean | false | Enable logging |
showChrome | boolean | false | Show browser window |
Example
await reader.crawl({
url: "https://docs.example.com",
depth: 3,
maxPages: 100,
scrape: true,
scrapeConcurrency: 3,
formats: ["markdown"],
includePatterns: ["^https://docs\\.example\\.com/(api|guides)/"],
excludePatterns: ["/changelog/", "/archive/"],
delayMs: 1500,
});