Scrape-LE
Check whether a page is scrapeable before the scraper is written, and say when it cannot tell
No editor in the loop? npx -y scrape-le-mcp runs the same engine as an MCP server, so an agent can call analyze_robots_txt with nothing else installed, and cargo install scrape-le puts the same check in a terminal. Every way to install it.
Overview
What Scrape-LE does
Whether a page can be scraped is a question best answered before the scraper is written. Scrape-LE loads a URL in a real headless Chromium and reports what it found: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections covering anti-bot measures, authentication requirements, rate limiting and robots.txt rules.
What people use it for
Before you write the scraper
Find out whether a page is reachable, gated or defended — before the work, not after.
Real browser, real answer
The page loads in headless Chromium, so client-rendered content and bot checks behave as they actually will.
Respect robots.txt
The origin rules are fetched and reported alongside everything else, evaluated as your crawler when you name one. A scrapeability report is information, not permission.
Palette
5 commands
Open the command palette and type the name. Nothing is bound to a shortcut by default — the editor's keymap is the user's, not ours.
- Check URL Scrapeability
scrape-le.checkUrl - Check Selected URL
scrape-le.checkSelection - Setup Browser
scrape-le.setup - Open Settings
scrape-le.openSettings - Help & Troubleshooting
scrape-le.help
Install
Install Scrape-LE
VS Code
ext install nolindnaidoo.scrape-leOpen Quick Open (Cmd/Ctrl+P) and paste.
Cursor / Windsurf
cursor --install-extension nolindnaidoo.scrape-leCursor, Windsurf, VSCodium and the other VS Code forks pull from Open VSX. The id is the same as on the Marketplace.
AI agents
npx -y scrape-le-mcpRuns Scrape-LE's engine as an MCP server, so an agent can call analyze_robots_txt with no editor involved. VS Code 1.101+ needs nothing — the extension registers it for you.
Channels
Where it ships
One engine, 5 places to get it. Copy the one for the editor you use.
Links
Where Scrape-LE lives
- VS Code Marketplacescrape-le
- Open VSXfor Cursor, Windsurf and VSCodium
- npmscrape-le-mcp
- GitHubsource, issues, changelog
- MCP registryio.github.nolindnaidoo/scrape-le
The family
The other 16
String-LEExtract every string in a codebase, so a person can read them
Numbers-LEExtract every hardcoded number in a codebase, so a person can check them
Paths-LEExtract every file path in a codebase, and say whether it still points at anything
Colors-LEExtract every color in a codebase, and say which ones are not in your palette
URLs-LEExtract every URL in a codebase, with its protocol and exact position
Dates-LEExtract every date and timestamp, and the exact instant each one resolves to
Units-LEExtract every quantity with its unit, normalized, and refuse the ambiguous ones by name
IDs-LEExtract every UUID, ULID, NanoID, ObjectId and Snowflake, and decode the time inside
IPs-LEExtract every IP address, CIDR block and MAC, normalized and classified by scope
Regex-LEFind every regex in a codebase, and report which can be driven into catastrophic backtracking
Versions-LEFind where one dependency is constrained differently across a repository's manifests
i18n-LEIdentify the i18n library a project uses, then audit its catalogs by that library's rules
DecisionLintLint Jev and Luna questions before they are sent: the questions you write for TypeSafe's Jev, OpenAI's Decisions API (gpt-6-luna) or any platform that serves one, as you type, with no API key and no network calls. An optional command asks the model itself to check a file
Secrets-LEFind hardcoded credentials in a codebase, and never print one into the report
EnvSync-LECompare the dotenv files in a tree, and say which keys are missing from which
Unicode-LEFind the Unicode that hides meaning — bidi controls, invisibles, homoglyphs, mixed scripts