Scrape-LE
Check whether a page is scrapeable before the scraper is written, and say when it cannot tell
No editor in the loop? npx -y scrape-le-mcp runs the same engine as an MCP server, so an agent can call analyze_robots_txt with nothing else installed, and cargo install scrape-le puts the same check in a terminal. Every way to install it.
What Scrape-LE does
Whether a page can be scraped is a question best answered before the scraper is written. Scrape-LE loads a URL in a real headless Chromium and reports what it found: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections covering anti-bot measures, authentication requirements, rate limiting and robots.txt rules.
What people use it for
Before you write the scraper
Find out whether a page is reachable, gated or defended — before the work, not after.
Real browser, real answer
The page loads in headless Chromium, so client-rendered content and bot checks behave as they actually will.
Respect robots.txt
The origin rules are fetched and reported alongside everything else. A scrapeability report is information, not permission.
5 commands in the palette
Open the command palette and type the name. Nothing is bound to a shortcut by default — the editor's keymap is the user's, not ours.
scrape-le.checkUrlscrape-le.checkSelectionscrape-le.setupscrape-le.openSettingsscrape-le.helpInstall Scrape-LE
VS Code
ext install nolindnaidoo.scrape-leOpen Quick Open (Cmd/Ctrl+P) and paste.
Cursor / VSCodium
cursor --install-extension OffensiveEdge.scrape-leVS Code forks pull from Open VSX, where the namespace is OffensiveEdge.
Zed
npx -y scrape-le-mcpWorks in Zed today — add the command above as a custom MCP server from the agent panel, and analyze_robots_txt appears in its tool list. There is no one-click listing in Zed's extension registry yet.
AI agents
npx -y scrape-le-mcpRuns Scrape-LE's engine as an MCP server, so an agent can call analyze_robots_txt with no editor involved. VS Code 1.101+ needs nothing — the extension registers it for you.
Where it ships
One engine, 6 places to get it. The ids differ by registry — copy the one for the editor you use.
Zed
built from Rust in the tool reposcrape-leOpenno listing yet — this links Zed’s instructions for adding it by hand
Where Scrape-LE lives
- VS Code Marketplacescrape-le
- Open VSXfor Cursor and VSCodium
- npmscrape-le-mcp
- GitHubsource, issues, changelog
- MCP registryio.github.nolindnaidoo/scrape-le
The other 15
Extract every string in a codebase, with its position, so a person can read them
Extract every hardcoded number in a codebase, so a person can check them
Extract every file path in a codebase, and say whether it still points at anything
Extract every color in a codebase, and say which ones are not in your palette
Extract every URL in a codebase, with its protocol and exact position
Extract every date and timestamp, and the exact instant each one resolves to
Extract every quantity with its unit, normalized, and refuse the ambiguous ones by name
Extract every UUID, ULID, NanoID, ObjectId and Snowflake, and decode the time inside
Extract every IP address, CIDR block and MAC, normalized and classified by scope
Find every regex in a codebase, and report which can be driven into catastrophic backtracking
Find where one dependency is constrained differently across a repository's manifests
Identify the i18n library a project uses, then audit its catalogs by that library's rules
Find hardcoded credentials in a codebase, and never print one into the report
Compare the dotenv files in a tree, and say which keys are missing from which
Find the Unicode that hides meaning — bidi controls, invisibles, homoglyphs, mixed scripts