TheCrawler
Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support
What you can do with it
Share of each task's required capabilities it coversPrefer a ready-made agent?
An MCP server is a connector for your own assistant. These finished products do the job for you: answer questions from internal documents.Similar tools
Listed for the same task: answer questions from internal documentsReviews
No review yet. Used TheCrawler? Be the first to help others choose.What it does
TheCrawler is an MCP server, a connector that lets an AI assistant use it as a tool. It is installed locally as a package and runs on your machine.
From the publisher's description, Electra Index matched it to 3 capabilities: document understanding, knowledge base qa and web scraping. That makes it a candidate for tasks such as answer questions from internal documents or answer support phone calls.
Facts come from the Official MCP Registry, as declared by the publisher. Capabilities are matched automatically (low confidence). None of this is a test of the tool.
At a glance
How to connect
As declared in the Official MCP Registry · not tested by Electra IndexInstall package
- NPM
thecrawlerv0.5.8 · runs locally (stdio)
MCP settings (Claude Desktop, Cursor and other MCP clients)
{
"mcpServers": {
"thecrawler": {
"command": "npx",
"args": [
"-y",
"thecrawler"
]
}
}
}Facts & evidence
Each fact shows where it was seen and when.| Fact | Value | Evidence | Source | Last confirmed |
|---|---|---|---|---|
| Install package | thecrawler | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Package registries | npm | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Local install (package) | yes | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Remote access (hosted) | no | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Repository | https://github.com/manchittlab/TheCrawler | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Registry status | active | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| Version | 0.5.8 | DECLARED | SRC-001 Official MCP Registry | 1d ago |
| MCP server | yes | DECLARED | SRC-001 Official MCP Registry | 1d ago |
Change history
Is this page accurate?
Flag it and our robot re-checks the official source. Reports never change a score.Is TheCrawler still online?
Free alerts · no account▲ 0Do you publish TheCrawler? Get it seen by people choosing a tool
For publishers · optionalLaunch Spotlight: 60 days of clearly labelled Sponsored visibility on the use-case pages where TheCrawler already appears, plus the weekly Pulse. No click in 60 days? The next 60 days are free.
Sponsored placements never change a score or a rank.
Questions about TheCrawler
How do I connect TheCrawler?
Install the npm package thecrawler. The exact settings are in "How to connect" above, as declared in the Official MCP Registry.
What can I use TheCrawler for?
Based on its description, it is most relevant to: answer questions from internal documents, answer support phone calls, collect data behind logins, extract website data.