19d • 💬 General discussion
Web Researcher Agent
I’m working on a small project to create a research agent that can:
- Crawl an entire website (including all subpages under the same domain).
- Extract and save all the data into a single text file.
- Download every attachment available on the site (PDFs, docs, etc.).
- Later, I’ll feed all this collected data into an LLM-powered notebook for deep analysis and insights.
The idea is to make information gathering automatic and efficient, so I can focus on using the data instead of spending hours collecting it manually.
If anyone has experience building similar agents or optimizing crawlers, I’d love to hear your tips and feedback!
4
6 comments
Web Researcher Agent
powered by
skool.com/automate-with-n8n-7409
Ready-to-use N8N workflows with step-by-step guides. Learn automation fundamentals while building real solutions for your business or clients.
Suggested communities
Powered by