Activity
Mon
Wed
Fri
Sun
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
What is this?
Less
More
Memberships
Automate with N8N
162 members • Free
Ai Automation Vault
12.3k members • Free
1 contribution to Automate with N8N
19d •
Web Researcher Agent
I’m working on a small project to create a research agent that can: 1. Crawl an entire website (including all subpages under the same domain). 2. Extract and save all the data into a single text file. 3. Download every attachment available on the site (PDFs, docs, etc.). 4. Later, I’ll feed all this collected data into an LLM-powered notebook for deep analysis and insights. The idea is to make information gathering automatic and efficient, so I can focus on using the data instead of spending hours collecting it manually. If anyone has experience building similar agents or optimizing crawlers, I’d love to hear your tips and feedback!
2 likes • 18d
Thanks, Tiago! I’m building a research agent that can (1) crawl a domain (all subpages), (2) download every page and attachment (PDFs, docs, etc.), and (3) feed the corpus into an LLM-powered notebook for Q&A. The goal is to create an “expert” on a specific asset allocator (my field). For example, ask something about NBIM Or Vanguard and have the agent search across everything on their site, including PDFs and publications. I’m currently keeping the crawl/download step separate from the LLM notebook, but if you’ve stitched this into an end-to-end workflow, I’d love to learn from it. If you’re able to share your Firecrawl MCP code, that would be amazing. Appreciate the pointers!
2 likes • 18d
In the short term I’m going to focus on the download/scrape piece only: crawl the domain, enumerate subpages, and fetch all attachments (PDFs, docs). I won’t build the Q&A layer yet—I’ll prototype that with an off-the-shelf solution like Google’s NotebookLM or OpenAI Projects. Once the crawling + attachments flow is solid, I’ll think about the full end-to-end pipeline. Thanks Tiago!
1-1 of 1
