🚀 WaterCrawl 0.9.2 Is Here: Explore Deeper with Sitemap & Batch Crawling Tools

🗺️ Generate and Visualize Sitemaps with Ease

One of the standout features in this release is sitemap generation. With just a few clicks, you can now generate sitemaps from any Websites and view them in multiple formats:

• Graph view for intuitive visual understanding

• Markdown view for quick documentation or LLM usage

• JSON Format as an Array that you can iterate over

With this tool, you can have an overview of the website's available URLs before crawling pages.

🌀 Batch Crawling for High-Throughput Tasks

Crawling multiple URLs just got a major upgrade. Our new batch crawling support lets you process multiple URLs in a single request—perfect for users managing large projects, automating workflows, or monitoring dynamic content across multiple domains.

With batch crawling, you can:

• Submit a list of URLs in one go

• Monitor real-time crawl status as before

🔧 Engine feedback

With this new feature, you now have better visibility into blockers, helping you fine-tune your crawling parameters for improved data extraction.

• Better understanding of crawl, search, and sitemap errors
• Real-time information direct from the crawler engine

Other Notable Improvements

While sitemap and batch crawl lead the charge, we’ve also packed this release with quality-of-life updates:

• Breadcrumb navigation to keep you oriented in complex interfaces

• Improved API documentation for easier integration and development

• Enhanced SSE (server-sent events) handling for more reliable updates

• UI and terminology polish for a cleaner, more consistent experience

---

Try It Out

These features are live now. Head over to your dashboard, start a new crawl, and explore the new Sitemap and Batch Crawl options.

As always, we’d love to hear your feedback. Let us know what’s working, what’s not, and what you’d like to see next.

Happy crawling!
— The WaterCrawl Team