Product Updates & Changes
Stay up to date with the latest improvements, new features, and bug fixes in WaterCrawl. We're constantly evolving to serve you better.
v0.6.0
Added
- Unified Docker build workflow for backend, frontend, and docs
- Manual version bump and release PR workflow for GitHub Actions
- Enhanced sitemap visualization and markdown export
Changed
- Improved error handling and PR creation in CI workflows
- Improved deployment infrastructure with dynamic Nginx config for MinIO buckets
- Enhanced API docs UI and added code examples (Go/Node/Python)
- Refactored hooks and improved API documentation rendering
Fixed
- Security audit issues and dependency updates
- Build issues and minor fixes
Infrastructure
- Updated CONTRIBUTING.md with correct file paths and commands
v0.5.0
Added
- Invitation-based user registration.
- Contribution guide and GitHub templates.
- Badges and emojis to README.
- Docker setup documentation.
- packageManager field in package.json.
Fixed
- Docker build warnings.
- Security vulnerabilities via dependency updates.
Features
- MinIO consistency check enabled on startup.
Infrastructure
- General project configuration and documentation updates.
- Switched to mono-repository structure.
- Moved documentation to main repository.
v0.3.3
Changed
- Updated user models, serializers, and services
- Modified common serializers and services
- Updated project settings and version
- Updated Privacy Policy
Added
- New user migration for newsletter and privacy confirmation
v0.3.2
Added
- Automated daily page credit reset for active subscriptions
- Celery beat schedule for running daily tasks
v0.3.1
Changed
- Removed custom billing cycle anchor from Stripe checkout to fix subscription timing issues
- Simplified Stripe checkout session configuration for better compatibility
- Fixed Stripe webhook handling for default plan subscriptions
v0.3.0
Added
- Team plan management system with Stripe integration
- Usage tracking and plan-based limits
- Enterprise mode configuration
- Resend email verification endpoint
- Comprehensive admin panel for User, Team, TeamMember, and API key management
- Unique email constraint with case-insensitive lookups
- Plan-based crawl request validation
- Usage history tracking
- Stripe webhook handling for subscription management
Changed
- Enhanced user authentication system
- Improved email templates (removed emojis, simplified text)
- Added translations for user-facing messages
- Better spider include/exclude paths handling
- Updated admin interface with improved filters and search
- Enhanced team member management
- Improved error handling and validation messages
Infrastructure
- Spider options handling for include/exclude paths
- Email verification and authentication issues
- Case-sensitive email lookup issues
- Team member invitation process
- Added unique email constraint
- Improved API key management
- Enhanced authentication validation
v0.2.1
Added
- New TeamSchemaView for dedicated team API documentation
- Custom schema generator for filtering team-specific endpoints
- Separate documentation files for better organization
- Tags for all API endpoints (Auth, Profile, Team, API Key, Crawl Requests, Crawl Results, Reports, Plugins, Common)
- Token refresh and verify views with proper documentation
- Theme customization for documentation UI
Changed
- Moved API documentation strings to dedicated files
- Improved endpoint descriptions and documentation structure
- Updated API key authentication to track last used timestamp
- Enhanced schema security configuration
- Reorganized URL patterns for better documentation access
- Updated settings for better API documentation title and description
- Changed "Get the current team" to "Get/Update the current team" for clarity
Infrastructure
- API key authentication display in ReDoc
- Documentation organization and endpoint grouping
- Swagger UI and ReDoc configuration
- Unused imports and code cleanup
v0.2.0
Added
- Integrated Playwright for dynamic page rendering and JavaScript execution
- Support for PDF and Screenshot attachments for crawl results
- Advanced page interaction options (wait time, cookie acceptance, locale settings)
- Improved Docker build process with multi-platform support
- Added API version endpoint
- Extended crawler options with timeout, cookies, locale, and headers support
- Duration tracking for crawl requests
- Support for longer URLs (up to 2048 characters)
Changed
- Enhanced page rendering with Playwright middleware
- Improved JavaScript handling and dynamic content extraction
- Enhanced Docker workflow with better caching and versioning
- Improved domain handling in spider options
- Updated concurrent request settings
- Better organization of crawler constants and types
Infrastructure
- Added multi-platform Docker builds (linux/amd64, linux/arm64)
- Improved Docker caching and build optimization
- Added version tracking in Docker builds
v0.0.2
Added
- Email service for sending templated emails
- Frontend settings service for managing UI configurations
- OAuth integration with GitHub and Google
- Password reset functionality with email notifications
- Email verification system
- Team invitation system improvements
- New user templates for email communications
Changed
- Enhanced user authentication system
- Improved team management functionality
- Updated permission system with login and signup controls
- Modified team invitation workflow
- Restructured common services and views
Dependencies
- Added html2text for email template processing
v0.0.1
Features
- Initial release of WaterCrawl
- Core web crawling functionality using Scrapy (v2.12.0)
- Django-based web application (v5.1.4)
- REST API using Django REST Framework (v3.15.2)
- Asynchronous task processing with Celery (v5.4.0)
- Redis integration for task queue management
- MinIO integration for file storage
- User authentication and authorization system
- OpenAI integration capabilities
Infrastructure
- Docker support with multi-container setup
- Swagger/OpenAPI documentation using drf-spectacular
- Scalable architecture with separate services
- Database integration with PostgreSQL
- File storage system using MinIO
- Celery beat for scheduled tasks
Dependencies
- Python 3.11+
- Django 5.1.4
- Scrapy 2.12.0
- Celery 5.4.0
- Redis
- PostgreSQL 17.2
- Nginx
- GunicornWSGI server
- MinIO (optional, can use S3 or local storage)