Product Updates & Changes

Stay up to date with the latest improvements, new features, and bug fixes in WaterCrawl. We're constantly evolving to serve you better.

v0.6.0

Added

  • Unified Docker build workflow for backend, frontend, and docs
  • Manual version bump and release PR workflow for GitHub Actions
  • Enhanced sitemap visualization and markdown export

Changed

  • Improved error handling and PR creation in CI workflows
  • Improved deployment infrastructure with dynamic Nginx config for MinIO buckets
  • Enhanced API docs UI and added code examples (Go/Node/Python)
  • Refactored hooks and improved API documentation rendering

Fixed

  • Security audit issues and dependency updates
  • Build issues and minor fixes

Infrastructure

  • Updated CONTRIBUTING.md with correct file paths and commands
v0.5.0

Added

  • Invitation-based user registration.
  • Contribution guide and GitHub templates.
  • Badges and emojis to README.
  • Docker setup documentation.
  • packageManager field in package.json.

Fixed

  • Docker build warnings.
  • Security vulnerabilities via dependency updates.

Features

  • MinIO consistency check enabled on startup.

Infrastructure

  • General project configuration and documentation updates.
  • Switched to mono-repository structure.
  • Moved documentation to main repository.
v0.3.3

Changed

  • Updated user models, serializers, and services
  • Modified common serializers and services
  • Updated project settings and version
  • Updated Privacy Policy

Added

  • New user migration for newsletter and privacy confirmation
v0.3.2

Added

  • Automated daily page credit reset for active subscriptions
  • Celery beat schedule for running daily tasks
v0.3.1

Changed

  • Removed custom billing cycle anchor from Stripe checkout to fix subscription timing issues
  • Simplified Stripe checkout session configuration for better compatibility
  • Fixed Stripe webhook handling for default plan subscriptions
v0.3.0

Added

  • Team plan management system with Stripe integration
  • Usage tracking and plan-based limits
  • Enterprise mode configuration
  • Resend email verification endpoint
  • Comprehensive admin panel for User, Team, TeamMember, and API key management
  • Unique email constraint with case-insensitive lookups
  • Plan-based crawl request validation
  • Usage history tracking
  • Stripe webhook handling for subscription management

Changed

  • Enhanced user authentication system
  • Improved email templates (removed emojis, simplified text)
  • Added translations for user-facing messages
  • Better spider include/exclude paths handling
  • Updated admin interface with improved filters and search
  • Enhanced team member management
  • Improved error handling and validation messages

Infrastructure

  • Spider options handling for include/exclude paths
  • Email verification and authentication issues
  • Case-sensitive email lookup issues
  • Team member invitation process
  • Added unique email constraint
  • Improved API key management
  • Enhanced authentication validation
v0.2.1

Added

  • New TeamSchemaView for dedicated team API documentation
  • Custom schema generator for filtering team-specific endpoints
  • Separate documentation files for better organization
  • Tags for all API endpoints (Auth, Profile, Team, API Key, Crawl Requests, Crawl Results, Reports, Plugins, Common)
  • Token refresh and verify views with proper documentation
  • Theme customization for documentation UI

Changed

  • Moved API documentation strings to dedicated files
  • Improved endpoint descriptions and documentation structure
  • Updated API key authentication to track last used timestamp
  • Enhanced schema security configuration
  • Reorganized URL patterns for better documentation access
  • Updated settings for better API documentation title and description
  • Changed "Get the current team" to "Get/Update the current team" for clarity

Infrastructure

  • API key authentication display in ReDoc
  • Documentation organization and endpoint grouping
  • Swagger UI and ReDoc configuration
  • Unused imports and code cleanup
v0.2.0

Added

  • Integrated Playwright for dynamic page rendering and JavaScript execution
  • Support for PDF and Screenshot attachments for crawl results
  • Advanced page interaction options (wait time, cookie acceptance, locale settings)
  • Improved Docker build process with multi-platform support
  • Added API version endpoint
  • Extended crawler options with timeout, cookies, locale, and headers support
  • Duration tracking for crawl requests
  • Support for longer URLs (up to 2048 characters)

Changed

  • Enhanced page rendering with Playwright middleware
  • Improved JavaScript handling and dynamic content extraction
  • Enhanced Docker workflow with better caching and versioning
  • Improved domain handling in spider options
  • Updated concurrent request settings
  • Better organization of crawler constants and types

Infrastructure

  • Added multi-platform Docker builds (linux/amd64, linux/arm64)
  • Improved Docker caching and build optimization
  • Added version tracking in Docker builds
v0.0.2

Added

  • Email service for sending templated emails
  • Frontend settings service for managing UI configurations
  • OAuth integration with GitHub and Google
  • Password reset functionality with email notifications
  • Email verification system
  • Team invitation system improvements
  • New user templates for email communications

Changed

  • Enhanced user authentication system
  • Improved team management functionality
  • Updated permission system with login and signup controls
  • Modified team invitation workflow
  • Restructured common services and views

Dependencies

  • Added html2text for email template processing
v0.0.1

Features

  • Initial release of WaterCrawl
  • Core web crawling functionality using Scrapy (v2.12.0)
  • Django-based web application (v5.1.4)
  • REST API using Django REST Framework (v3.15.2)
  • Asynchronous task processing with Celery (v5.4.0)
  • Redis integration for task queue management
  • MinIO integration for file storage
  • User authentication and authorization system
  • OpenAI integration capabilities

Infrastructure

  • Docker support with multi-container setup
  • Swagger/OpenAPI documentation using drf-spectacular
  • Scalable architecture with separate services
  • Database integration with PostgreSQL
  • File storage system using MinIO
  • Celery beat for scheduled tasks

Dependencies

  • Python 3.11+
  • Django 5.1.4
  • Scrapy 2.12.0
  • Celery 5.4.0
  • Redis
  • PostgreSQL 17.2
  • Nginx
  • GunicornWSGI server
  • MinIO (optional, can use S3 or local storage)