Fixes: 1) Update worker entrypoints to adjust prefetch multiplier and memory/task limits for optimized resource usage.
Extra: 1) Refactor Celery settings into a dedicated file for improved organization and maintainability; 2) Adjust Docker entrypoints to align with updated task configurations; 3) Register `orjson` serializer in a separate module for cleaner code structure.
Fixes: 1) Correct type signature in `validate` method for serializers.
Extra: 1) Remove redundant worker log format configurations in Celery; 2) Simplify log message format in log configuration for conciseness.
Fixes: 1) Extract and centralize broker transport options in `settings/celery.py`; 2) Remove redundant Celery settings from `celery.py`.
Extra: 1) Cleanup and streamline Celery configuration in `celery.py`.
Increased `task_soft_time_limit` to 7200 seconds and `task_time_limit` to 10800 seconds in both `celery.py` and `docker-compose.yml`. This change allows longer-running tasks to complete without premature termination.
Updated `task_soft_time_limit` to 3600 seconds and `task_time_limit` to 7200 seconds in both `celery.py` and `docker-compose.yml`. This allows tasks to run longer before being terminated, ensuring support for extended processing requirements.
Refactored Celery configurations to enhance task reliability, including changes to prefetch multiplier and timeout settings. Updated the worker command in `docker-compose.yml` to specify concurrency, memory limits, and task timeouts. Added a memory limit for the worker service to improve resource management.
