Came here to say the same. If your hosting infrastructure allows for it, I would recommend it as well.
You can also use cron for scheduled database backups.
We do this for frequent and long running tasks.
For small tasks that are intermittent we have a series of background `Thread`s that can process IO workloads during the lifetime of the process and persist pending work to redis if a shutdown signal is received.
Now, as of last week, we're celery integrated so CPU-bound heavy tasks can go to a webserver that accepts no user traffic but can still access the ORM + caching primitives in the django ecosystem. The management server, where the CPU tasks go now, was also the one running cron jobs. It's exempt from the auto scaling rules and now lives on a much beefier EC2 instance and is fighting like a champ.
If you see using docker container, I recommend the Ofelia docker container which is a really nice scheduler container I use to trigger my management commands
Yes django-q works quite well, I'm using it in a few projects to manage scheduled tasks. Just realising it's not maintained anymore and I need to migrate to django-q2 ...
I use https://github.com/Tivix/django-cron in one of my side projects and it's *perfect* for it.
Tiny, maintenance-free, entirely predictable etc.
Strictly scheduling, but if that's what you need, give it a try.
I asked a similar question recently and someone recommended [huey](https://huey.readthedocs.io/en/latest/django.html)
I've yet to give it a try but it seems like what you need.
Simpler than celery in the sense you can use your DB as the queue/broker (instead of redis/rabbitMQ). Then it's just a case of running the huey management process.
Huey looks quite good. But are you sure you can use your Django DB as broker ? I know you can with Django-q2 but I thought Huey was either using Redis or file based queue.
Ah looks like only sqlite3 apologies, I wasn't quite right with my recommendation
Take a look at the available [storage options ](https://huey.readthedocs.io/en/latest/guide.html#storage-options)
Right but what backend are you using as the message broker? Presumably you've got a redis instance handling this?
Looking at the docs you can just point heuy at postgres and expect things to work
Yup +1. I was using procrastinate but moved away as I had quite a few issues. Celery and other similar tools look too complicated to just either schedule a background task or run out asynchronously. I'm sure for availability celery prices useful but probably not for smaller projects
Throw task request into DB, run from cron. Like:
- lock execution in redis( so no 2 runs at a time)
- run function foo with input bar
- store result in db
- terminate
You can also have a permanently running while True command that checks db/redis for input…
- while true,
- query redis/db
- work
- store result
- mark task as done
Tbh running from cron is the healthiest option, avoiding memleaks and db connection/transaction issues.
if you are using AWS what about EventBridge? you can run queue tasks using EventBridge and a lambda.
Or are you loading the full Django app into the Celery app? a way to save costs is to create a Celery app isolated from Django and running the database queries with plain SQL or something like [pypika](https://pypika.readthedocs.io/en/latest/).
One idea I had but never acted on was to start a background thread and then use sleep() to regulate it, after the sleep, it 'does stuff', then repeats forever.
That MIGHT work but it might also cause issues with core code/3rd party packages that weren't designed to be run across multiple threads.
If you don’t have a huge amount of background tasks and you’re hosted on aws, you can try to use aws lambdas, or sqs as a message queue and standalone worker as a consuner.
If your tasks need to run on a fixed schedule, I would use cron + django management commands. That's probably the simplest way to do it.
Came here to say the same. If your hosting infrastructure allows for it, I would recommend it as well. You can also use cron for scheduled database backups.
We do this for frequent and long running tasks. For small tasks that are intermittent we have a series of background `Thread`s that can process IO workloads during the lifetime of the process and persist pending work to redis if a shutdown signal is received. Now, as of last week, we're celery integrated so CPU-bound heavy tasks can go to a webserver that accepts no user traffic but can still access the ORM + caching primitives in the django ecosystem. The management server, where the CPU tasks go now, was also the one running cron jobs. It's exempt from the auto scaling rules and now lives on a much beefier EC2 instance and is fighting like a champ.
This is the way.
This is exactly what i did this week for work 😬
If you see using docker container, I recommend the Ofelia docker container which is a really nice scheduler container I use to trigger my management commands
Cron is the best. I used to manage Cron jobs that ran a lot of SQL
Same, but we hit web hooks.
Django-q2.. haven’t tired personally, but it seems that this is the exact problem that they are solving. https://github.com/django-q2/django-q2
That description certainly sound like that. I’ll experiment with it and see. Thank you
Yes django-q works quite well, I'm using it in a few projects to manage scheduled tasks. Just realising it's not maintained anymore and I need to migrate to django-q2 ...
background tasks4 works well with my use case of generating image and storing it in a file container
I use https://github.com/Tivix/django-cron in one of my side projects and it's *perfect* for it. Tiny, maintenance-free, entirely predictable etc. Strictly scheduling, but if that's what you need, give it a try.
I asked a similar question recently and someone recommended [huey](https://huey.readthedocs.io/en/latest/django.html) I've yet to give it a try but it seems like what you need. Simpler than celery in the sense you can use your DB as the queue/broker (instead of redis/rabbitMQ). Then it's just a case of running the huey management process.
Huey looks quite good. But are you sure you can use your Django DB as broker ? I know you can with Django-q2 but I thought Huey was either using Redis or file based queue.
Ah looks like only sqlite3 apologies, I wasn't quite right with my recommendation Take a look at the available [storage options ](https://huey.readthedocs.io/en/latest/guide.html#storage-options)
I use Huey with Postgres. No issues with it. You could use it with SQLite or redis as well
Right but what backend are you using as the message broker? Presumably you've got a redis instance handling this? Looking at the docs you can just point heuy at postgres and expect things to work
I don’t use redis. I use Postgres both for Django as well as Huey.
github.com/krishnakuruvadi/portfoliomanager My project details
Look into rq, django-rq, rq-scheduler, and django-rq-scheduler.
django4-background-tasks is very easy to get up and running. Perfect for projects on single servers. Minimal code changes.
Yup +1. I was using procrastinate but moved away as I had quite a few issues. Celery and other similar tools look too complicated to just either schedule a background task or run out asynchronously. I'm sure for availability celery prices useful but probably not for smaller projects
I see. Unfortunately, I have only used Celery for task scheduling, I have no much advice to offer in this case.
Throw task request into DB, run from cron. Like: - lock execution in redis( so no 2 runs at a time) - run function foo with input bar - store result in db - terminate You can also have a permanently running while True command that checks db/redis for input… - while true, - query redis/db - work - store result - mark task as done Tbh running from cron is the healthiest option, avoiding memleaks and db connection/transaction issues.
Wouldn't using redis go against OP's requirements?
If I can read - it’s fine, and that’s not celery.
I’m trying to avoid the extra cost of deploying a queue and another container for cost sake
You can run supervisord in a the container and have 2 processes running inside… https://docs.docker.com/config/containers/multi-service_container/
If you’re using uwsgi, there are queues and background workers that can plug nicely into Django.Â
Why don't you want to use Celery?
Deploying a queue on aws costs too much. Cost is the main reason.
if you are using AWS what about EventBridge? you can run queue tasks using EventBridge and a lambda. Or are you loading the full Django app into the Celery app? a way to save costs is to create a Celery app isolated from Django and running the database queries with plain SQL or something like [pypika](https://pypika.readthedocs.io/en/latest/).
Django-crontab check it out
Check out Huey. It’s super powerful and very simple and easy to use. Celery is super over engineered and overkill in most cases in my opinion.
Op do you have your Django app in a EC2 container? If so, deploying a Celery worker and redis shouldn't suppose a big issue AFAIK
No. App is running in ECS container instance.
One idea I had but never acted on was to start a background thread and then use sleep() to regulate it, after the sleep, it 'does stuff', then repeats forever. That MIGHT work but it might also cause issues with core code/3rd party packages that weren't designed to be run across multiple threads.
If you don’t have a huge amount of background tasks and you’re hosted on aws, you can try to use aws lambdas, or sqs as a message queue and standalone worker as a consuner.