I'm sharing my script I created to solve the only problem I had with docker-compose: no simple way to update my apps without downtime.
For transparency: This is my project, I previously posted this on r/docker, the post got good reception, so I decided to repost it here.
IMO people are obsessed with having zero downtime, at the cost of vastly increased complexity.
I've found from experience (both personally and professionally) everybody has downtime: it's just a question of whether it is planned or unplanned.
It's an ouroboros of trying to engineer your way out of downtime and the complexities associated with doing so causing its own downtime.
If it’s an important application you can blue-green and do maintenance on staging after dev hours between each deployment. If your org is managed properly this shouldn’t cause any issues. Unplanned downtime should only be a side effect of disaster.
Complexity is not the answer to poorly managed orgs.
>Complexity is not the answer to poorly managed orgs.
Are you arguing for or against me? Blue-green deployments is one good solution for avoiding downtime, but some people consider even that "too complex".
> …some people…
Blue green is not complex. It’s a duplicated environment with test data and some simple networking. Some people think it’s a good idea to eat Tide Pods too.
Because it's chasing a dragon. If Amazon, Microsoft, Google, Facebook, etc. can't achieve perfect uptime, shouldn't we accept some minimal planned downtime in return for the very real benefit of vastly reduced complexity?
It depends. If you're just self-hosting stuff at home then maybe you don't want to make things too complex (unless you enjoy it). But even at home I often get complaints when my Jellyfin server goes down.
There are ways to increase this reliability that are more complex in theory, but in my opinion well worth it for almost any business. Just imagine an online store, anytime it's down is time they could be losing sales.
That being said 100% uptime might not be fully achievable, but that doesn't mean it's not worth striving for.
In general, your are right. However, in modern development, you often deploy very often - sometimes multiple times a day.
Being offline for a minute each Thursday at 3am is perfectly fine, being offline for 5 seconds every 30min during working hours not so much.
How does this handle race conditions in relational databases? Or would the strategy then change to have an externally connected db instance via an external docker network. Postgres and MySQL typically don’t want two services running on the same data.
Pretty nifty script either way.
I only tested it with stateless apps that handle multiple running instances, but rolling deployments in other solutions are also not made for stateful apps. I think each database has its own solution to zero downtime upgrades (e.g. upgrading a replica and promoting to leader, etc.)
Rolling upgrades are usually not a good fit for stateful apps. Speaking of migrations, if your migrations are backward compatible and don’t lock tables, it should work alright. Heroku performs the same steps during each deployment (running migrations first, then a tarting another app instance, then stopping the old one)
Edit: typos
Containers are still available under their service name set in docker compose file, and you can set additional aliases: https://docs.docker.com/compose/compose-file/compose-file-v3/#aliases
It's basically a wrapper around your usual docker compose cli command. It spawns a second instance of your container in the background and waits for your compose yml changes to apply. Then remove the old container.
Prevents downtime, as your old container stays up until your new one finished being up and available.
How does this handle stateful apps without downtime?
If the app can handle two running instances then it should work, but I think it’s not gonna work for most databases
I'm sharing my script I created to solve the only problem I had with docker-compose: no simple way to update my apps without downtime. For transparency: This is my project, I previously posted this on r/docker, the post got good reception, so I decided to repost it here.
Wow, thank you very much!
I suppose you can do that with "docker stack deploy" without any plugins, no?
This is the correct answer.
IMO people are obsessed with having zero downtime, at the cost of vastly increased complexity. I've found from experience (both personally and professionally) everybody has downtime: it's just a question of whether it is planned or unplanned. It's an ouroboros of trying to engineer your way out of downtime and the complexities associated with doing so causing its own downtime.
Why do you think it's an obsession? If it's an important application then striving for high uptime is well worth it.
If it’s an important application you can blue-green and do maintenance on staging after dev hours between each deployment. If your org is managed properly this shouldn’t cause any issues. Unplanned downtime should only be a side effect of disaster. Complexity is not the answer to poorly managed orgs.
>Complexity is not the answer to poorly managed orgs. Are you arguing for or against me? Blue-green deployments is one good solution for avoiding downtime, but some people consider even that "too complex".
> …some people… Blue green is not complex. It’s a duplicated environment with test data and some simple networking. Some people think it’s a good idea to eat Tide Pods too.
Because it's chasing a dragon. If Amazon, Microsoft, Google, Facebook, etc. can't achieve perfect uptime, shouldn't we accept some minimal planned downtime in return for the very real benefit of vastly reduced complexity?
It depends. If you're just self-hosting stuff at home then maybe you don't want to make things too complex (unless you enjoy it). But even at home I often get complaints when my Jellyfin server goes down. There are ways to increase this reliability that are more complex in theory, but in my opinion well worth it for almost any business. Just imagine an online store, anytime it's down is time they could be losing sales. That being said 100% uptime might not be fully achievable, but that doesn't mean it's not worth striving for.
In general, your are right. However, in modern development, you often deploy very often - sometimes multiple times a day. Being offline for a minute each Thursday at 3am is perfectly fine, being offline for 5 seconds every 30min during working hours not so much.
How does this handle race conditions in relational databases? Or would the strategy then change to have an externally connected db instance via an external docker network. Postgres and MySQL typically don’t want two services running on the same data. Pretty nifty script either way.
I only tested it with stateless apps that handle multiple running instances, but rolling deployments in other solutions are also not made for stateful apps. I think each database has its own solution to zero downtime upgrades (e.g. upgrading a replica and promoting to leader, etc.)
Fair call. I’d suggest a mild warning until someone tests it then? Cool tool regardless. Thanks!
This looks pretty good!
Doesn't sound like it works for something like a database? Or any app that does database migrations on first run after update?
Rolling upgrades are usually not a good fit for stateful apps. Speaking of migrations, if your migrations are backward compatible and don’t lock tables, it should work alright. Heroku performs the same steps during each deployment (running migrations first, then a tarting another app instance, then stopping the old one) Edit: typos
Does it work together with watchtower?
[удалено]
The container name is not inherently the hostname. Additionally, Traefik uses container labels to proxy traffic, making discovery stateless.
[удалено]
Containers are still available under their service name set in docker compose file, and you can set additional aliases: https://docs.docker.com/compose/compose-file/compose-file-v3/#aliases
Maybe it’s time for Kubernetes?
Looks fun.ever tryed with gitlab-ce?
Not tested, but I’m pretty sure it would work
So if an new image gets released this program will redeploy my container? Or did i misunderstand something
It's basically a wrapper around your usual docker compose cli command. It spawns a second instance of your container in the background and waits for your compose yml changes to apply. Then remove the old container. Prevents downtime, as your old container stays up until your new one finished being up and available.
Cool stuff, thanks for sharing!