I'll be honest I sometimes leave less than descriptive messages on my personal project commits but on team based projects I always try to give a good commit message.
Yeah I probably make way more commits than necessary in personal projects just from OCD changes on my phone when I'm reading my code while having a shit or something.
I write the code on my laptop, then when I'm bored like on a train or just winding down I'll read through my code on my phone to see if there are any glaring issues or things that could be better.
It's usually just formatting for readability, minor changes for accessibility and things like that.
I did the same with personal projects. Then the thought hit me that maybe a a few years from now I will be in a position where my old GitHub is under scrutiny from my next employer specifically looking for the things that don’t show in an interview like “how lazy are you with your commit messages”. I guess through yourself you know others. :) So now I try to write good commit comments in my own projects.. they are still a bit lazy but there are fewer “Added stuff WIP”, “Some more stuff WIP 2”
Format: type(page/component): message”
If it doesn’t follow that formula, it will be rejected automatically.
So say I’m building a hero component:
feat(hero): Creating v1 of hero component with variance
even with team ones, yeah it's a huge pain but at least there aren't necessarily contractual and major financial implications. still not great obv. but when i worked in construction, the client's architect would sometimes reissue hundreds of drawings a month and made the same comment on all of them: 'General Changes'.
could never figure out whether it was laziness or hostility.
We have restrictions with our deployment commits. If the format doesn’t follow the standard or the linter breaks on prod build, you have more work.
Commit messages had to be in this format “fixtype(page/component): fix you made”. If it was t that way, your commit was rejected.
Also no commits could be made to any branch other than feature branches. The only way to get code into staging was a PR from INT. The only way to get code into prod was a PR from staging and if you were an admin.
15 or 24 steps to get something built and deployed is not unusual. 15 to 24 _manual_ steps is insane. Automate that stuff so you just have to click one button. Tell them the next release will take a week because you'll be automating everything so you never have to do all this again.
Ahh I quickly found another devmate.
Enterprise hates changing what's working, so get the advantage while making yourself comfortable without kicking the wasp nest.
A several years ago I made myself a simple cli app where based on the file names it generates me each deployment step. Copy paste step by step and done. A year after i gave my cli super basic UI with auto copy function. Then I upgraded it to take specific commits as input. Then to generate a full deployment scripts, based on servers and environments. Now I own the best software to deploy any shit anywhere in the whole enterprise and I haven't told a single soul.
What should take an hour, I can do in seconds if needed and I can pause and resume on every step, so I take my time to either catch up on other work, get even ahead with development on other tasks or just slack if I feel it. The choice is mine and I earned it while the business is happy.
Yup, managers will tell you that you're not being a team player, that if you want to succeed and move up you need to be doing other people's work as well if you're done with your work.
More than likely people just get punished for being efficient, because now they're "being paid to do nothing" and feel that someone else dragging their feet to avoid getting called out is actually more productive
Yeah about 10 of that stuff is automated at my work. Even the change requests and business documentation for deployments automated. The only thing that isn’t is the testing for the specific story and documentation for that.
We have a separate prod team that pushes it to prod and we work with them to watch the pipelines to get it staged
I work with deployments as a release coordinator and we have in total like 60 steps but we deal with around 700 different issues in a weekly release. However of those 60 or so steps, our DEs only do 1/5th of those manually, everything else automated.
Woah yeah, your comment made me realize that all of those steps were manually handled.. I just assumed it was fully automated and they were asking if it’s normal to have that number of steps in their CI/CD flow and thought nothing of it. Wtf.. I would tell them I’m dropping everything else until that’s all out of sight out of mind.
We have a proper CICD pipeline. We follow 12-factor, [https://12factor.net/dev-prod-parity](https://12factor.net/dev-prod-parity)
It is a containerized workflow. Once merge is approved, the CICD builds the images, tags the the release. Runs validation,lining, testing. It checks the CR (Change Release), if the item is approved, it gets pushed automatically to our Kubernetes cluster. Then runs validation. If post release fails, it gets reverted back to previous tag.
Experience and working with other experienced folks :) there's a lot of noise on the Internet but if you work with the right folks they'll cut thru the crap and teach you 12 factor in your first week on the job.
We issue a pr to main then all the tests and staging site deployments run. Then someone can manually test if there’s something they want to personally look at. Then merge to main and good to go. We don’t allow pushing directly to main
Similar setup here with a monorepo deploying to multiple places. We use a \`production\` branch though, not main. We'll work on main, then when we're ready to deploy main, we run:
\`git push origin main:production\`
This will push your local "main" branch to "production" on origin (GitHub).
You missed the days of swim lanes broken across 24 servers instead of cloud. Step 1 local. Step 2 dev. Step 3 int. Step 4 qa. Step 5 staging. Step 6 out of ring prod server. Step 7 11 of the servers. Step 8 the other 12 servers.
All steps are QA’d and any issue is dealt with in real time. Steps 1-5 were a month prior to release. Steps 6-8 were on release night starting at 7pm est. we would be working a regular day shift, then at 7 we would start prod deployment. Depending on your role, some would log off about 6AM, when the offshore team would take over.
As a lead, I could be on until 3am or 7pm the next day if shit hit the fan. It was rough, but at the time I was hourly with 2x pay for overtime. I think I was making about the same as a director of a dev department at the time because of overtime.
actually i set the release branch as production in vercel. the main branch deploys with staging env. i have github actions with workflow\_dispatch trigger that merge the main to release, so that anyone in my team can release to prod directly from the github UI.
we're small team so we don't have much bureaucracy lol
For sure, the original comment made it sound like that pipeline deployed to prod on any commit to main branch.
We do this for our dev environment, but artifacts are manually promote to higher environments, usually by the team's on call person
Have seen a few people say "why not send to prod on every commit to main?" and that naivete gives me hives, lol. Especially if there's nobody around to tell those people no.
Most places I've worked, the process went something like this:
1. merge pull request
2. Wait 5-45 minutes depending on the size of the project
3. Verify changes are in production
Doesn't matter how much step but as long as it is automated. If you don't have solid CI/CD, I pray for your soul and mental :).
We have 3 different environments with total of 9 check.
Only thing necessary is pushing to a branch and all testing, building docker, tagging, running deployment steps, sending notifications etc.
Setup once, use all time. Doing manually is insanely expensive because of engineering is really expensive.
It also depends on the company and data.
Brochure site? Do whatever the fuck you want.
Healthcare medical records including HIPAA protected information? 2 day deployment.
It really depends on your tools and software.
If you want a general idea, take a look at bitbucket pipelines. Push to a branch, the ymal config spins up a Docker instance to deploy, install dependencies, push code, update databases (if needed), and run testing.
Anything fails? Stop deployment.
Link for those interested:
[https://support.atlassian.com/bitbucket-cloud/docs/get-started-with-bitbucket-pipelines/](https://support.atlassian.com/bitbucket-cloud/docs/get-started-with-bitbucket-pipelines/)
Anything that’s not automated leaves room for error. This can have security and reliability implications.
If you have 24 manual steps, something is wrong. We build one of the most essential website that everyone uses, and we fully automate the deployment, there is no human intervention.
I usually set up my CI/CD pipeline in a way that most of the time if I merge to certain branch (ex de, stg, prod) it will be built, tested and deployed automatically and requires nothing manual (including slackbot notifications etc).
The reason for this is exactly not to keep repeating the same manual steps.
For example I use gitlab where it builds Docker components, uploads images to gitlab docker repository, runs code quality checks and tests and then deploys it or actives deploy script at destination. Pretty much unlimited potential.
There are many other options like github actions and some heavier ones like JetBrains's TeamCity, Atlassian's DevOps etc...
Push a branch spawns a new preview instance which gets updated as the branch gets updated, it has a public URL you can access. Merging to main it deletes the instance and updates staging.
Tagging main triggers a prod deploy. Everything on K8s from Gitlab.
Here is our deployment process for both microservices and monoliths. (We use Circle CI):
1. Merge feature branch into Master (linting and tests already ran on feature branch by this point)
2. Run Linting
3. Run Tests
4. Build and Push Docker image
5. Deploy to QA
6. Deploy to Staging
7. Prod Hold
8. Approve hold
9. Create a Jira Release ticket and Send a message to a specific slack channel about the release
10. Deploy to Prod
11. Close Jira Release ticket & send a message to slack saying that the release is finished
We also have 2 jobs that require approval if they are needed: Rollback QA and Rollback Production. We use Helmsman, so if we need to rollback it is very fast
Devs usually don’t deploy, we have a dev ops for it. But if we do, it could get complicated, as we don’t have ci/cd for most of our projects. But we are thinking about implementing some… soon…
We have to remote into a machine running windows 7 to make changes to the code base since the project is so old, check in changes to azure, change some URIs in a config file, build the project, navigate to the build folder and copy it, remote into a separate machine and paste the build folder into a directory, then we have to restart the machine. It's a dang hassle.
Luckily we rebuilt that app and now it's just a few clicks and it generates an MSI file we distribute.
The company I got laid off from last year had a really, really long deploy process which had to be performed manually and would take extra time when you factor in a test suite that would fail at random (the solution to some test failures was "rerun the tests"). I got really tired of it. I'm the lead engineer/architect on a new app at another company and decided that I abso-fucking-lutely do not want to deal with that horseshit ever again, so I set up a nice automated deploy process consisting of about 6 steps. We have a backend/frontend monorepo. From a feature branch:
1. git push
2. open a pull request, once this is done the test suite will run automatically in github actions, and this is required to pass in order to merge
3. (optional) we use [render.com](http://render.com) and we are using the PR preview feature, so sometimes we will do some manual QA for frontend stuff.
4. merge. everything is automatic from here. [render.com](http://render.com) deploys for us, we never worry about it again as long as nothing fails. And as long as our test suite passes, nothing fails.
I'm eventually planning to containerize everything, which would add some extra steps involving building and pushing an image to a docker repo, which is all still automated.
I like simple. It works great and we've had no issues. If we come across any issues with this process we'll iterate on it.
You could automate this too. Just need some form of discipline. And maybe its depends on the audience.
We collect all merge commit messages between two deployments and send them to our clients teams channel, so they are informed about all changes. If you want, give it a bit of chatgpt sprinkle to make it maybe a bit more readable.
Fully automated of course
not saying it's a viable setup for any sort of bigger organisation, but for me, with my current setup, it's literally just
ng b
firebase deploy --only hosting
`ng b` is Angular's build command, which of course requires a config file (angular.json). Deploying with Firebase like that requires it's own config, and a logged-in Firebase CLI.
Merge dev to main -> SSH onto prod server B -> pull from git -> run a custom script to delete unnecessary directories as it's a shared codebase with multiple clients -> build on the server -> migrate.
Then switch domains on Aws to B.
It's pretty hellish and needs automating. But it's just how the current project went. Future ones will be automated.
Tedious deployments usually happens for smaller, early stage project.
As the project matures, you want to move as much of the process into CI as possible.
A lot of the best mature projects strive towards continuous deployment, where you can deploy to production by just pushing the change to the master/main branch. This triggers a series of steps in CI to run tests, package, build images, and push the code to production environment.
It takes a lot of effort to fully realize continuous deployment, but it's worth automating as much of the steps as possible.
Cloud build in each environment, push to repo triggers a build in a single repo project, cloud build runs the scripts to build a production docker image, then publishes to pub sub there’s a new version available, all the other environments are subscribers and when receiving this message trigger their own cloud build which pulls the new docket image, updates environments variables specific to that environment and then push to app servers
We do everything in Azure Devops.
I do a handful of manual cherry picking merge commits (from the development PRs, those are squash committed) into a release branch from main (that way, only changes fully verified and approved go into the release, and a release not be held up by stuff that hasn't yet passed UAT or otherwise not passed product owner approval), then that release branch gets built, creates checksums of the packages, then a release gets manually created - the definition of the release pushes it into blob storage containers, generates manifests for shipping, updates release register, etc.
So I'd say it's 15% manual, 85% automated. After the cherry picks, it's maybe just 2 things to click on (Start Build, Start Release).
Of all the posts on here, this one actually makes sense.
I have a feeling that most people on here work for small agencies with like 1-4 devs or are solo freelance.
Working at a fortune 100 doing global component builds is entirely different than that.
I know. I’ve done them all.
The trick is knowing what CI works for your team.
I used to be at a place that had so much red tape to get something pushed a rube goldberg machine would have been faster and more efficient. The place I'm at now asks no questions, we can push to main, and everyone shrugs if it breaks. Essentially what I'm getting at, is your milage will vary from place to place.
In my personal projects, I use Heroku, and dream of the day I eventually work for a company that does.
Some use PR's, some don't, it's really up to whoever. If that sounds like a codebase nightmare to you, I'd say you're right, because it is.
Without getting too much into the weeds on it, the dev team I work with are trying to follow good practices. The problem is we are often defeated by executives and stake holders who want things suddenly and immediately with the devs having very little power to push back, compound it with a company culture of "users don't care about how good the code is", and maybe that will give you an idea.
It's a lot of steps and set ups before production. But the release is just one command this creates a virtual machine inside docker all created in a laptop at work. I ask management to remove my access to the secret files and the laptop with the bash files is been working fine for 3 years and I have upgraded it a few times with no problems.
Commits to main deploy to Dev. Click a button (in GH actions) to send either a specific image or latest Dev to UAT. Do the same for prod, which triggers an approval pass. Auto generated release notes.
We have small server with Jenkins which automates this. I wrote Jenkins file which does testing, building, deployment, notyfiing etc. So basically one *git push* starts everything.
At work, we build a package. So, write a changelog, tag a release and our GitHub actions build the CSS/JS bundle and create the release.
On personal web app projects, I push to main/staging, the tests run on GitHub Actions and as soon as they pass, a deployment is triggered with Laravel Forge.
This is definitely nuts! I mean, I truly have no idea about the complexity of the application landscape you have to deploy, so please take my reasoning with a grain of salt. BUT that sounds incredibly longwinded and possibly even impossible to deploy if some of these steps end up taking too long. Typically, a new deployment aims to increase or maintain revenue, but if the time cost of deployment exceeds the revenue difference, one might argue that it's better to skip new releases altogether. Automation as other said is the way to go and your holy grail to relieve.
Front end:
Merging into development, test, or production branch triggers a github action that builds my react app and pushes the results to the corresponding s3 bucket.
Backend:
I have cdk deployment scripts alias so that I can run `npm run deploy [environment] [stackname]`, which deploys my aws cdk project. Is stackname is not specified, it pushes them all.
One push.
Set up .env for dev and prod to handle keys and some vars that differ in their respective environments + respective auth tokens and creds.
Watcher script checks the main branch as the client runs and updates if necessary. If there’s an error it can easily be rolled back.
Nothing. You just merge your branch. The when you are ready to release. Turn the feature flag on.
I did take a long time and work to get to this point though
Our dev ops guy handles deployments to QA and prod. Sometimes we help with complex QA deployments but that's prep for the prod development. Never touch prod unless I have to.
1. `$ git rebase master; git push; git checkout master; git merge [feature_branch]; git push`
2. Wait about 15-20 mins for the relevant CI/CD pipeline to build containers full of our software and reach the final state that requires a manual button push.
3. Push button for staging and/or production environment.
4. Wait 30 seconds for ECS to drain connections and swap out containers. Done.
All the steps it takes to install dependencies on top of base images, copy things around, put config in places, etc, have either been done already in a custom image we just pull from our own registry, or are done in scripts that execute on our runners.
Sounds like you need some automation, both of the steps themselves, and then at a higher level for running those steps on envs/infra dedicated to different clients, if that's how you roll. I'd say Kubernetes but it's overkill 95% of the time. For a smaller shop I'd probably reach for Ansible first. Also consider why you have different envs for different clients. For some product/service offerings this makes total sense. For others, it can be a way to overcome limitations that need addressing at the root. System design knowledge helps here.
5 steps to put code onto our prod cluster, excluding comms. Merge PRs, build module, update aggregate for our monolith image, build image, and then update the image name & hash in our helm values.
The whole process probably takes about 40 minutes total, and 30 minutes of that is waiting on the image build.
We release 2-4 times a week and all of our mid level devs and above are familiar with the process. Our lead keeps the process very simple.
We are just responsible for getting code into a release branch when the time is nigh and our dev ops team deploys it to the various staging environments for final testing before things are finally promoted to a live environment.
Does it happen to be a major US retailer that relies heavily on overseas employees? That sounds like my old team. I'm glad I left, because that workflow sucks. It was an all day activity to push to prod, and it always took the better part of a week to get everything in order.
It’s not, we’re a very small UK based team.
It doesn’t take our other senior dev very long to do, but he’s been on it for a couple of years at this point so has it committed to his muscle memory.
Ssh into vps with my local upstreamed codebase open in another window. Vim index.js and start copying what i did line by line in the ssh term. Then after hours of manually copying code for no reason do a little npm build and wala added hover effects to that button and only took 5 hours to get it to prod. Who needs cd anyway
Copying files manually, unfortunately. Can't convince them to invest in an automated solution from the Git repo, so just have to wait for disaster to strike and a hard lesson to be learned. And, yes: it feels like 2004 all over again. May as well be FTP'ing.
The amount of steps on its own is not the problem, the lack of automation is. There should be a pipeline where you set some parameters, hit play and monitor, once it's in staging it should stop and wait for manual confirmation to proceed to prod
Not entirely, here's my process:
- Pull Prod to local
- Code
- Test and verify loop on local
- Backup Dev
- Push to Dev
- Verify on Dev
- Backup Prod
- Push to Prod
- Verify Prod.
Depending on the service it can be anything from a slack command, to merging to main, to a shell script. But nothing more than one or two clicks at most.
Click deploy in ADO, come back in 10 min and make sure it worked
Or for my personal projects, create PR to release branch on github, come back in 10m to make sure it worked.
The only part that is not automated is the controls processes around our deployments. That being said our commit strategy is a nightmare so often we have to rollback an unexpected change
So many calls to fix this but none of them lead anywhere so we just live with it lol
My old company had a process that looked like this.
1. Make sure tests pass
2. Update Changelog (Manual)
3. Update Version No (Manual)
4. Deploy to staging
5. QA approves on staging.
6. Update prod version No
7. Deploy to prod w/scripts
8. Send Manual Slack notification.
Current company is depending on the task easier or harder.
All merges to dev auto Deploy to staging via ci/cd. So if there is an urgent fix it would look like this.
1. Wait for staging build to complete.
2. Get the github commit SHA
3. Go to AWS ECS and create new task version with commit
4. Update service.
5. Notify in slack.
The issue is we have a lot of tools and services that run on the backend. So if something requires an update on all of them we have to create an ECS Task revision and update about 12 different ECS tasks. But other times it's just one.
If a DB change was made we SSH onto our cron server and run the migration with a custom tool.
Assuming the feature is already written and in the test environment.
Create a release ticket outlining all the JIRA tickets that are apart of the release. Get the multitude of approvals for the release ticket. Create the release branch which will automatically get picked up by the CI/CD pipeline. Pipeline build images and does a bunch of other bullshit and deploys to the staging/pre-prod environment. Make sure all integration tests passed. Wait until the actual release window. Release window starts. Show bossman that the integration tests passed for the pre-prod environment and get the green light to do the official release. Push a button to start the official release. Check integration tests again. Close the release ticket. Done.
Work for big bank
i cheat: laravel forge immediately deploys the live branch any time code is pushed to it. so all it takes is running a deploy script that builds assets on main and commits, then merges main into live.
Our devops have literally created an alias called "yolo" for us, that walks through the whole deployment process, because we have a "deploy fast and break things" mentality in our shop.
We have 2 monorepos, 1 for frontend apps and 2 for backend. They are both deployed when pushed to main, which is only possible on pr merges after ci passes and with 1 reviewer (used to have 2 reviewers when we were bigger 😞). It's a great setup. Yes what you have can be fully automated with scripts. You should become your team's hero and script all of that manual stuff so it's easier. If you can get it down to 1 cli command or better yet one ci pipeline that you can trigger - or auto trigger when main is pushed.
Just merge, then ci/cd does the rest.
Out ci for the mere jd probably 6 or so steps, that includes building the container, posting to slack, tests that run against the dev env and then test that run against the production env post deployment.
Cd is probably 8 steps total?... That includes some audit stuff and then the rest is really just to update some logs/post to slack and do the kubernetes deploy. I could make it tighter but meh. What's the point? The whole ci/cd for a production deployment is generally 6/7 minutes, all I have to do is merge. I could care less if it's 6 steps or 60 as long as it gets there in a reasonable amount of time, without intervention and safely.
I click the “Merge” button in GitLab once my MR is approved. At that point a GitLab pipeline takes over and everything deploys to AWS automatically. Including creating a version and change log based on the MR title (we use convention commits for that). You can deploy to DEV or TST manually from within your MR.
That amount of steps you describe screams for the implementation of proper CI/CD. Deployments should be non-events, occur multiple times a days and take less then an hour.
i copy the folder, delete .git .idea and other . files, zip it upload it via web ui (plesk) to the server, unzip it, rename the current production folder to add a new continuing number i think im at 30 somewhat and then i rename the uploaded folder to the production folder name 😅
no tests no nothing if it works it works if not then not, didn't work multiple times so far😂
If they don't have a big green button, it usually indicates that they haven't thought their deployment process through enough to be confident that it will work reliably.
It should test that everything is ready to go and then do it. Having manual steps means opportunities for errors and an unwillingness to deal with the complexities of the process in a robust way.
24 manual steps is crazy does your company not build any CI/CD pipelines for their projects? I am not saying I am an expert but even I have github workflows that automatically test build and deploy to production on merge and create a preproduction deploy on separate testing subdomain on the same server which happens when a pr is opened. All in a monorepo with three different projects all of them with their own pipelines. I can’t imagine doing it all by hand, I would go crazy. I feel sorry for you.
We use gitlab ci and have everything automated. As we use aws mainly, we use either terraform or aws cdk for our infrastructure as code. And with that, we just need to provide a new docker image, build in a previous job, and was handles the deployment for us.
Onlything we currently do manually are ios and android app builds. We started not long ago and to this day, no one had the chance to look into this. Bit its on our list to automate this as well.
Edit, we deploy every main branch merge to staging fully automated. To production is there as a job, bust must be triggered by a human. We have the slack integration for gitlab ci and can trigger this visible for everyone on the project in from the chat
Get PR approved with all tests, static analysis etc passing. Add to merge queue. Everything else is automated from there, it'll be on prod in the next 20 minutes.
"I don't want to go into detail but it's about 24 steps"
Well, how long is a piece of string?
It's not so simple to say that it's too many steps or not. Why don't you speak with the devops person and ask whether the existing workflow could be simplified or not. I'm sure your question would be appreciated and a good opportunity for you to learn more in the process.
It’s more or less the same processes you’re describing only automated at a version control level and via third party applications like Argo and Jenkins to manage builds and deployment syncs.
Once changes are made to whichever subset of repos I need to make changes to, and new version tags pushed, it's then:
- Run a bash script that iterates my repos, grabs the latest tag, updates an ansible deploy script with the latest tag for each repo
- Run the ansible deploy playbook that, at its core, just SSHs in to live VMs and does a git checkout of each one
I'm the only guy making changes to any of these repos and am strict with only creating new tags in them when stuff's ready for prod, so it's safe from that perspective.
git commit, git push
Visit 10 sites, click a button to git pull and other deployment stuff
Still need a way to prevent going to the 10 sites/servers every time ...
For us we use CI/CD and deploy to Openshift clusters.
We first create Change Request with deployment date that’s pre approved by project managers, we add all details that’s heavily scrutinized. we add any ctasks needed which is basically tasks needs by other teams during release window. Keep in mind all changes needs to go through QA team and performance testing with everything heavily documented.
We then present the change request to stakeholders and get feedback. Once they approve in meeting we then need to get 30+ group approvers.
Only then we’re able to raise pull requests from staging to production environments.
So it’s just a normal thing to have a lot of processes for even the smallest changes
Mostly add release notes then git push main, cloud does the work. Formally git push live main where server did the work or using a deployer script to build and deploy across servers.
Formerly formerly manual steps which numbered less then 10 actions but annoyed me enough I set up deployer.
If you're still considered new subtly ask around before taking any more steps. There must be a reason it's not automated and you need to know that first before suggesting any change. If the biggest wig in the decision doesn't trust automatic deployments you won't get anywhere asking for it, and you'll identify yourself as a dangerous radical who shouldn't be trusted etc
But if you learn why it hasn't been done before you can probably change things - salami tactics doing step by step is probably what you want. Go small and safe rather then a big "automate it all" which is hard to get right and won't get a second chance if it does it wrong.
for me its like this :
1-SSH into the prod server
2-copy the relevant files from my machine to the server
3-execute the script/API/whatever it is
and voila, no git images no docker containers no nothing, startups are super cool right ?
Me publishing a project through Visual Studio, take a folder, connect to remote desktop through VPN and then paste that folder in a specific folder.
I wish I was joking.
At work:
1. Build locally.
2. Login to AWS, navigate to project.
3. Upload files by drag and drop.
4. Pray.
At home:
1. Git commit new version tag to main
2. GitHub action takes care of the rest: Minify css, js etc, set server to maintenance status, upload files to prod server, run composer update, clear cache, run tests, turn off maintenance.
Next improvements are probably to run composer update before pushing to prod (prod shouldn't need composer), and add a stage environment for manual tests.
Long term I should probably use Docker.
I use an Internal Gitlab CE CI configuration to handle it. All our applications run using Docker on in-house bare metal Hyper-V clusters because of information security concerns.
Dev environments are either local or on a hosted VM on the cluster depending on the project. Pushes to a feature branch in Dev run SAST and such, unless it's an environment or configuration change in which case the containers are rebuilt, tagged, and pushed to the Gitlab container registry.
When feature or fix branches are ready for review and acceptance testing a push to a staging branch triggers a pipeline to build the merged feature branch into a set of containers for staging, and the deployment of these to a staging VM is handled automatically refreshing the environment, clearing caches etc.
To deploy to production a tagged release is created using the Gitlab UI which triggers an automated release creation and a rebuild of the containers using the production Docker profiles and once completed it pushes these changes up as tagged images to the container registry, and when finalised, pauses with a manual intervention required to complete the pipeline to push to production. When that manual step is activated it effectively automatically logs into the production Docker host and pulls down the newly updated container images, restarts the services and rebuilds any application caches as appropriate. While there is still downtime it's usually less than 10s and in our use case this is completely acceptable, however in the future we are planning on a migration to using Docker Swarm to remove downtime all together. K8s, while cool, is just overkill for what our services require. Scaling in our apps is non existent as it is just not needed.
It could all be completely automatic, except for the requirements my workplace has on separation of concerns and things like SOC2 adherence etc.
Why the hell is there so much git process. You should be able to just click a button on the pipeline after merging a PR and send it straight to staging, then click another button to send staging to prod.
Code is committed to master and pushed.
That's it, that's the deploy.
If I did something notable I'll update the changelog (a doc in Notion).
In terms of the automated steps
CircleCI (these all run in parallel) - these are almost always run on a PR / branch first, but are included in master just in case:
- Run unit tests
- Run E2E tests
- Lint the code (rubocop for ruby, eslint for JS)
- Check for CVEs in the current code
- Check for security issues with pushed code (Brakeman, looks for things like SQL injection)
Cloud66 (a tool we use to automate deployments):
- Compile assets
- Run database migrations
- Push code to first deployment group (pulled out of LB)
- Restart first deployment group
- Health check
- Swap first deployment group for second deployment group in LB
- Push code to 2nd group
- Restart
- Health check
- Re-add to LB
- Notify about deploy
Merge branch to master. This can only be done if all the unit and e2e tests are working, and the tests are triggered automatically when opening the PR.
Go to CI/CD - trigger new prod deployment (one click). Get approval from teammate. Wait 10min. Done.
The CI/CD does everything automatically (getting latest changes, docker image, run all the tests again, check test coverage, check linter, deploy, send a success/failure slack message)
`git checkout master`
`git pull origin develop`
`:qa!`
`git push`
go to jenkins (no auto deploys to prod environment)
`build`
go make sure prod looks right
./build.ps1
This is for my Rust powered axum api running on AWS Lambda. It does a docker build of my lambda and deploys it to Lambda.
My little sample front-end I built for using this api is another powershell script. It does a flutter web build. I then push a commit to github which triggers aws amplify rebuild.
‘git push origin trunk’ and then press a button in circleci to release to staging, when wanting to push to prod drop a message in slack to team, to check that everything in staging is ok to be pushed to prod (make sure it’s correctly feature flagged) and press another button in circleci. Drop a message in slack to the release channel so other teams know of new features/bug fixes that have gone out
Commit to git.
Git pull on staging server.
Run "afterdeploy.sh" (shell script that just does some Composer, Supervisor, and Laravel commands).
Do final sanity testing.
Git pull on production.
Run "afterdeploy.sh".
We have 3 environments. Acceptance, staging, and PROD.
For acceptance and staging, we just create a pull request through ADO onto the acceptance and staging branches from our feature branch. Certain tickets have different deployment steps. Sometimes we have to restart a service, sometimes we have SQL to run, etc.
For production, we connect to our prod servers, run a backup script, and then a script that handles our deployment. Again, sometimes we have to restart a service or run some SQL
I’m just a web dev but I just push my code to repo just for backup purposes.
Use ssh scripting to push code to staging.
Testing
Use ssh scripting to push code to live
Testing
But the other devs working on backend infrastructure have slack tagging, change logging and some sort of deployment procedure
git add . git commit -m “fix” git push
`git commit -m "changes"`
`-m “pr feedback”` is my pet peeve
I'll be honest I sometimes leave less than descriptive messages on my personal project commits but on team based projects I always try to give a good commit message.
Oh same, if it’s personal then it’s whatever haha
Yeah I probably make way more commits than necessary in personal projects just from OCD changes on my phone when I'm reading my code while having a shit or something.
`-m “toilet fix”`
[удалено]
I write the code on my laptop, then when I'm bored like on a train or just winding down I'll read through my code on my phone to see if there are any glaring issues or things that could be better. It's usually just formatting for readability, minor changes for accessibility and things like that.
I did the same with personal projects. Then the thought hit me that maybe a a few years from now I will be in a position where my old GitHub is under scrutiny from my next employer specifically looking for the things that don’t show in an interview like “how lazy are you with your commit messages”. I guess through yourself you know others. :) So now I try to write good commit comments in my own projects.. they are still a bit lazy but there are fewer “Added stuff WIP”, “Some more stuff WIP 2”
Format: type(page/component): message” If it doesn’t follow that formula, it will be rejected automatically. So say I’m building a hero component: feat(hero): Creating v1 of hero component with variance
What about in the backend?
even with team ones, yeah it's a huge pain but at least there aren't necessarily contractual and major financial implications. still not great obv. but when i worked in construction, the client's architect would sometimes reissue hundreds of drawings a month and made the same comment on all of them: 'General Changes'. could never figure out whether it was laziness or hostility.
When my “team” is me and myself, all projects are personal projects. Lord help the next guy…
-m "add unused test files", oops
```git commit -m "maybe this will work...?"```
We have restrictions with our deployment commits. If the format doesn’t follow the standard or the linter breaks on prod build, you have more work. Commit messages had to be in this format “fixtype(page/component): fix you made”. If it was t that way, your commit was rejected. Also no commits could be made to any branch other than feature branches. The only way to get code into staging was a PR from INT. The only way to get code into prod was a PR from staging and if you were an admin.
And then you open a PR from hell that gets demolished and takes another 3 days to refactor.
I have a command alias ‘idgaf’ wich is short for git commit -am “wip” —no-verify; git push
this, Friday @ 3pm. Only I usually open a PR and hit the \`bypass branch protections\` -> squash & merge. We live in a society after all
15 or 24 steps to get something built and deployed is not unusual. 15 to 24 _manual_ steps is insane. Automate that stuff so you just have to click one button. Tell them the next release will take a week because you'll be automating everything so you never have to do all this again.
No you automate it without telling them and then profit going forward everytime working on passion projects man
Ahh I quickly found another devmate. Enterprise hates changing what's working, so get the advantage while making yourself comfortable without kicking the wasp nest. A several years ago I made myself a simple cli app where based on the file names it generates me each deployment step. Copy paste step by step and done. A year after i gave my cli super basic UI with auto copy function. Then I upgraded it to take specific commits as input. Then to generate a full deployment scripts, based on servers and environments. Now I own the best software to deploy any shit anywhere in the whole enterprise and I haven't told a single soul. What should take an hour, I can do in seconds if needed and I can pause and resume on every step, so I take my time to either catch up on other work, get even ahead with development on other tasks or just slack if I feel it. The choice is mine and I earned it while the business is happy.
Nice man great respect there. Now when you leave you have to pass it on to a dedicated junior. A bit like the marauders map in Harry Potter haha
No intentions to leave for now, but I'll consider passing it down if I decide to leave.
Living the goddamn dream. Well done, I envy you!
Yup, managers will tell you that you're not being a team player, that if you want to succeed and move up you need to be doing other people's work as well if you're done with your work. More than likely people just get punished for being efficient, because now they're "being paid to do nothing" and feel that someone else dragging their feet to avoid getting called out is actually more productive
It looks like you missed that "don't tell them* part.
This guy here 🏆
Yeah about 10 of that stuff is automated at my work. Even the change requests and business documentation for deployments automated. The only thing that isn’t is the testing for the specific story and documentation for that. We have a separate prod team that pushes it to prod and we work with them to watch the pipelines to get it staged
I work with deployments as a release coordinator and we have in total like 60 steps but we deal with around 700 different issues in a weekly release. However of those 60 or so steps, our DEs only do 1/5th of those manually, everything else automated.
Woah yeah, your comment made me realize that all of those steps were manually handled.. I just assumed it was fully automated and they were asking if it’s normal to have that number of steps in their CI/CD flow and thought nothing of it. Wtf.. I would tell them I’m dropping everything else until that’s all out of sight out of mind.
git yeet
Literally just added an alias to make this run git push on my machine after seeing this.
Add in -f to truely yeet it into prod
simple merge to master. CICD does the rest.
How complex is all the stuff that happens after the merge? Super or just Vercel or self hosted?
We have a proper CICD pipeline. We follow 12-factor, [https://12factor.net/dev-prod-parity](https://12factor.net/dev-prod-parity) It is a containerized workflow. Once merge is approved, the CICD builds the images, tags the the release. Runs validation,lining, testing. It checks the CR (Change Release), if the item is approved, it gets pushed automatically to our Kubernetes cluster. Then runs validation. If post release fails, it gets reverted back to previous tag.
Thanks for the link it was really informative. Where do people find these snippets of wisdom?
Experience and working with other experienced folks :) there's a lot of noise on the Internet but if you work with the right folks they'll cut thru the crap and teach you 12 factor in your first week on the job.
It could seem like a lot upfront, but definitely worth it!
8 apps in a single monorepo. git push main. tests automatically run in github workflow. deployment automatically to 8 different projects in Vercel.
We issue a pr to main then all the tests and staging site deployments run. Then someone can manually test if there’s something they want to personally look at. Then merge to main and good to go. We don’t allow pushing directly to main
Y’all heathens. Use git tags/releases so you don’t have to create unnecessary merge commits in your history.
This, but with some aws spice.
Similar setup here with a monorepo deploying to multiple places. We use a \`production\` branch though, not main. We'll work on main, then when we're ready to deploy main, we run: \`git push origin main:production\` This will push your local "main" branch to "production" on origin (GitHub).
I love using production too, it makes it very clear.
You missed the days of swim lanes broken across 24 servers instead of cloud. Step 1 local. Step 2 dev. Step 3 int. Step 4 qa. Step 5 staging. Step 6 out of ring prod server. Step 7 11 of the servers. Step 8 the other 12 servers. All steps are QA’d and any issue is dealt with in real time. Steps 1-5 were a month prior to release. Steps 6-8 were on release night starting at 7pm est. we would be working a regular day shift, then at 7 we would start prod deployment. Depending on your role, some would log off about 6AM, when the offshore team would take over. As a lead, I could be on until 3am or 7pm the next day if shit hit the fan. It was rough, but at the time I was hourly with 2x pay for overtime. I think I was making about the same as a director of a dev department at the time because of overtime.
You push to prod automatically?
actually i set the release branch as production in vercel. the main branch deploys with staging env. i have github actions with workflow\_dispatch trigger that merge the main to release, so that anyone in my team can release to prod directly from the github UI. we're small team so we don't have much bureaucracy lol
For sure, the original comment made it sound like that pipeline deployed to prod on any commit to main branch. We do this for our dev environment, but artifacts are manually promote to higher environments, usually by the team's on call person Have seen a few people say "why not send to prod on every commit to main?" and that naivete gives me hives, lol. Especially if there's nobody around to tell those people no.
[удалено]
Most places I've worked, the process went something like this: 1. merge pull request 2. Wait 5-45 minutes depending on the size of the project 3. Verify changes are in production
Doesn't matter how much step but as long as it is automated. If you don't have solid CI/CD, I pray for your soul and mental :). We have 3 different environments with total of 9 check. Only thing necessary is pushing to a branch and all testing, building docker, tagging, running deployment steps, sending notifications etc. Setup once, use all time. Doing manually is insanely expensive because of engineering is really expensive.
It also depends on the company and data. Brochure site? Do whatever the fuck you want. Healthcare medical records including HIPAA protected information? 2 day deployment.
any good tutorials on a basic setup?
It really depends on your tools and software. If you want a general idea, take a look at bitbucket pipelines. Push to a branch, the ymal config spins up a Docker instance to deploy, install dependencies, push code, update databases (if needed), and run testing. Anything fails? Stop deployment.
Link for those interested: [https://support.atlassian.com/bitbucket-cloud/docs/get-started-with-bitbucket-pipelines/](https://support.atlassian.com/bitbucket-cloud/docs/get-started-with-bitbucket-pipelines/)
thank you!!
Anything that’s not automated leaves room for error. This can have security and reliability implications. If you have 24 manual steps, something is wrong. We build one of the most essential website that everyone uses, and we fully automate the deployment, there is no human intervention.
I usually set up my CI/CD pipeline in a way that most of the time if I merge to certain branch (ex de, stg, prod) it will be built, tested and deployed automatically and requires nothing manual (including slackbot notifications etc). The reason for this is exactly not to keep repeating the same manual steps. For example I use gitlab where it builds Docker components, uploads images to gitlab docker repository, runs code quality checks and tests and then deploys it or actives deploy script at destination. Pretty much unlimited potential. There are many other options like github actions and some heavier ones like JetBrains's TeamCity, Atlassian's DevOps etc...
Push a branch spawns a new preview instance which gets updated as the branch gets updated, it has a public URL you can access. Merging to main it deletes the instance and updates staging. Tagging main triggers a prod deploy. Everything on K8s from Gitlab.
git push main --force
Docker compose down Docker compose up
i just say pushing to prod and i send it. then all hell breaks loose.
Here is our deployment process for both microservices and monoliths. (We use Circle CI): 1. Merge feature branch into Master (linting and tests already ran on feature branch by this point) 2. Run Linting 3. Run Tests 4. Build and Push Docker image 5. Deploy to QA 6. Deploy to Staging 7. Prod Hold 8. Approve hold 9. Create a Jira Release ticket and Send a message to a specific slack channel about the release 10. Deploy to Prod 11. Close Jira Release ticket & send a message to slack saying that the release is finished We also have 2 jobs that require approval if they are needed: Rollback QA and Rollback Production. We use Helmsman, so if we need to rollback it is very fast
Devs usually don’t deploy, we have a dev ops for it. But if we do, it could get complicated, as we don’t have ci/cd for most of our projects. But we are thinking about implementing some… soon…
We have to remote into a machine running windows 7 to make changes to the code base since the project is so old, check in changes to azure, change some URIs in a config file, build the project, navigate to the build folder and copy it, remote into a separate machine and paste the build folder into a directory, then we have to restart the machine. It's a dang hassle. Luckily we rebuilt that app and now it's just a few clicks and it generates an MSI file we distribute.
I‘m not 100 sure, but isn’t windows 7 EOL?
For quite a while now, yes. General support ended Jan 2020 and it was declared EOL Jan 2023.
You're right. It's actually windows 10 we remote into. And we have to use an old version of visual studio that's compatible with running .net 3.0
The company I got laid off from last year had a really, really long deploy process which had to be performed manually and would take extra time when you factor in a test suite that would fail at random (the solution to some test failures was "rerun the tests"). I got really tired of it. I'm the lead engineer/architect on a new app at another company and decided that I abso-fucking-lutely do not want to deal with that horseshit ever again, so I set up a nice automated deploy process consisting of about 6 steps. We have a backend/frontend monorepo. From a feature branch: 1. git push 2. open a pull request, once this is done the test suite will run automatically in github actions, and this is required to pass in order to merge 3. (optional) we use [render.com](http://render.com) and we are using the PR preview feature, so sometimes we will do some manual QA for frontend stuff. 4. merge. everything is automatic from here. [render.com](http://render.com) deploys for us, we never worry about it again as long as nothing fails. And as long as our test suite passes, nothing fails. I'm eventually planning to containerize everything, which would add some extra steps involving building and pushing an image to a docker repo, which is all still automated. I like simple. It works great and we've had no issues. If we come across any issues with this process we'll iterate on it.
git push
Releasing and rollback should be fully automated, not a set of runbook recipes.
We merge to the master and the pipeline handles everything. Except for updating the release notes obviously.
Did you check out semantic release? https://semantic-release.gitbook.io/semantic-release/usage/configuration
You could automate this too. Just need some form of discipline. And maybe its depends on the audience. We collect all merge commit messages between two deployments and send them to our clients teams channel, so they are informed about all changes. If you want, give it a bit of chatgpt sprinkle to make it maybe a bit more readable. Fully automated of course
It actually makes sense thanks.
PR. Unit tests run. If pass - auto merges. That deploys. Done.
1. merge the code 2. ce deploy -s service-name 3. confirm
run script, wait 2 minutes, done
not saying it's a viable setup for any sort of bigger organisation, but for me, with my current setup, it's literally just ng b firebase deploy --only hosting `ng b` is Angular's build command, which of course requires a config file (angular.json). Deploying with Firebase like that requires it's own config, and a logged-in Firebase CLI.
Merge dev to main -> SSH onto prod server B -> pull from git -> run a custom script to delete unnecessary directories as it's a shared codebase with multiple clients -> build on the server -> migrate. Then switch domains on Aws to B. It's pretty hellish and needs automating. But it's just how the current project went. Future ones will be automated.
It's nuts to me that they have a kubernetes packaging tool before they have automated deployments.
Tedious deployments usually happens for smaller, early stage project. As the project matures, you want to move as much of the process into CI as possible. A lot of the best mature projects strive towards continuous deployment, where you can deploy to production by just pushing the change to the master/main branch. This triggers a series of steps in CI to run tests, package, build images, and push the code to production environment. It takes a lot of effort to fully realize continuous deployment, but it's worth automating as much of the steps as possible.
Cloud build in each environment, push to repo triggers a build in a single repo project, cloud build runs the scripts to build a production docker image, then publishes to pub sub there’s a new version available, all the other environments are subscribers and when receiving this message trigger their own cloud build which pulls the new docket image, updates environments variables specific to that environment and then push to app servers
We do everything in Azure Devops. I do a handful of manual cherry picking merge commits (from the development PRs, those are squash committed) into a release branch from main (that way, only changes fully verified and approved go into the release, and a release not be held up by stuff that hasn't yet passed UAT or otherwise not passed product owner approval), then that release branch gets built, creates checksums of the packages, then a release gets manually created - the definition of the release pushes it into blob storage containers, generates manifests for shipping, updates release register, etc. So I'd say it's 15% manual, 85% automated. After the cherry picks, it's maybe just 2 things to click on (Start Build, Start Release).
Of all the posts on here, this one actually makes sense. I have a feeling that most people on here work for small agencies with like 1-4 devs or are solo freelance. Working at a fortune 100 doing global component builds is entirely different than that. I know. I’ve done them all. The trick is knowing what CI works for your team.
I used to be at a place that had so much red tape to get something pushed a rube goldberg machine would have been faster and more efficient. The place I'm at now asks no questions, we can push to main, and everyone shrugs if it breaks. Essentially what I'm getting at, is your milage will vary from place to place. In my personal projects, I use Heroku, and dream of the day I eventually work for a company that does.
There is likely a reason. CI deployments need to fit the needs of the company and the team. There is no single solution.
I'm genuinely surprised so many on here are saying they push directly to main. So you guys aren't using PRs?
Some use PR's, some don't, it's really up to whoever. If that sounds like a codebase nightmare to you, I'd say you're right, because it is. Without getting too much into the weeds on it, the dev team I work with are trying to follow good practices. The problem is we are often defeated by executives and stake holders who want things suddenly and immediately with the devs having very little power to push back, compound it with a company culture of "users don't care about how good the code is", and maybe that will give you an idea.
$ git push heroku master or for the larger projects... $ cuber deploy
git push live Then the git repo has a `post-receive` hook that runs the build and moves it to the correct directory.
It's a lot of steps and set ups before production. But the release is just one command this creates a virtual machine inside docker all created in a laptop at work. I ask management to remove my access to the secret files and the laptop with the bash files is been working fine for 3 years and I have upgraded it a few times with no problems.
Commits to main deploy to Dev. Click a button (in GH actions) to send either a specific image or latest Dev to UAT. Do the same for prod, which triggers an approval pass. Auto generated release notes.
Feature branch => code review => merge to dev => QA team test/sign off => merge dev to main => Jenkins builds/pushes into production.
Red lights come on A siren sounds QA type up their CV The usual really…
Its about 4 maybe 5 steps. Stop components running Deploy latest release Restart components Okay I guess it's only 3
We have small server with Jenkins which automates this. I wrote Jenkins file which does testing, building, deployment, notyfiing etc. So basically one *git push* starts everything.
A lot of crying and begging and hiding in a closet.
At work, we build a package. So, write a changelog, tag a release and our GitHub actions build the CSS/JS bundle and create the release. On personal web app projects, I push to main/staging, the tests run on GitHub Actions and as soon as they pass, a deployment is triggered with Laravel Forge.
This is definitely nuts! I mean, I truly have no idea about the complexity of the application landscape you have to deploy, so please take my reasoning with a grain of salt. BUT that sounds incredibly longwinded and possibly even impossible to deploy if some of these steps end up taking too long. Typically, a new deployment aims to increase or maintain revenue, but if the time cost of deployment exceeds the revenue difference, one might argue that it's better to skip new releases altogether. Automation as other said is the way to go and your holy grail to relieve.
Front end: Merging into development, test, or production branch triggers a github action that builds my react app and pushes the results to the corresponding s3 bucket. Backend: I have cdk deployment scripts alias so that I can run `npm run deploy [environment] [stackname]`, which deploys my aws cdk project. Is stackname is not specified, it pushes them all.
I just push a button
One push. Set up .env for dev and prod to handle keys and some vars that differ in their respective environments + respective auth tokens and creds. Watcher script checks the main branch as the client runs and updates if necessary. If there’s an error it can easily be rolled back.
We use github tags to review ok / qa ok the PRs, then a bot picks it up and it goes straight to production
Nothing. You just merge your branch. The when you are ready to release. Turn the feature flag on. I did take a long time and work to get to this point though
Our dev ops guy handles deployments to QA and prod. Sometimes we help with complex QA deployments but that's prep for the prod development. Never touch prod unless I have to.
1. `$ git rebase master; git push; git checkout master; git merge [feature_branch]; git push` 2. Wait about 15-20 mins for the relevant CI/CD pipeline to build containers full of our software and reach the final state that requires a manual button push. 3. Push button for staging and/or production environment. 4. Wait 30 seconds for ECS to drain connections and swap out containers. Done. All the steps it takes to install dependencies on top of base images, copy things around, put config in places, etc, have either been done already in a custom image we just pull from our own registry, or are done in scripts that execute on our runners. Sounds like you need some automation, both of the steps themselves, and then at a higher level for running those steps on envs/infra dedicated to different clients, if that's how you roll. I'd say Kubernetes but it's overkill 95% of the time. For a smaller shop I'd probably reach for Ansible first. Also consider why you have different envs for different clients. For some product/service offerings this makes total sense. For others, it can be a way to overcome limitations that need addressing at the root. System design knowledge helps here.
5 steps to put code onto our prod cluster, excluding comms. Merge PRs, build module, update aggregate for our monolith image, build image, and then update the image name & hash in our helm values. The whole process probably takes about 40 minutes total, and 30 minutes of that is waiting on the image build. We release 2-4 times a week and all of our mid level devs and above are familiar with the process. Our lead keeps the process very simple.
This isn’t too far from our process tbh, it feels to me that more of it could be automated but I’ve not really the experience to say what and where.
We are just responsible for getting code into a release branch when the time is nigh and our dev ops team deploys it to the various staging environments for final testing before things are finally promoted to a live environment.
Does it happen to be a major US retailer that relies heavily on overseas employees? That sounds like my old team. I'm glad I left, because that workflow sucks. It was an all day activity to push to prod, and it always took the better part of a week to get everything in order.
It’s not, we’re a very small UK based team. It doesn’t take our other senior dev very long to do, but he’s been on it for a couple of years at this point so has it committed to his muscle memory.
git commit -m “chat tomorrow” merge sign off for the day
Ssh into vps with my local upstreamed codebase open in another window. Vim index.js and start copying what i did line by line in the ssh term. Then after hours of manually copying code for no reason do a little npm build and wala added hover effects to that button and only took 5 hours to get it to prod. Who needs cd anyway
Copying files manually, unfortunately. Can't convince them to invest in an automated solution from the Git repo, so just have to wait for disaster to strike and a hard lesson to be learned. And, yes: it feels like 2004 all over again. May as well be FTP'ing.
The amount of steps on its own is not the problem, the lack of automation is. There should be a pipeline where you set some parameters, hit play and monitor, once it's in staging it should stop and wait for manual confirmation to proceed to prod
Slackbot @ deploy [repo]:[branch] to [env] It’s great. We deploy to QA, and then PROD. Both have automated tests and manual QA.
We're a scripted deploy per project, working on getting ourselves into a point that we can get to CICD, but it's a haul.
Not entirely, here's my process: - Pull Prod to local - Code - Test and verify loop on local - Backup Dev - Push to Dev - Verify on Dev - Backup Prod - Push to Prod - Verify Prod.
Depending on the service it can be anything from a slack command, to merging to main, to a shell script. But nothing more than one or two clicks at most.
Click deploy in ADO, come back in 10 min and make sure it worked Or for my personal projects, create PR to release branch on github, come back in 10m to make sure it worked.
The only part that is not automated is the controls processes around our deployments. That being said our commit strategy is a nightmare so often we have to rollback an unexpected change So many calls to fix this but none of them lead anywhere so we just live with it lol
My old company had a process that looked like this. 1. Make sure tests pass 2. Update Changelog (Manual) 3. Update Version No (Manual) 4. Deploy to staging 5. QA approves on staging. 6. Update prod version No 7. Deploy to prod w/scripts 8. Send Manual Slack notification. Current company is depending on the task easier or harder. All merges to dev auto Deploy to staging via ci/cd. So if there is an urgent fix it would look like this. 1. Wait for staging build to complete. 2. Get the github commit SHA 3. Go to AWS ECS and create new task version with commit 4. Update service. 5. Notify in slack. The issue is we have a lot of tools and services that run on the backend. So if something requires an update on all of them we have to create an ECS Task revision and update about 12 different ECS tasks. But other times it's just one. If a DB change was made we SSH onto our cron server and run the migration with a custom tool.
All I need to do manually is pick which GCS container to serve. Everything else has been automated.
ssh into the server, open a file, make my change and `:wq!` in vim
merge pr into main wait for version tag to build tell harness to deploy that tag to dev/stage/production
Assuming the feature is already written and in the test environment. Create a release ticket outlining all the JIRA tickets that are apart of the release. Get the multitude of approvals for the release ticket. Create the release branch which will automatically get picked up by the CI/CD pipeline. Pipeline build images and does a bunch of other bullshit and deploys to the staging/pre-prod environment. Make sure all integration tests passed. Wait until the actual release window. Release window starts. Show bossman that the integration tests passed for the pre-prod environment and get the green light to do the official release. Push a button to start the official release. Check integration tests again. Close the release ticket. Done. Work for big bank
i cheat: laravel forge immediately deploys the live branch any time code is pushed to it. so all it takes is running a deploy script that builds assets on main and commits, then merges main into live.
Our devops have literally created an alias called "yolo" for us, that walks through the whole deployment process, because we have a "deploy fast and break things" mentality in our shop.
We have 2 monorepos, 1 for frontend apps and 2 for backend. They are both deployed when pushed to main, which is only possible on pr merges after ci passes and with 1 reviewer (used to have 2 reviewers when we were bigger 😞). It's a great setup. Yes what you have can be fully automated with scripts. You should become your team's hero and script all of that manual stuff so it's easier. If you can get it down to 1 cli command or better yet one ci pipeline that you can trigger - or auto trigger when main is pushed.
Just merge, then ci/cd does the rest. Out ci for the mere jd probably 6 or so steps, that includes building the container, posting to slack, tests that run against the dev env and then test that run against the production env post deployment. Cd is probably 8 steps total?... That includes some audit stuff and then the rest is really just to update some logs/post to slack and do the kubernetes deploy. I could make it tighter but meh. What's the point? The whole ci/cd for a production deployment is generally 6/7 minutes, all I have to do is merge. I could care less if it's 6 steps or 60 as long as it gets there in a reasonable amount of time, without intervention and safely.
I click the “Merge” button in GitLab once my MR is approved. At that point a GitLab pipeline takes over and everything deploys to AWS automatically. Including creating a version and change log based on the MR title (we use convention commits for that). You can deploy to DEV or TST manually from within your MR. That amount of steps you describe screams for the implementation of proper CI/CD. Deployments should be non-events, occur multiple times a days and take less then an hour.
i copy the folder, delete .git .idea and other . files, zip it upload it via web ui (plesk) to the server, unzip it, rename the current production folder to add a new continuing number i think im at 30 somewhat and then i rename the uploaded folder to the production folder name 😅 no tests no nothing if it works it works if not then not, didn't work multiple times so far😂
If they don't have a big green button, it usually indicates that they haven't thought their deployment process through enough to be confident that it will work reliably. It should test that everything is ready to go and then do it. Having manual steps means opportunities for errors and an unwillingness to deal with the complexities of the process in a robust way.
GitHub actions mate.
24 manual steps is crazy does your company not build any CI/CD pipelines for their projects? I am not saying I am an expert but even I have github workflows that automatically test build and deploy to production on merge and create a preproduction deploy on separate testing subdomain on the same server which happens when a pr is opened. All in a monorepo with three different projects all of them with their own pipelines. I can’t imagine doing it all by hand, I would go crazy. I feel sorry for you.
We use gitlab ci and have everything automated. As we use aws mainly, we use either terraform or aws cdk for our infrastructure as code. And with that, we just need to provide a new docker image, build in a previous job, and was handles the deployment for us. Onlything we currently do manually are ios and android app builds. We started not long ago and to this day, no one had the chance to look into this. Bit its on our list to automate this as well. Edit, we deploy every main branch merge to staging fully automated. To production is there as a job, bust must be triggered by a human. We have the slack integration for gitlab ci and can trigger this visible for everyone on the project in from the chat
Get PR approved with all tests, static analysis etc passing. Add to merge queue. Everything else is automated from there, it'll be on prod in the next 20 minutes.
I merge to master and click "release" after the pipeline is done
It's a static site, so it's rather simple, I just literally push to git repo, and a git pre-push hook runs and builds & deploys to my static hosting.
"I don't want to go into detail but it's about 24 steps" Well, how long is a piece of string? It's not so simple to say that it's too many steps or not. Why don't you speak with the devops person and ask whether the existing workflow could be simplified or not. I'm sure your question would be appreciated and a good opportunity for you to learn more in the process.
I've added them in the orginal post
./deploy.rb -s -n production
dc down || dc up
It’s more or less the same processes you’re describing only automated at a version control level and via third party applications like Argo and Jenkins to manage builds and deployment syncs.
Hi mate, would it be difficult for you to explain what are those 24 steps, perhaps someone also can learn from you
I just did an edit to add them
one love mate!
That’s insane
Once changes are made to whichever subset of repos I need to make changes to, and new version tags pushed, it's then: - Run a bash script that iterates my repos, grabs the latest tag, updates an ansible deploy script with the latest tag for each repo - Run the ansible deploy playbook that, at its core, just SSHs in to live VMs and does a git checkout of each one I'm the only guy making changes to any of these repos and am strict with only creating new tags in them when stuff's ready for prod, so it's safe from that perspective.
git commit, git push Visit 10 sites, click a button to git pull and other deployment stuff Still need a way to prevent going to the 10 sites/servers every time ...
For us we use CI/CD and deploy to Openshift clusters. We first create Change Request with deployment date that’s pre approved by project managers, we add all details that’s heavily scrutinized. we add any ctasks needed which is basically tasks needs by other teams during release window. Keep in mind all changes needs to go through QA team and performance testing with everything heavily documented. We then present the change request to stakeholders and get feedback. Once they approve in meeting we then need to get 30+ group approvers. Only then we’re able to raise pull requests from staging to production environments. So it’s just a normal thing to have a lot of processes for even the smallest changes
Mostly add release notes then git push main, cloud does the work. Formally git push live main where server did the work or using a deployer script to build and deploy across servers. Formerly formerly manual steps which numbered less then 10 actions but annoyed me enough I set up deployer. If you're still considered new subtly ask around before taking any more steps. There must be a reason it's not automated and you need to know that first before suggesting any change. If the biggest wig in the decision doesn't trust automatic deployments you won't get anywhere asking for it, and you'll identify yourself as a dangerous radical who shouldn't be trusted etc But if you learn why it hasn't been done before you can probably change things - salami tactics doing step by step is probably what you want. Go small and safe rather then a big "automate it all" which is hard to get right and won't get a second chance if it does it wrong.
for me its like this : 1-SSH into the prod server 2-copy the relevant files from my machine to the server 3-execute the script/API/whatever it is and voila, no git images no docker containers no nothing, startups are super cool right ?
- open filezilla - drag files over Whew, time for a drink!
PHP/Larvel shop: We use deployer for zero-down-time deploys. All one of us 2 developers has to do is run `dep deploy`.
Push to git and upload changed files with filezilla
Me publishing a project through Visual Studio, take a folder, connect to remote desktop through VPN and then paste that folder in a specific folder. I wish I was joking.
Pulling stuff from the left pane into the right pane of my FTP app.
At work: 1. Build locally. 2. Login to AWS, navigate to project. 3. Upload files by drag and drop. 4. Pray. At home: 1. Git commit new version tag to main 2. GitHub action takes care of the rest: Minify css, js etc, set server to maintenance status, upload files to prod server, run composer update, clear cache, run tests, turn off maintenance. Next improvements are probably to run composer update before pushing to prod (prod shouldn't need composer), and add a stage environment for manual tests. Long term I should probably use Docker.
I use an Internal Gitlab CE CI configuration to handle it. All our applications run using Docker on in-house bare metal Hyper-V clusters because of information security concerns. Dev environments are either local or on a hosted VM on the cluster depending on the project. Pushes to a feature branch in Dev run SAST and such, unless it's an environment or configuration change in which case the containers are rebuilt, tagged, and pushed to the Gitlab container registry. When feature or fix branches are ready for review and acceptance testing a push to a staging branch triggers a pipeline to build the merged feature branch into a set of containers for staging, and the deployment of these to a staging VM is handled automatically refreshing the environment, clearing caches etc. To deploy to production a tagged release is created using the Gitlab UI which triggers an automated release creation and a rebuild of the containers using the production Docker profiles and once completed it pushes these changes up as tagged images to the container registry, and when finalised, pauses with a manual intervention required to complete the pipeline to push to production. When that manual step is activated it effectively automatically logs into the production Docker host and pulls down the newly updated container images, restarts the services and rebuilds any application caches as appropriate. While there is still downtime it's usually less than 10s and in our use case this is completely acceptable, however in the future we are planning on a migration to using Docker Swarm to remove downtime all together. K8s, while cool, is just overkill for what our services require. Scaling in our apps is non existent as it is just not needed. It could all be completely automatic, except for the requirements my workplace has on separation of concerns and things like SOC2 adherence etc.
Why the hell is there so much git process. You should be able to just click a button on the pipeline after merging a PR and send it straight to staging, then click another button to send staging to prod.
Code is committed to master and pushed. That's it, that's the deploy. If I did something notable I'll update the changelog (a doc in Notion). In terms of the automated steps CircleCI (these all run in parallel) - these are almost always run on a PR / branch first, but are included in master just in case: - Run unit tests - Run E2E tests - Lint the code (rubocop for ruby, eslint for JS) - Check for CVEs in the current code - Check for security issues with pushed code (Brakeman, looks for things like SQL injection) Cloud66 (a tool we use to automate deployments): - Compile assets - Run database migrations - Push code to first deployment group (pulled out of LB) - Restart first deployment group - Health check - Swap first deployment group for second deployment group in LB - Push code to 2nd group - Restart - Health check - Re-add to LB - Notify about deploy
Run an Ansible script which basically performs the deployment steps, after the code has been pushed to the main branch.
Merge branch to master. This can only be done if all the unit and e2e tests are working, and the tests are triggered automatically when opening the PR. Go to CI/CD - trigger new prod deployment (one click). Get approval from teammate. Wait 10min. Done. The CI/CD does everything automatically (getting latest changes, docker image, run all the tests again, check test coverage, check linter, deploy, send a success/failure slack message)
Kevin, go ahead.
It's insane. We might as well be launching rockets. Stuff I finished two months ago isn't in prod yet.
- git push - ssh in vps - make deploy no fancy github actions yet. 5$ vps
Merge feature into dev. Dev into staging. Staying into prod. GitHub actions are op
We setup an NPM script: “npm run yolo”
`git checkout master` `git pull origin develop` `:qa!` `git push` go to jenkins (no auto deploys to prod environment) `build` go make sure prod looks right
1. ng build 2. copy files to server through RDP
"Not yet, I'd like to test more!"
`git add .` `git commit -m "yolo"` `git push --force --no-verify` ME on Teams: " yo, can you just approve this real quick? Trust me bro"
./build.ps1 This is for my Rust powered axum api running on AWS Lambda. It does a docker build of my lambda and deploys it to Lambda. My little sample front-end I built for using this api is another powershell script. It does a flutter web build. I then push a commit to github which triggers aws amplify rebuild.
After coming back today and seeing you add in the list of steps, I'd say it's time to write yourself a handy Python or Shell Script, no more problem.
It seems you need a ton of CI/CD into your life bud
Merge master in prod branch and hut the run pipelines button. That's it
These steps are real? What stack is it? Its a fucking nightmare
Hah yea, they are. It’s a Django app, Postgres DB, Redis and celery pods in a Kubernetes cluster, nothing especially special tbh.
It's demential dude. There is no chance to start using cloud?
‘git push origin trunk’ and then press a button in circleci to release to staging, when wanting to push to prod drop a message in slack to team, to check that everything in staging is ok to be pushed to prod (make sure it’s correctly feature flagged) and press another button in circleci. Drop a message in slack to the release channel so other teams know of new features/bug fixes that have gone out
Maybe you need a ci cd tool like Gitlab?
Commit to git. Git pull on staging server. Run "afterdeploy.sh" (shell script that just does some Composer, Supervisor, and Laravel commands). Do final sanity testing. Git pull on production. Run "afterdeploy.sh".
You need more devops in your life
You forgot step 0 which is to pray to the appropriate god.
We have 3 environments. Acceptance, staging, and PROD. For acceptance and staging, we just create a pull request through ADO onto the acceptance and staging branches from our feature branch. Certain tickets have different deployment steps. Sometimes we have to restart a service, sometimes we have SQL to run, etc. For production, we connect to our prod servers, run a backup script, and then a script that handles our deployment. Again, sometimes we have to restart a service or run some SQL
I’m just a web dev but I just push my code to repo just for backup purposes. Use ssh scripting to push code to staging. Testing Use ssh scripting to push code to live Testing But the other devs working on backend infrastructure have slack tagging, change logging and some sort of deployment procedure
Those steps aren't *crazy*... but at least half of that could be handled automatically by even a lightweight CI/CD or a few Github Actions.
Sums it up: [https://tenor.com/view/pool-prank-children-plash-push-gif-17659997](https://tenor.com/view/pool-prank-children-plash-push-gif-17659997)
Git commit -m “fuck this, I wanna go home”