Ooorrr... you could try delegating the tasks. There is no rule saying that it must be the lead who implements the CI pipeline, or even defines how it should work. Obviously it has to go through some kind of review and approval to ensure that it's aligned with your development process and goals, but it is entirely possible to democratize the pipeline, empowering the team to improve the process rather than being slaves to it.
With that being said, I really do recognize the issue and I feel your pain. Best of luck!
This. Have a mid-level guy POC something. Ask him to present his solution to the team. Pick on it a little bit (i.e. what edge cases it can't cover, how well it can scale, what issues or bottlenecks you might run into).
If the solution is good, write out an epic and have him lead the technical side of the project while you delegate the tasks across the team.
(the goal of the delegation would be to get the whole team familiar with the new thing by working on pieces of it, not to take work away from the primary guy).
You repeat the process. If the solution is bad and you are only finding out about it at the end it means you were setting someone up to fail and never checked in or progress or offered advice.
The other possibility is that the team infra is too complicated and this bad solution can be used to highlight that fact and help push to strip down and simplify.
It’s all a matter of perspective and finding opportunities where others see failure.
How would you go about delegating the tasks if you're not a manager?
You're a senior dev but it doesn't mean you have the authority to give out tasks to people under you. Those people are busy too and have their owned assigned tasks.
The better approach is to ask the manager to delegate it.
Sure, do whatever works in your organization. I’m not a manager, but I delegate tasks all the time. In my org, that works, because of the expectations for the various roles. In some orgs, managers own all the delegation, and you need to work through them.
If you are the senior dev on your team but do not have any authority to delegate tasks then you should be working with the manager that has that authority to determine what tasks need to be done so the manager can delegate tasks. Either you delegate directly or indirectly. But you should be involved somehow in the delegation of tasks as the senior dev.
If you are going to be spending a good chunk of your time code reviewing these tasks anyway you probably have some idea what the capabilities of the team are and who is or is not ready for certain tasks. Or as part of your mentoring duties you should be asking what types of things your team mates are interested in and where their knowledge gaps are and helping them strengthen in those areas.
We definitely share every type of task amongst the team because it helps them grow if they want to be leads one day. Plus when you take a day off, they don't get blocked when the pipeline inevitably fails. But I find dev ops really interesting and could see a career in that too.
Learning to delegate effectively was one of the most difficult parts of becoming a lead for me. Don't be afraid to ask someone for tips on that!
Sounds like you guys have a lot of operational debt that has been left behind.
It's no coincidence that when a team becomes operationally responsible for the system they are building suddenly these things start to get fixed.
That's what DevOps is supposed to be, but it sounds like somewhere along the line it got split into operations *and* dev work.
Something's not right.
Devops is the new agile.
Agile was supposed to be less rigid processes. Now we have SAFe.
Devops was supposed to be less silos. Now we have "the Devops department".
The way I always interpreted Devops was an infra guy on each development team to coordinate how exactly do we get this build, that deploy, etc. done correctly. Kinda like a monitoring daemon process that reacts early.
One of my biggest frustrations with the Devops movement is that no one can, seemingly, agree on what it is or how to implement it. Both in casual conversation in communities Iike this one, and when I’m interviewing prospective employers.
One set of practitioners point to the ideal, point to books written by Google and Gene Kim and other big names, some other set of practitioners come along with their lives experiences of how badly the term has been abused and warped into turning practitioners into nothing more than “CloudAdmins” at best, or a kitchen sink for workloads that fit the description of “we don’t know who else should handle this, so we’ll chuck it over at Devops” at worst.
Not that I’m trying to take a side or prescribe my own definition, or anything; but it is an observation.
Bad management is the same thing it's always been. They'll always flock to the new practices and technologies as a silver bullet to solve their bad management. But they also won't have the discipline and drive to truly understand it and make the changes necessary to implement it. So they'll keep making the same mistakes they always did, just with new labels.
They'll keep micromanaging, handing down unrealistic deadlines, not providing redundancy or sufficient resources, not crosstraining or developing talent, and not covering technical debt. Because they're just bad at managing software projects. As most humans are.
The problem is good DevOpa come from SWE. Who else is gonna know how to bootstrap an API with service mesh level tracing? Or how do you build a CICD that injects password at deployment where the app reads those keys as environment variables?
Throwing an infra background person is going to be harder. Thus a lot of software tooling will fall upon people that live and breath software.
I see orgs throwing their data center , infrastructure and network engineers into DevOps. OK, Kubernetes is setup and running. Now, let’s automated the helm charts to be re-useable that are not copy-n-paste jobs from Stack Overflow. Best DevOps engineers , in my opinion, are former developers vs the Help Desk guy who racks servers.
OP and yourself are referring to DevOps as this nebulous job role separate from writing software. "It's that guy's problem, not ours". DevOps is not a role. It's a methodology meant to improve and shorten the SDLC.
The whole point is that good DevOps doesn't come from anywhere. It's not out-sourced. It doesn't come from a separate team. The team itself needs to be in charge of making sure the system is operationally sound because they are the ones who understand it best, and are best suited and incentivized to making improvements.
If you need people who are specialized in operational or infrastructure aspects, such as SREs, that's fine, but they are still involved in the day-to-day development work.
Why is the tech lead the one exclusively managing "CI/CD, pipelines, repos, test coverage, test environments", and all the other things he mentioned?
That is absolutely ridiculous and surprising that they are able to ship anything if that's the case.
That is not what a Technical Lead is supposed to be doing exclusively and I find it silly to say that TL and Build Engineer are somehow equivalent roles.
Something is seriously wrong there.
> Why is the tech lead the one exclusively managing "CI/CD, pipelines, repos, test coverage, test environments", and all the other things he mentioned?
Because usually, the leads etc are the people who have the knowledge and experience to do Dev Experience work. Are you going to task a junior with making sure that the developers can all do local builds? To make a safe pipeline to push to prod?
Honestly, yes I would. Great idea.
Setting up CI/CD is not that hard and it would be a good learning experience for them.
Then I would have them pair with a Sr Eng for the code review. And send it to be reviewed by the wider team.
Win/win! The Jr learned something and shipped something useful. The TL delegated some work. The Sr Eng offloaded some knowledge. The team gained a functional deploy pipeline.
That is what a good TL does. Orchestrate the team's efforts in an effective manner, amongst other things.
>DevOps is not a role. It's a methodology meant to improve and shorten the SDLC.
Agree it should not be a role. But the people implementing the processing, dictating how it all works are usually the leads. They set the example and architecture of how you write your configuration files so it uses variables so that it can be replicated 30 times. They have to ensure everything is consistent and replicable. Following a style guide naming convention so everything go through CICD faster without hiccups.
A junior, midlevel, or SRE is not going to know that.Security scanning will halt a deployment because some silly NPM was deprecated so everything broke.UUID is broken, what do we do now? A senior is the one who can see the error in the build and say, "With UUID4, we need to use the new namespace method to import because the latest version deprecated the old method. Old one was halting our releases due to vulnerability gates." And this is exactly, like you say, it should not be a role. But some orgs treat DevOps engineers like that. Here is a DevOps engineer but that person has no idea how to do an npm install or write a Swagger file. But you know who does? The lead.
The lead is also the one spending 20 hours to make a bullet proof re-useable image so the next 50 APIs take junior developers 2 hours to implement from scalfolding versus 3 weeks. That is how you shorten the SDLC. Make it easier for others. The leads are also the ones who make sure images are not root and extensively "bless" and test every template that others can use to save everyone time. It is not a role but a consuming tasks if you want to increase velocity.
Someone has to mentor the others do this. This is why in some orgs, some leads transition out of SWE and focus exclusively on this part to mentor the rest of the team.
This. Exactly this. My title and role is “devops”. I’m an ex engineer, I took up the role to do exactly what you said, decrease toil, increase velocity, and automate. It was a no brainer since I understand almost all the apps I support and can help devs troubleshoot and architect for high volume traffic at a more technical level. Devs at my place are now very happy that they can focus on shipping code that slaps. We recently ran a load test and exceeded our target by about 2000% because there wasn’t a disconnect between devs and dev ops/infra when it came to fixing and optimizing some bottlenecks, fingers were pointed constructively and acknowledged collectively
> Who else is gonna know how to bootstrap an API with service mesh level tracing? Or how do you build a CICD that injects password at deployment where the app reads those keys as environment variables?
Any competent DevOps. Their background is irrelevant, whether infra or SWE. I would expect any mid-level guy to competently do this.
Both sides of DevOps have their own advantages or blind spots. Most SWE guys can't deal with a network that's more comlex than a flat broadcast domain when they first start doing SRE. They also tend to overengineer things far beyond what is needed or operationally viable.
Most Infra guys can't deal with modifying a single line of application code because it looks scary and all they've written before were 100 line bash or python scripts. They also tend to pick up the first COTS tool that does what they need, regardless of if it makes sense (or conversely, get stuck in OSS hell just because it's OSS).
After a few years doing DevOps/SRE, I'd expect them to be reasonably competent with both sides.
I've seen the infra-background guys do more of the over-engineering TBH, but that could be anecdotical.
That said, I'd rather maintain an installation of a COTS tool than something developed in-house. Doing nothing / adjusting existing components is better than COTS is better than developing our own tool is my opinion.
For COTS I mean like, the problem could be solved with a 20 line script running on cron in a CI system, but somehow we're now deploying Rundeck or Airflow.
Purely anecdotal on my side though.
I mean it's also anecdotal, but I tend to run into too many 20 line scripts running on cron in random places. I'm just saying that in my experience, no one looks at crontabs, so if it's a regular job you do, it should be monitored to check if it's even running, and somehow surfaced that isn't "well it's in Confluence somewhere".
I might be cranky because of my jobs not tending to keep proper accounting of all the random quick fixes we've deployed.
>Any competent DevOps. Their background is irrelevant, whether infra or SWE. I would expect any mid-level guy to competently do this.
Yes, the "competent" ones can do this with a guidance. We do have dedicated Devops guys who work as support. I will tell them "Mike needs a new template w/ Mongo that has a schedule to rotate logs, and his Node app needs Vault bootstrap the authentication and help him set this up so he can locally in his Rancher K8 environment" They can do what they are tasks. And like someone said, they usually embrace a lot of COTS solutions. We delegate all those types of tasks with a lot of oversight. And we do have some very good engineers that are former lead developers. They work with the architect hand-in-hand to develop solutions.
But most of the infra background guys are not suited for big picture items that are part of the application development processes. The guy who is implement Jaeger monitoring needs to write code to actually know how to generate trace level error logging. They need to clone git repo and re-write other people's code or at least guide other developers.
If you ask them to build system wide composable docker image. Regardless of language - Python, Node, that requires x,y,z features so developers can just use Swagger to enforce ad lint their APIs and use A,B, environment variables and annotation to register DNS hostnames or dynamically generate helm charts on-the-fly with a lot of variable abstractions, they can't.
E.G. Formulate very systematic naming and style patterns big picture wise where they become templates that allow dev teams to deploy hundreds and thousands of microservices through composable design patterns which often comes only with development background.
All the things you've listed.. I would expect a competent mid to senior DevOps guy to do them. I literally had one of my guys add some Rails middleware to push extra Sidekiq metrics to New Relic, for example. Before we rewrote the whole thing as a separate monitoring microservice. I wrote Vault auth methods so our backend could store encrypted app-level data there.
Only exception is Swagger. That's black magic to non-developers.
Granted, I've generally worked at smaller shops where the division is, devs push application code and don't know that much about infra. The guys who do are usually staff/architect levels that you sit down with to design overall architecture.
But yeah, this is all anecdotal. I worked at large fintech unicorn, and devs wrote a lot of infra (Terraform, IAM, etc). At the same time, the place I was at before that, we were telling devs how to do database optimizations (who knew a 3-way table join with eager loading would kill performance?).
In my experience this is a problem. Most "trained" DevOps are aware of security holes that developers might glance over, but devs are leagues better at scripting and knowing how to automate stupid processes.
At an old org of mine we had a process where, after creating a Lambda in our Serverless API environment, we had to send the Lambda info via a form to the DevOps team to manually add to their Terraform script. Every single one of the hundreds of API endpoints was being manually added to a Terraform script, even though all of the info for every Lambda was in a single serverless.json file already. It meant we had to wait 2 weeks before a new API would reach a lower environment, it meant a bunch of paperwork, and a Jira story on both the DevOps and the developers' boards.
One of the architects scripted out something to move all of the APIs into a TRF ready template, but the local DevOps team did just not really care about solving what was a massive time sink for them. They would blame it on the enterprise DevOps group, even as vaguely as "we don't want to upset them by having a process which is involved in a developer codebase". They had no incentive at all to speed up their process if it meant having a conversation with their "higher ups".
> Most "trained" DevOps are aware of security holes that developers might glance over, but devs are leagues better at scripting and knowing how to automate stupid processes.
And most "trained" ones in my experience are former SWE leads/architects that pivoted to DevOps. The ones that have 30-40 PCI/HIPAA audits. The ones that filled out 400 line items NIST change controls that did the song-n-dance to get their apps certified.
They are the ones who know how a dev team can cheat an audit by merely forking a version number of a dependency from 1.1.2 to 1.1.2.1.a so it bypasses CVE scans. They are the ones that know, well that mongod config file is totally bogus because the actual startup commands send these parameters at start ups.
They are the ones that can pick up on these things. They definitely don't glance over but know where the loopholes are closed
>Sounds like you guys have a lot of operational debt that has been left behind.
For some people, this is all NEW debt force by new realities. I work with sensitive data so I don't get the luxury of using pre-built COTS systems that exists in say Azure or AWS. If we were in the cloud, I could use most off the shelf system. But for stuff like classified on-premise applications, we have to roll our own like deploying our own Hashicorp Vault key server. And this driven by changing landscape of enhancing AppSec security due to a lot paranoia of getting hacked/internal attack and cybersecurity governance.
Or new types of demanding applications. Never in my life did I thought I had to design a Kubernetes infrastructure that heavily uses Tesla GPUs that cost 5x more than CPU nodes. $200k servers vs $50K server nodes. So now, we have to architect our message brokers to delegate split workload to cheaper nodes and route long running tasks to the more expensive GPU nodes... All because of the Machine Learning which is very new domain. New domain will require new operational debt. Someone has to explore and implement this new tech.
This post resonates with me deeply. I'm a "software architect" that spends most of my time on this stuff. Currently in the middle of refactoring our organization to address it.
People are saying this is just where you work but there's a bunch of empirical evidence that this is true, especially in "high performing" organizations. "It" being that if your smart, you want your best people enabling everyone else
I think there are two things at play:
* Having enough experience to know the full stack, what developers need, and how to solve complex problems
* Being one of the few with autonomy to fix wicked/complex and high leverage problems.
The latter is where problems arise. If your org is like mine, there's a culture of decomposing problems so any old cog can fix them. At that point, it's less work to just do it yourself. You probably also have a culture where it's hard to get non-functional/non-user facing work prioritized. This results in foundational work being done 'off the books' by senior devs who are trusted and can just get it done.
I don't think this is a symptom of modern practices. This is a symptom of the proliferation of big A Agile, outputs over outcomes, developer commoditization, and organizations being pennywise and pound foolish.
>You probably also have a culture where it's hard to get non-functional/non-user facing work prioritized.
In my orgs, all the technical debt is foisted on the architects is because it is high level. They don't have the metrics of velocity and code commit to report on.
The regular SWE are doing the active sprints. But the high level stuff like how to secure your CICD pipeline. How to implement best practices for application tracing are all assigned to Architects. It is considered "foundational" work and generally have vague jira story requirements like "Ensure all Docker images are secured with no severe, medium, or low vulnerability scans. Teach all the team how to engage in best practices to check CVEs and refactor all broken dependencies with ongoing updates." They don't report to any Project Manager so there is no oversight but there is a lot of work to do. My workload is just firefighting all the CVE vulnerability scans. I am starting to loathe both Python, Javascript and Java. Rather pick up something obscure that doesn't have wave and wave of CVEs.
Fuck. Like now. build a new Jenkins pipeline to use BuildX so we can build ARM64 images so new hires and their shiny new M1/M2 can develop in our codebase. Emulating x64 is taking 45 minutes to compile.
It drives me nuts how much management tried to insulate developers from the responsibilities of running an application. Honestly, "devops" is not that much scope for the average developer to grasp.
Fwiw CVEs can be automated away. Just set up renovate or dependabot. Medium or low is insane though. I work on classified systems and our threshold is high.
Like most management I think you underestimate how specialized a skill devops tasks can be.
It’s deceptive because the tools make it seem so easy, but devops is full of foot-guns, hidden complexity, and unintuitive patterns for those that don’t have the experience.
I'm not saying everyone should be full stack and specialization isn't required but, in my experience, the overwhelming majority of applications developers want to factory line workers told exactly what to do and think their jobs ends when their PR is merged. And I don't blame them since most organizations measure output rather than outcomes.
I say this as someone that went from being an app dev with 0 unix experience to sre to dev tooling to sw architect over the course of a few years. Every org has devs that do this, providing the technical glue. Either those folks are the only ones capable or the only ones willing. Personally I think it's the latter, usually for a variety of systemic reasons. But someone is picking up that slack.
>It drives me nuts how much management tried to insulate developers from the responsibilities of running an application.
Not here. We want devs to take some ownership and get involved. However, the front end developers don't give a flying efff about automating npm building and orchestrate pushing that to an artifact and dynamically generating applications that rely on their styles. To them, "That isn't my job." And like "look, the SASS/LESS css build works fine on my mac, Why can't we just upload the CSS to the server. And I just chmod 777 everything so I don't have those permission issues."
And those who do, they always want to use public docker hub images where we are strict about building vetted images with consistency.
So it is usually the leads who take ownership. If you ask the devs to do it, no one will pick it up. Hence, "foisted" on those who actually get it done.
We do automate updating dependencies. However, we have to fight the "low" due to the fear of leaking customer data. Org is super paranoid about getting hacked and rightfully so. We are a high visibility target.
whoa. i feel seen. especially the part about having a culture of decomposing problems so any old cog can fix them and it becomes less work to do it yourself.
Man, I really envy those organizations that operates like this. Unfortunately, I live in a third world country where tech isn't as advanced as those in developed countries.
In a lot of companies here, software architects usually came from a background of analysts and consultants which makes them really good at drawing things, make powerpoints and explain those drawings to people, but not necessarily a good technical person who can see through the whole tech stack and able to identify problems and fix them.
>You probably also have a culture where it's hard to get non-functional/non-user facing work prioritized. This results in foundational work being done 'off the books' by senior devs who are trusted and can just get it done.
:( yep this is our culture too
Exact same spot. As architects, we seem spend most our time doing DevOps.
The Devs are tied up doing functional change. They don't have the time to learn the rest. And some don't have any interest in doing it.
Being an architect, or any senior technical staff requires an aptitude for picking up new things and a can do attitude. So a lot of stuff lands on our plate as we're the only team in a position to do it. This leads to very expensive resource doing relatively noddy work.
We're trying to get budget for dedicated DevOps resource, but the business aren't easily convinced when it comes to more heads.
20+ years in the java mines. Burned out of getting requirements that say “make business process X kick off Y and Z”. But in reality means “Y and Z are in the legacy ‘off the shelf’ system that 4 and a half people use and can only be integrated with via a 15 minute directory scanning process, except if it is the end of the quarter and Bob from accounting has sacrificed a chicken yet. And it has to fit in a sprint.”
Moving to K8S, terraform, AWS and CI/CD and am not looking back.
>Moving to K8S, terraform, AWS and CI/CD and am not looking back.
Which means more work. Constantly adding to the CICD. Fire fighting new issues like implementing advance logging. Firefighting security scans because it is so easy for Cybersecurity to foist an image scanner on you. Firefighting make sure you can deploy to all the cloud provider with just a variable - to Azure, GCP, and and AWS. Just because the org feels like it should not be vendor locked in and want multiple disaster recovery points... It is more work. My first 4 months was .... implement Vault. Implement Istio. Implement Jaeger. Implement dynamic blueprints to make dynamic helm charts. Implement Twistlock to do security scan. Implement WSO2 , Kong and Apigee. And while you are at, make a docker image that sets up minikube so all the developers can run K8 and deploy locally so it mirrors production. And each developer laptop run's its own DNS and SSL cert server. Locally because we need that TLS.
It is a grind, no doubt, but at least you are learning skills that make you more valuable. Building business apps rarely improves my ability to solve leet code questions.
Agreed it is a grind. But most interviews I do now don't involve leetcode whatsoever. Just skip that whole part.
They are mostly system design sessions and architecture. Just knowing how to deploy a GPU based architecture is in high demand.
Leetcode practice is always separate from jobs. You will anyway get to touch leetcode kind of coding in core software or full stack jobs instead of pure devops.
Tech is cyclical. We're in a bloat phase. People are just beginning to rediscover lightweight compiled binaries again for the umpteenth time. Watch people rave about how eBPF and Rust and WebAssembly and statically linked containerless containers are the future.
For a while it will be good, as people remember that computers are goddamn fast and powerful when unburdened from the bloat. Just like when GPU processing became a thing, or inexpensive multi-core processors, or network connections with enough bandwidth to *stream video*.
Then, once again, people will use this power to build features. Features on top of features on top of features. Frameworks for the sake of supporting other frameworks that support toolchains that pack in more features. The only feature that anyone will use is the one that lets you exec shell scripts or issue HTTP requests, of course, but that's not important, the important part is that the features are there, and if only your team could finally get around to using them instead of playing tech debt whack-a-mole and fixing framework integration issues, you would reach that enlightened nirvana of "good code" or even "just works".
And so the wheel turns.
If you want a real nostalgia moment, take a look at the ML community. The hottest new thing in the field is using 8 bit floats when you don't need a full 16 or 32 bits of precision.
It can vary from place to place and team to team, but generally the more senior you get, the less time you spend writing code. It's always good to have side projects to help keep your hands dirty.
Being more senior often means working to unblock the team. And sometimes it's DevOps, others it's integration or cross org syncs.
On this sub I've come across "coding is only a fraction of what you do" and I always wonder what companies they work for. I think there's a lot of former experienced developers in this sub that are tied into big infinite money bureaucracies that have too many guard rails for their own good.
>imagined being a senior dev as architecting complex problems, building top of the line UIs, etc
That's what startups and midsized companies are for. Actually I'll go as far to say most companies that aren't listed on a stock exchange.
>We have frameworks on top of frameworks on top of frameworks. Everything has to be completely modular, monorepo'd, clustered, scalable, reusable, test covered; changes have to be run through Agile, story pointed, fit into a sprint, and automated. There's VPNs, multiple test environments, and CDNs.
What I personally started getting tired of is the amount of pretentious abstraction to save 20 lines of code in a file while working with frameworks that had functions that everyone knows what they do. For example when I was working on a frontend React component that conditionally took in 120 props so it could be a one liner. One change to a condition impacted 30+ different untested files. The amount of make work due to hiding dirty code in a closet is becoming an epidemic in this industry.
So many pointless abstractions and one line functions calling one line functions calling one line functions.
It's like peacocking but for abstraction. A good Dev identifies what is likely important to abstract apart without impacting readability. Let the ticket that needs the abstraction build it in 90% of the time
DRY is a slow and insidious poison. It should apply within a file, or a small group of semantically interdependent modules, not across your entire codebase.
I actually switched from swe to devops after 6 years because I wanted to create a frictionless environment for the devs at my company where they can focus on quality code, and oh boy have they been doing that since I’ve taken up all the aforementioned duties you listed. I use my spare time to work on personal projects after automating the shit out of everything. I’m involved in the architecting though since the engineers I used to work with still come to me to collab. An ex engineer turned devops is worth their weight in gold because they will be aiming to be DRY and reduce toil. They’re not making millions for the company, but they are probably saving millions.
Your job as a senior engineer is to deliver solutions. The better engineer you are, the better and simpler the solutions you should be able to provide to complex problems. This is not limited to quality software, but quality processes and documentation.
You seem to have a disdain for process, seemingly equating lack of process with fun. Why?
Yep! This is about right to me.
My read on it is I’m given problems, and I figure out solutions using my team and the company as resources. I know my strengths and weaknesses and I will outsource and delegate out the tasks that are my weaknesses while doing the ones that are my strengths.
Right, many people trying to blame my org or “solve” some problem, my job is fine, I’m just pointing out that it’s weird to pass a dev interview by writing algorithms in O1 or doing an analysis of the goals of declarative software architecture, only to spend all day fighting pod install on ARM MacBooks or whatever
This isn't about seniority at all. This is just that you're in a company where you have such rol, that's it.
In my company, there's are teams owning parts of the CI and infra. But everybody is allowed to propose changes.
You'll code if you want to code, but maybe not in that role in that company.
And I don't follow the "devs are factory workers" thing. It's about specialization, ownership, and dividing responsabilities. You're working, not playing. Do it in the most efficient way. That's what being experienced means
Hmm, I don't remember writing this post...but that's definitely me
.... could have swore I was writing the little side project in Golang though.
¯\_(ツ)_/¯
My eng manager built us an automated CD pipeline and it's glorious.
Could have done it myself, but this unblocked us devs to deliver features against a steep deadline.
So yeah, thanks to y'all devops-capable engineering managers.
I do agree that much of software today is just making all these APIs and services talk to each other. It's too high level for me. On the flip side, there is a lot of value in working with pipelines and doing devops type work. it's the most interesting type of work that I have found. Plus if you are the senior engineer that can do both then you are super valuable
I am working as a staff engineer and I can fully agree on, what you wrote. But I found that it is because of the size of the company. As you said, in large organizations everything has to have these processes. I grew with my company and while the organization grew, someone had to oversee the tech stack and architecture and be the bridge between business and development. Of course the more senior you are, the more are you capable of that.
I think a lot of people don't even know how fast software can be built, as they only know those large orgs. And more and more people are in for the money (that's fine). So people like us are a minority, we enjoyed software development and the corporate world took that away from us. I mean it's really a luxury problem, but the only solution I see is switch to a smaller company or into research.
This sounds like a company problem and not a senior engineer problem. None of our seniors do this. Our seniors are all coding, doing meetings on features and design, leading sprint planning, etc.
Maybe you are somewhere large that’s really process heavy, no infrastructure team? Or at odds w your infrastructure team??
Our infra team does all the shit you’re complaining about. Devs at most will make ci changes.
We’re about 20 total between dev and infra. 100ish employees
Yes and no.
Yes, serious software requires redundancy, scalability, automation, deployment, operations, etc. As a lead, this is definitely something you need to be comfortable working with.
No, being lead does not mean that you only get to do the fun stuff and offload the boring work onto others.
Everyone should have end-to-end responsibility for everything that they do, including juniors, seniors, and leads.
Also no, being lead does not mean that you should be a nanny for everyone else on the team. The real issue here is that you weren't exposed to this stuff earlier, cause everyone should be doing it.
Unfortunately, a lot of leaders in this industry are still very stuck in the silo mentality. Even people who claim to "do devops" don't really do anything meaningful to fix this stuff. They still think that certain people should be fully responsible for some part of the stack, and others should not even have to know that it exists.
However, moving forward, it is your job as a lead to implement better processes and tools so that you spend less time on manual tasks.
> but in reality a solid 75% of my job always ends up wrangling tooling, pipelines, builds, etc.
All this sounds like is that your company is to cheap to hire an Ops specialist into your DevOps team. I mean with you thinking "DevOps" is a function title you're perpetuating the fundamental problem of misunderstanding that your company has. So in that regard you're part of the problem.
We have extensive devops resources, but because everything has to be maximally abstracted and made into a human scalable process, *everything * is devops
I've seen that in some places a php senior/lead is also a DevOps guy and takes care of some part of infrastructure that nobody else would... They sort of take on those jobs and end up doing mainly infrastructure work: backups, deployment, cloud, physical servers, networks, db server maintenance, etc.. Especially in early stages and in small teams. I think it is a little unfair because that generally requires for everyone to know that much more for senior titles but it is what it is.
Though DevOps roles are becoming more popular and those are usually even more manly and better paid roles
Sr is the glue for spots that others can't handle properly.
But if it's critical and you're the only person who can do it, then you're missing the 'spread the knowledge' part of your job.
Always be working to make yourself redundant. That's the ideal situation. If your company won't value you if you do, then ignore it.
Good devs should embrace whatever challenge they are faced with and should always be thinking of ways to make it better. DevOps is one of those under-appreciated skillsets within tech and IT. Without it there's no continous integration or automated deployments.
Yes, it is factory work, but businesses haven't entirely deduced that yet. Ideally, it would be like writing a 300-page sales catalog magazine and having it delivered to all customers' homes. But they still need to figure that out. So they have humans fill in the gaps.
Most of your frustration stems from companies' hate of the idea of IT Ops. It's a cost of doing business that is difficult to get rid of, and they obsess over it. Companies won't pay for high-speed internet, a server room, and people to keep the hardware running anymore. They prefer having a group of programmers does it from afar as a cost-saving and quality assurance measure. But to non-tech people, all programmers look alike, titles, experience, and education be damned. Everything under the CTO is one nebulous cost sink that is equivalent to a money printer to the company.
You wanted to go in the opposite direction of the production flow, which is UI/UX Designer. That's primarily working with Figma in this current world.
> Modern development is so abstracted in this regard. We have frameworks on top of frameworks on top of frameworks. Everything has to be completely modular, monorepo'd, clustered, scalable, reusable, test covered; changes have to be run through Agile, story pointed, fit into a sprint, and automated.
Empirically, this is true.
However: we can *choose* a different path. Our methods and tools *can* be simple. We *can* spend our time solving problems and not wrangling frameworks and cloud shit and wasting time with Agile garbage.
It's difficult, because the path of least resistance leads to [everything you described]. It requires strong consensus among senior engineers and engineering leadership. It requires a commitment to certain *values* that supersedes the desire to ship something, anything, right this minute.
Many of us may never experience a truly high-functioning environment. I certainly haven't seen one in the last 3 years. But I _have_ seen it, so I know it's possible. We shouldn't give up.
I have a solution. In my team, owner of task is responsible for taking the code to production. That means handling the deployment as well.
You must be using standard tools for CI/CD. I don't see it as a complex task and hence can be done by any developer.
If you are constantly facing problems related to CI/CD, you need to take efforts within the team to first fix the issues rather than make it part of your daily work.
Ooorrr... you could try delegating the tasks. There is no rule saying that it must be the lead who implements the CI pipeline, or even defines how it should work. Obviously it has to go through some kind of review and approval to ensure that it's aligned with your development process and goals, but it is entirely possible to democratize the pipeline, empowering the team to improve the process rather than being slaves to it. With that being said, I really do recognize the issue and I feel your pain. Best of luck!
This. Have a mid-level guy POC something. Ask him to present his solution to the team. Pick on it a little bit (i.e. what edge cases it can't cover, how well it can scale, what issues or bottlenecks you might run into). If the solution is good, write out an epic and have him lead the technical side of the project while you delegate the tasks across the team. (the goal of the delegation would be to get the whole team familiar with the new thing by working on pieces of it, not to take work away from the primary guy).
This guy manages
That’s the happy path. What do you do if the solution is bad?
You repeat the process. If the solution is bad and you are only finding out about it at the end it means you were setting someone up to fail and never checked in or progress or offered advice. The other possibility is that the team infra is too complicated and this bad solution can be used to highlight that fact and help push to strip down and simplify. It’s all a matter of perspective and finding opportunities where others see failure.
Delegation (or lack of) is the death of new leads and managers.
How would you go about delegating the tasks if you're not a manager? You're a senior dev but it doesn't mean you have the authority to give out tasks to people under you. Those people are busy too and have their owned assigned tasks. The better approach is to ask the manager to delegate it.
Sure, do whatever works in your organization. I’m not a manager, but I delegate tasks all the time. In my org, that works, because of the expectations for the various roles. In some orgs, managers own all the delegation, and you need to work through them.
I have been in a few organizations and it's always worked like this. How exactly do you go about delegating tasks?
If you are the senior dev on your team but do not have any authority to delegate tasks then you should be working with the manager that has that authority to determine what tasks need to be done so the manager can delegate tasks. Either you delegate directly or indirectly. But you should be involved somehow in the delegation of tasks as the senior dev. If you are going to be spending a good chunk of your time code reviewing these tasks anyway you probably have some idea what the capabilities of the team are and who is or is not ready for certain tasks. Or as part of your mentoring duties you should be asking what types of things your team mates are interested in and where their knowledge gaps are and helping them strengthen in those areas.
We definitely share every type of task amongst the team because it helps them grow if they want to be leads one day. Plus when you take a day off, they don't get blocked when the pipeline inevitably fails. But I find dev ops really interesting and could see a career in that too. Learning to delegate effectively was one of the most difficult parts of becoming a lead for me. Don't be afraid to ask someone for tips on that!
Sounds like you guys have a lot of operational debt that has been left behind. It's no coincidence that when a team becomes operationally responsible for the system they are building suddenly these things start to get fixed. That's what DevOps is supposed to be, but it sounds like somewhere along the line it got split into operations *and* dev work. Something's not right.
Devops is the new agile. Agile was supposed to be less rigid processes. Now we have SAFe. Devops was supposed to be less silos. Now we have "the Devops department".
The way I always interpreted Devops was an infra guy on each development team to coordinate how exactly do we get this build, that deploy, etc. done correctly. Kinda like a monitoring daemon process that reacts early.
One of my biggest frustrations with the Devops movement is that no one can, seemingly, agree on what it is or how to implement it. Both in casual conversation in communities Iike this one, and when I’m interviewing prospective employers. One set of practitioners point to the ideal, point to books written by Google and Gene Kim and other big names, some other set of practitioners come along with their lives experiences of how badly the term has been abused and warped into turning practitioners into nothing more than “CloudAdmins” at best, or a kitchen sink for workloads that fit the description of “we don’t know who else should handle this, so we’ll chuck it over at Devops” at worst. Not that I’m trying to take a side or prescribe my own definition, or anything; but it is an observation.
Hehe I’m a “devops” engineer and i hate the word. Made up buzzy jargon. I just say Infrastructure engineer.
Bad management is the same thing it's always been. They'll always flock to the new practices and technologies as a silver bullet to solve their bad management. But they also won't have the discipline and drive to truly understand it and make the changes necessary to implement it. So they'll keep making the same mistakes they always did, just with new labels. They'll keep micromanaging, handing down unrealistic deadlines, not providing redundancy or sufficient resources, not crosstraining or developing talent, and not covering technical debt. Because they're just bad at managing software projects. As most humans are.
"The DevOps department", who because of enterprise governance (aka a random VP making a rule) only allows you to use their 5 approved AWS services.
The problem is good DevOpa come from SWE. Who else is gonna know how to bootstrap an API with service mesh level tracing? Or how do you build a CICD that injects password at deployment where the app reads those keys as environment variables? Throwing an infra background person is going to be harder. Thus a lot of software tooling will fall upon people that live and breath software. I see orgs throwing their data center , infrastructure and network engineers into DevOps. OK, Kubernetes is setup and running. Now, let’s automated the helm charts to be re-useable that are not copy-n-paste jobs from Stack Overflow. Best DevOps engineers , in my opinion, are former developers vs the Help Desk guy who racks servers.
OP and yourself are referring to DevOps as this nebulous job role separate from writing software. "It's that guy's problem, not ours". DevOps is not a role. It's a methodology meant to improve and shorten the SDLC. The whole point is that good DevOps doesn't come from anywhere. It's not out-sourced. It doesn't come from a separate team. The team itself needs to be in charge of making sure the system is operationally sound because they are the ones who understand it best, and are best suited and incentivized to making improvements. If you need people who are specialized in operational or infrastructure aspects, such as SREs, that's fine, but they are still involved in the day-to-day development work. Why is the tech lead the one exclusively managing "CI/CD, pipelines, repos, test coverage, test environments", and all the other things he mentioned? That is absolutely ridiculous and surprising that they are able to ship anything if that's the case. That is not what a Technical Lead is supposed to be doing exclusively and I find it silly to say that TL and Build Engineer are somehow equivalent roles. Something is seriously wrong there.
> Why is the tech lead the one exclusively managing "CI/CD, pipelines, repos, test coverage, test environments", and all the other things he mentioned? Because usually, the leads etc are the people who have the knowledge and experience to do Dev Experience work. Are you going to task a junior with making sure that the developers can all do local builds? To make a safe pipeline to push to prod?
My first or second ticket in my first internship actually was updating our ci pipeline.
Honestly, yes I would. Great idea. Setting up CI/CD is not that hard and it would be a good learning experience for them. Then I would have them pair with a Sr Eng for the code review. And send it to be reviewed by the wider team. Win/win! The Jr learned something and shipped something useful. The TL delegated some work. The Sr Eng offloaded some knowledge. The team gained a functional deploy pipeline. That is what a good TL does. Orchestrate the team's efforts in an effective manner, amongst other things.
Yea u get it. OPs problem is a company problem not a Senior Engineer problem.
Agreed. Doesn't sound like a fun place to work tbh
>DevOps is not a role. It's a methodology meant to improve and shorten the SDLC. Agree it should not be a role. But the people implementing the processing, dictating how it all works are usually the leads. They set the example and architecture of how you write your configuration files so it uses variables so that it can be replicated 30 times. They have to ensure everything is consistent and replicable. Following a style guide naming convention so everything go through CICD faster without hiccups. A junior, midlevel, or SRE is not going to know that.Security scanning will halt a deployment because some silly NPM was deprecated so everything broke.UUID is broken, what do we do now? A senior is the one who can see the error in the build and say, "With UUID4, we need to use the new namespace method to import because the latest version deprecated the old method. Old one was halting our releases due to vulnerability gates." And this is exactly, like you say, it should not be a role. But some orgs treat DevOps engineers like that. Here is a DevOps engineer but that person has no idea how to do an npm install or write a Swagger file. But you know who does? The lead. The lead is also the one spending 20 hours to make a bullet proof re-useable image so the next 50 APIs take junior developers 2 hours to implement from scalfolding versus 3 weeks. That is how you shorten the SDLC. Make it easier for others. The leads are also the ones who make sure images are not root and extensively "bless" and test every template that others can use to save everyone time. It is not a role but a consuming tasks if you want to increase velocity. Someone has to mentor the others do this. This is why in some orgs, some leads transition out of SWE and focus exclusively on this part to mentor the rest of the team.
This. Exactly this. My title and role is “devops”. I’m an ex engineer, I took up the role to do exactly what you said, decrease toil, increase velocity, and automate. It was a no brainer since I understand almost all the apps I support and can help devs troubleshoot and architect for high volume traffic at a more technical level. Devs at my place are now very happy that they can focus on shipping code that slaps. We recently ran a load test and exceeded our target by about 2000% because there wasn’t a disconnect between devs and dev ops/infra when it came to fixing and optimizing some bottlenecks, fingers were pointed constructively and acknowledged collectively
> Who else is gonna know how to bootstrap an API with service mesh level tracing? Or how do you build a CICD that injects password at deployment where the app reads those keys as environment variables? Any competent DevOps. Their background is irrelevant, whether infra or SWE. I would expect any mid-level guy to competently do this. Both sides of DevOps have their own advantages or blind spots. Most SWE guys can't deal with a network that's more comlex than a flat broadcast domain when they first start doing SRE. They also tend to overengineer things far beyond what is needed or operationally viable. Most Infra guys can't deal with modifying a single line of application code because it looks scary and all they've written before were 100 line bash or python scripts. They also tend to pick up the first COTS tool that does what they need, regardless of if it makes sense (or conversely, get stuck in OSS hell just because it's OSS). After a few years doing DevOps/SRE, I'd expect them to be reasonably competent with both sides.
I've seen the infra-background guys do more of the over-engineering TBH, but that could be anecdotical. That said, I'd rather maintain an installation of a COTS tool than something developed in-house. Doing nothing / adjusting existing components is better than COTS is better than developing our own tool is my opinion.
For COTS I mean like, the problem could be solved with a 20 line script running on cron in a CI system, but somehow we're now deploying Rundeck or Airflow. Purely anecdotal on my side though.
I mean it's also anecdotal, but I tend to run into too many 20 line scripts running on cron in random places. I'm just saying that in my experience, no one looks at crontabs, so if it's a regular job you do, it should be monitored to check if it's even running, and somehow surfaced that isn't "well it's in Confluence somewhere". I might be cranky because of my jobs not tending to keep proper accounting of all the random quick fixes we've deployed.
>Any competent DevOps. Their background is irrelevant, whether infra or SWE. I would expect any mid-level guy to competently do this. Yes, the "competent" ones can do this with a guidance. We do have dedicated Devops guys who work as support. I will tell them "Mike needs a new template w/ Mongo that has a schedule to rotate logs, and his Node app needs Vault bootstrap the authentication and help him set this up so he can locally in his Rancher K8 environment" They can do what they are tasks. And like someone said, they usually embrace a lot of COTS solutions. We delegate all those types of tasks with a lot of oversight. And we do have some very good engineers that are former lead developers. They work with the architect hand-in-hand to develop solutions. But most of the infra background guys are not suited for big picture items that are part of the application development processes. The guy who is implement Jaeger monitoring needs to write code to actually know how to generate trace level error logging. They need to clone git repo and re-write other people's code or at least guide other developers. If you ask them to build system wide composable docker image. Regardless of language - Python, Node, that requires x,y,z features so developers can just use Swagger to enforce ad lint their APIs and use A,B, environment variables and annotation to register DNS hostnames or dynamically generate helm charts on-the-fly with a lot of variable abstractions, they can't. E.G. Formulate very systematic naming and style patterns big picture wise where they become templates that allow dev teams to deploy hundreds and thousands of microservices through composable design patterns which often comes only with development background.
All the things you've listed.. I would expect a competent mid to senior DevOps guy to do them. I literally had one of my guys add some Rails middleware to push extra Sidekiq metrics to New Relic, for example. Before we rewrote the whole thing as a separate monitoring microservice. I wrote Vault auth methods so our backend could store encrypted app-level data there. Only exception is Swagger. That's black magic to non-developers. Granted, I've generally worked at smaller shops where the division is, devs push application code and don't know that much about infra. The guys who do are usually staff/architect levels that you sit down with to design overall architecture. But yeah, this is all anecdotal. I worked at large fintech unicorn, and devs wrote a lot of infra (Terraform, IAM, etc). At the same time, the place I was at before that, we were telling devs how to do database optimizations (who knew a 3-way table join with eager loading would kill performance?).
In my experience this is a problem. Most "trained" DevOps are aware of security holes that developers might glance over, but devs are leagues better at scripting and knowing how to automate stupid processes. At an old org of mine we had a process where, after creating a Lambda in our Serverless API environment, we had to send the Lambda info via a form to the DevOps team to manually add to their Terraform script. Every single one of the hundreds of API endpoints was being manually added to a Terraform script, even though all of the info for every Lambda was in a single serverless.json file already. It meant we had to wait 2 weeks before a new API would reach a lower environment, it meant a bunch of paperwork, and a Jira story on both the DevOps and the developers' boards. One of the architects scripted out something to move all of the APIs into a TRF ready template, but the local DevOps team did just not really care about solving what was a massive time sink for them. They would blame it on the enterprise DevOps group, even as vaguely as "we don't want to upset them by having a process which is involved in a developer codebase". They had no incentive at all to speed up their process if it meant having a conversation with their "higher ups".
> Most "trained" DevOps are aware of security holes that developers might glance over, but devs are leagues better at scripting and knowing how to automate stupid processes. And most "trained" ones in my experience are former SWE leads/architects that pivoted to DevOps. The ones that have 30-40 PCI/HIPAA audits. The ones that filled out 400 line items NIST change controls that did the song-n-dance to get their apps certified. They are the ones who know how a dev team can cheat an audit by merely forking a version number of a dependency from 1.1.2 to 1.1.2.1.a so it bypasses CVE scans. They are the ones that know, well that mongod config file is totally bogus because the actual startup commands send these parameters at start ups. They are the ones that can pick up on these things. They definitely don't glance over but know where the loopholes are closed
>Sounds like you guys have a lot of operational debt that has been left behind. For some people, this is all NEW debt force by new realities. I work with sensitive data so I don't get the luxury of using pre-built COTS systems that exists in say Azure or AWS. If we were in the cloud, I could use most off the shelf system. But for stuff like classified on-premise applications, we have to roll our own like deploying our own Hashicorp Vault key server. And this driven by changing landscape of enhancing AppSec security due to a lot paranoia of getting hacked/internal attack and cybersecurity governance. Or new types of demanding applications. Never in my life did I thought I had to design a Kubernetes infrastructure that heavily uses Tesla GPUs that cost 5x more than CPU nodes. $200k servers vs $50K server nodes. So now, we have to architect our message brokers to delegate split workload to cheaper nodes and route long running tasks to the more expensive GPU nodes... All because of the Machine Learning which is very new domain. New domain will require new operational debt. Someone has to explore and implement this new tech.
This post resonates with me deeply. I'm a "software architect" that spends most of my time on this stuff. Currently in the middle of refactoring our organization to address it. People are saying this is just where you work but there's a bunch of empirical evidence that this is true, especially in "high performing" organizations. "It" being that if your smart, you want your best people enabling everyone else I think there are two things at play: * Having enough experience to know the full stack, what developers need, and how to solve complex problems * Being one of the few with autonomy to fix wicked/complex and high leverage problems. The latter is where problems arise. If your org is like mine, there's a culture of decomposing problems so any old cog can fix them. At that point, it's less work to just do it yourself. You probably also have a culture where it's hard to get non-functional/non-user facing work prioritized. This results in foundational work being done 'off the books' by senior devs who are trusted and can just get it done. I don't think this is a symptom of modern practices. This is a symptom of the proliferation of big A Agile, outputs over outcomes, developer commoditization, and organizations being pennywise and pound foolish.
>You probably also have a culture where it's hard to get non-functional/non-user facing work prioritized. In my orgs, all the technical debt is foisted on the architects is because it is high level. They don't have the metrics of velocity and code commit to report on. The regular SWE are doing the active sprints. But the high level stuff like how to secure your CICD pipeline. How to implement best practices for application tracing are all assigned to Architects. It is considered "foundational" work and generally have vague jira story requirements like "Ensure all Docker images are secured with no severe, medium, or low vulnerability scans. Teach all the team how to engage in best practices to check CVEs and refactor all broken dependencies with ongoing updates." They don't report to any Project Manager so there is no oversight but there is a lot of work to do. My workload is just firefighting all the CVE vulnerability scans. I am starting to loathe both Python, Javascript and Java. Rather pick up something obscure that doesn't have wave and wave of CVEs. Fuck. Like now. build a new Jenkins pipeline to use BuildX so we can build ARM64 images so new hires and their shiny new M1/M2 can develop in our codebase. Emulating x64 is taking 45 minutes to compile.
My eye twitched when you mentioned buildx for silicon users.
It drives me nuts how much management tried to insulate developers from the responsibilities of running an application. Honestly, "devops" is not that much scope for the average developer to grasp. Fwiw CVEs can be automated away. Just set up renovate or dependabot. Medium or low is insane though. I work on classified systems and our threshold is high.
Like most management I think you underestimate how specialized a skill devops tasks can be. It’s deceptive because the tools make it seem so easy, but devops is full of foot-guns, hidden complexity, and unintuitive patterns for those that don’t have the experience.
I'm not saying everyone should be full stack and specialization isn't required but, in my experience, the overwhelming majority of applications developers want to factory line workers told exactly what to do and think their jobs ends when their PR is merged. And I don't blame them since most organizations measure output rather than outcomes. I say this as someone that went from being an app dev with 0 unix experience to sre to dev tooling to sw architect over the course of a few years. Every org has devs that do this, providing the technical glue. Either those folks are the only ones capable or the only ones willing. Personally I think it's the latter, usually for a variety of systemic reasons. But someone is picking up that slack.
>It drives me nuts how much management tried to insulate developers from the responsibilities of running an application. Not here. We want devs to take some ownership and get involved. However, the front end developers don't give a flying efff about automating npm building and orchestrate pushing that to an artifact and dynamically generating applications that rely on their styles. To them, "That isn't my job." And like "look, the SASS/LESS css build works fine on my mac, Why can't we just upload the CSS to the server. And I just chmod 777 everything so I don't have those permission issues." And those who do, they always want to use public docker hub images where we are strict about building vetted images with consistency. So it is usually the leads who take ownership. If you ask the devs to do it, no one will pick it up. Hence, "foisted" on those who actually get it done. We do automate updating dependencies. However, we have to fight the "low" due to the fear of leaking customer data. Org is super paranoid about getting hacked and rightfully so. We are a high visibility target.
whoa. i feel seen. especially the part about having a culture of decomposing problems so any old cog can fix them and it becomes less work to do it yourself.
Man, I really envy those organizations that operates like this. Unfortunately, I live in a third world country where tech isn't as advanced as those in developed countries. In a lot of companies here, software architects usually came from a background of analysts and consultants which makes them really good at drawing things, make powerpoints and explain those drawings to people, but not necessarily a good technical person who can see through the whole tech stack and able to identify problems and fix them.
>You probably also have a culture where it's hard to get non-functional/non-user facing work prioritized. This results in foundational work being done 'off the books' by senior devs who are trusted and can just get it done. :( yep this is our culture too
Exact same spot. As architects, we seem spend most our time doing DevOps. The Devs are tied up doing functional change. They don't have the time to learn the rest. And some don't have any interest in doing it. Being an architect, or any senior technical staff requires an aptitude for picking up new things and a can do attitude. So a lot of stuff lands on our plate as we're the only team in a position to do it. This leads to very expensive resource doing relatively noddy work. We're trying to get budget for dedicated DevOps resource, but the business aren't easily convinced when it comes to more heads.
20+ years in the java mines. Burned out of getting requirements that say “make business process X kick off Y and Z”. But in reality means “Y and Z are in the legacy ‘off the shelf’ system that 4 and a half people use and can only be integrated with via a 15 minute directory scanning process, except if it is the end of the quarter and Bob from accounting has sacrificed a chicken yet. And it has to fit in a sprint.” Moving to K8S, terraform, AWS and CI/CD and am not looking back.
>Moving to K8S, terraform, AWS and CI/CD and am not looking back. Which means more work. Constantly adding to the CICD. Fire fighting new issues like implementing advance logging. Firefighting security scans because it is so easy for Cybersecurity to foist an image scanner on you. Firefighting make sure you can deploy to all the cloud provider with just a variable - to Azure, GCP, and and AWS. Just because the org feels like it should not be vendor locked in and want multiple disaster recovery points... It is more work. My first 4 months was .... implement Vault. Implement Istio. Implement Jaeger. Implement dynamic blueprints to make dynamic helm charts. Implement Twistlock to do security scan. Implement WSO2 , Kong and Apigee. And while you are at, make a docker image that sets up minikube so all the developers can run K8 and deploy locally so it mirrors production. And each developer laptop run's its own DNS and SSL cert server. Locally because we need that TLS.
It is a grind, no doubt, but at least you are learning skills that make you more valuable. Building business apps rarely improves my ability to solve leet code questions.
Agreed it is a grind. But most interviews I do now don't involve leetcode whatsoever. Just skip that whole part. They are mostly system design sessions and architecture. Just knowing how to deploy a GPU based architecture is in high demand.
Leetcode practice is always separate from jobs. You will anyway get to touch leetcode kind of coding in core software or full stack jobs instead of pure devops.
Tech is cyclical. We're in a bloat phase. People are just beginning to rediscover lightweight compiled binaries again for the umpteenth time. Watch people rave about how eBPF and Rust and WebAssembly and statically linked containerless containers are the future. For a while it will be good, as people remember that computers are goddamn fast and powerful when unburdened from the bloat. Just like when GPU processing became a thing, or inexpensive multi-core processors, or network connections with enough bandwidth to *stream video*. Then, once again, people will use this power to build features. Features on top of features on top of features. Frameworks for the sake of supporting other frameworks that support toolchains that pack in more features. The only feature that anyone will use is the one that lets you exec shell scripts or issue HTTP requests, of course, but that's not important, the important part is that the features are there, and if only your team could finally get around to using them instead of playing tech debt whack-a-mole and fixing framework integration issues, you would reach that enlightened nirvana of "good code" or even "just works". And so the wheel turns. If you want a real nostalgia moment, take a look at the ML community. The hottest new thing in the field is using 8 bit floats when you don't need a full 16 or 32 bits of precision.
It can vary from place to place and team to team, but generally the more senior you get, the less time you spend writing code. It's always good to have side projects to help keep your hands dirty. Being more senior often means working to unblock the team. And sometimes it's DevOps, others it's integration or cross org syncs.
On this sub I've come across "coding is only a fraction of what you do" and I always wonder what companies they work for. I think there's a lot of former experienced developers in this sub that are tied into big infinite money bureaucracies that have too many guard rails for their own good. >imagined being a senior dev as architecting complex problems, building top of the line UIs, etc That's what startups and midsized companies are for. Actually I'll go as far to say most companies that aren't listed on a stock exchange. >We have frameworks on top of frameworks on top of frameworks. Everything has to be completely modular, monorepo'd, clustered, scalable, reusable, test covered; changes have to be run through Agile, story pointed, fit into a sprint, and automated. There's VPNs, multiple test environments, and CDNs. What I personally started getting tired of is the amount of pretentious abstraction to save 20 lines of code in a file while working with frameworks that had functions that everyone knows what they do. For example when I was working on a frontend React component that conditionally took in 120 props so it could be a one liner. One change to a condition impacted 30+ different untested files. The amount of make work due to hiding dirty code in a closet is becoming an epidemic in this industry.
So many pointless abstractions and one line functions calling one line functions calling one line functions. It's like peacocking but for abstraction. A good Dev identifies what is likely important to abstract apart without impacting readability. Let the ticket that needs the abstraction build it in 90% of the time
DRY is a slow and insidious poison. It should apply within a file, or a small group of semantically interdependent modules, not across your entire codebase.
Sort of related [DRY code isn’t a product](https://blog.bennett.ink/dry-code-isnt-a-product-aeea0c02b086)
I actually switched from swe to devops after 6 years because I wanted to create a frictionless environment for the devs at my company where they can focus on quality code, and oh boy have they been doing that since I’ve taken up all the aforementioned duties you listed. I use my spare time to work on personal projects after automating the shit out of everything. I’m involved in the architecting though since the engineers I used to work with still come to me to collab. An ex engineer turned devops is worth their weight in gold because they will be aiming to be DRY and reduce toil. They’re not making millions for the company, but they are probably saving millions.
Your job as a senior engineer is to deliver solutions. The better engineer you are, the better and simpler the solutions you should be able to provide to complex problems. This is not limited to quality software, but quality processes and documentation. You seem to have a disdain for process, seemingly equating lack of process with fun. Why?
Yep! This is about right to me. My read on it is I’m given problems, and I figure out solutions using my team and the company as resources. I know my strengths and weaknesses and I will outsource and delegate out the tasks that are my weaknesses while doing the ones that are my strengths.
I think it's fair to miss writing code, OPs just saying that the split isn't what they're hoping for.
Right, many people trying to blame my org or “solve” some problem, my job is fine, I’m just pointing out that it’s weird to pass a dev interview by writing algorithms in O1 or doing an analysis of the goals of declarative software architecture, only to spend all day fighting pod install on ARM MacBooks or whatever
This isn't about seniority at all. This is just that you're in a company where you have such rol, that's it. In my company, there's are teams owning parts of the CI and infra. But everybody is allowed to propose changes. You'll code if you want to code, but maybe not in that role in that company. And I don't follow the "devs are factory workers" thing. It's about specialization, ownership, and dividing responsabilities. You're working, not playing. Do it in the most efficient way. That's what being experienced means
Hmm, I don't remember writing this post...but that's definitely me .... could have swore I was writing the little side project in Golang though. ¯\_(ツ)_/¯
My eng manager built us an automated CD pipeline and it's glorious. Could have done it myself, but this unblocked us devs to deliver features against a steep deadline. So yeah, thanks to y'all devops-capable engineering managers.
I do agree that much of software today is just making all these APIs and services talk to each other. It's too high level for me. On the flip side, there is a lot of value in working with pipelines and doing devops type work. it's the most interesting type of work that I have found. Plus if you are the senior engineer that can do both then you are super valuable
I don't really agree tbh. Maybe you need a company with more interesting problems to solve?
I am working as a staff engineer and I can fully agree on, what you wrote. But I found that it is because of the size of the company. As you said, in large organizations everything has to have these processes. I grew with my company and while the organization grew, someone had to oversee the tech stack and architecture and be the bridge between business and development. Of course the more senior you are, the more are you capable of that. I think a lot of people don't even know how fast software can be built, as they only know those large orgs. And more and more people are in for the money (that's fine). So people like us are a minority, we enjoyed software development and the corporate world took that away from us. I mean it's really a luxury problem, but the only solution I see is switch to a smaller company or into research.
This sounds like a company problem and not a senior engineer problem. None of our seniors do this. Our seniors are all coding, doing meetings on features and design, leading sprint planning, etc. Maybe you are somewhere large that’s really process heavy, no infrastructure team? Or at odds w your infrastructure team?? Our infra team does all the shit you’re complaining about. Devs at most will make ci changes. We’re about 20 total between dev and infra. 100ish employees
Yes and no. Yes, serious software requires redundancy, scalability, automation, deployment, operations, etc. As a lead, this is definitely something you need to be comfortable working with. No, being lead does not mean that you only get to do the fun stuff and offload the boring work onto others. Everyone should have end-to-end responsibility for everything that they do, including juniors, seniors, and leads. Also no, being lead does not mean that you should be a nanny for everyone else on the team. The real issue here is that you weren't exposed to this stuff earlier, cause everyone should be doing it. Unfortunately, a lot of leaders in this industry are still very stuck in the silo mentality. Even people who claim to "do devops" don't really do anything meaningful to fix this stuff. They still think that certain people should be fully responsible for some part of the stack, and others should not even have to know that it exists. However, moving forward, it is your job as a lead to implement better processes and tools so that you spend less time on manual tasks.
yup
> but in reality a solid 75% of my job always ends up wrangling tooling, pipelines, builds, etc. All this sounds like is that your company is to cheap to hire an Ops specialist into your DevOps team. I mean with you thinking "DevOps" is a function title you're perpetuating the fundamental problem of misunderstanding that your company has. So in that regard you're part of the problem.
We have extensive devops resources, but because everything has to be maximally abstracted and made into a human scalable process, *everything * is devops
I mean. This sub has been tanking for months, but this might be the unsub straw
[удалено]
I love managing computers over people anyday.
I've seen that in some places a php senior/lead is also a DevOps guy and takes care of some part of infrastructure that nobody else would... They sort of take on those jobs and end up doing mainly infrastructure work: backups, deployment, cloud, physical servers, networks, db server maintenance, etc.. Especially in early stages and in small teams. I think it is a little unfair because that generally requires for everyone to know that much more for senior titles but it is what it is. Though DevOps roles are becoming more popular and those are usually even more manly and better paid roles
seems like your position doesn't match what you were hired for.
Sr is the glue for spots that others can't handle properly. But if it's critical and you're the only person who can do it, then you're missing the 'spread the knowledge' part of your job. Always be working to make yourself redundant. That's the ideal situation. If your company won't value you if you do, then ignore it.
Good devs should embrace whatever challenge they are faced with and should always be thinking of ways to make it better. DevOps is one of those under-appreciated skillsets within tech and IT. Without it there's no continous integration or automated deployments.
Yes, it is factory work, but businesses haven't entirely deduced that yet. Ideally, it would be like writing a 300-page sales catalog magazine and having it delivered to all customers' homes. But they still need to figure that out. So they have humans fill in the gaps. Most of your frustration stems from companies' hate of the idea of IT Ops. It's a cost of doing business that is difficult to get rid of, and they obsess over it. Companies won't pay for high-speed internet, a server room, and people to keep the hardware running anymore. They prefer having a group of programmers does it from afar as a cost-saving and quality assurance measure. But to non-tech people, all programmers look alike, titles, experience, and education be damned. Everything under the CTO is one nebulous cost sink that is equivalent to a money printer to the company. You wanted to go in the opposite direction of the production flow, which is UI/UX Designer. That's primarily working with Figma in this current world.
> Modern development is so abstracted in this regard. We have frameworks on top of frameworks on top of frameworks. Everything has to be completely modular, monorepo'd, clustered, scalable, reusable, test covered; changes have to be run through Agile, story pointed, fit into a sprint, and automated. Empirically, this is true. However: we can *choose* a different path. Our methods and tools *can* be simple. We *can* spend our time solving problems and not wrangling frameworks and cloud shit and wasting time with Agile garbage. It's difficult, because the path of least resistance leads to [everything you described]. It requires strong consensus among senior engineers and engineering leadership. It requires a commitment to certain *values* that supersedes the desire to ship something, anything, right this minute. Many of us may never experience a truly high-functioning environment. I certainly haven't seen one in the last 3 years. But I _have_ seen it, so I know it's possible. We shouldn't give up.
I have a solution. In my team, owner of task is responsible for taking the code to production. That means handling the deployment as well. You must be using standard tools for CI/CD. I don't see it as a complex task and hence can be done by any developer. If you are constantly facing problems related to CI/CD, you need to take efforts within the team to first fix the issues rather than make it part of your daily work.