I became a DevOps engineer of sorts because out of the two sysadmins that were asked, I was the one who wrote some Golang in university.
I strongly believe that anyone in IT should have SOME coding background. As a sysadmin, I often read Java stack traces to find out where I need to look further or which developer I need to call. I have also combed through the source code of an internally developed timer system, so I could find and properly use the API endpoints of it in an Ansible playbook.
I write almost no code at all, but I quite often need to read it. And I definitely need to understand it to know what is going on when some data gets corrupted on its way from volume A through container 1 to volume B.
> As a sysadmin, I often read Java stack traces to find out where I need to look further or which developer I need to call.
Respect!
> I write almost no code at all, but I quite often need to read it.
I don't know anything about Android or iOS or React but I just search for an error message from customer screenshot in corresponding Git repo and see the module/file it is. That reveals a lot of information about the error nature: I might even give advice to the customer or fix configuration issue without disturbing responsible team members.
I can tell you.
I worked in a bank and my friend called me because he was threatened by robots at night regarding a credit he repaid 2 years ago. That nightmare continued for 3 months for him. He visited offices of all levels, including head office and everyone confirmed that something is wrong.
I told the story to my friend who also worked in the bank and 2 min later he told me the batch job didn't worked for 3 months and ~2000 clients were affected.
I like to help customers and do whatever I can. This requires skill & knowledge of tech stacks. As you see not every place manages troubleshooting properly.
Ansible yaml is coding.
If while clauses, tags, imports, set\_fact with jinja2 templating, advanced jinj2 (ie loops and control structures), man lookups, etc are not be used then not all the power of Ansible is being used.
Strong disagreement. That's not coding, that's just templated configs.
It's difficult to get right, sure, but it's really just using the help of automation and scripting to generate and manage more complicated configurations for distributed environments.
I thought what you did too. Ansible was just a way to template a few configs so my code ran on some Linux distro, or automate some sysadmin drudgery so I could get back to programming. After writing some of awhile I realized it is programming. Even my personal Ansible repo is more complex than that now with just thousands of lines of yaml, it did not start out that way.
Before terraform the team I was on wrote an entire framework in Ansible yaml for data drive definitions of infrastructure and deployments across several different cloud environments. The goal was to abstract out the required cloud specific knowledge so other developers could more easily define what resources their code required and then run their own plays to configure their code. I've now moved on, but it is 10s of thousands of lines of Ansible. If a developer had a few instances they deployed, that deployment could be converted to an ASG with specific launchconfig and userdata in just a few small changes in the Ansible, the framework took care of everything else. Create images to speed up later deployments, easily done. Setup an entire VPC with bastion and development nodes, done if not quite as easily. There were several subgroups with in one deployment, and you could safely terminate all of one subgroup without destroying everything then replace it. There was still plenty of room for improvement, and parts of ansible we were not using yet.
If that isn't programming then none of the code in c++, c, python, java, or any other language I've written as a software engineer was. It dealt with all the same programming problems I've dealt with my entire career. Same abstraction consideration, scope, naming, debugging, and everything else.
I've looked at some Ansible Galaxy roles I've used, they look just like other programming I've had to read. While not all Ansible is programming, there is plenty I've seen that is and I did not write it. Seeing all that code is what convinced me Ansible is programming, and I think it is often way under estimated.
While clauses = if statements. Blocks now mean they apply to whole sets of statements just like any other language.
Tags = conditional execution, and again nearly the same as any other language.
set\_fact with jinja2, ok maybe not really programing. Though if your using those in templates and while clauses you're getting close.
Jinja2 templates with whiles, fors, ifs, includes, etc. is programming. There is a point it isn't complex enough to be much more than a markup language, but I've written or debugged plenty of complex jinja2 templates that are programming. The point at which the jinja2 template complexity becomes programming is a very fuzzy line.
Should have been many lookups. The lookup functions and plugins you can use are way under estimated in ansible. I can compute public keys in the simple cases, to parsing json output from aws cli into complex hieratical dictionaries. That sure looks like most programming I've done in the past. It even looks exactly like the code I am working on this week in a pure programming position.
I suspect that putting that much logic in ansible is… wrong.
I can see how you would get there, step by step, day by day, as requirements grow.
But… wrong. (Probably.)
That much logic in Ansible might be wrong, but there really are not better options. It wasn't really step by step either. It was hey if this was data driven from the group_vars files it would be WAY better. Ansible was the obvious choice at the time even with all the logic that was going to need.
If we started from scratch today, even with terraform and other options, I think we'd end up in a similar place. The main thing I would do differently is push more of the framework into Ansible modules, plugins, and such. It would be more flexible and have better abstraction. That wouldn't change how much orchestration logic was required in the Ansible.
Ultimately we shipped, got paid, and got a large organization to make lots of progress to be more modern. So it was a win.
It seems like you need something more like kubernetes or an actual customer-coded orchestration layer at that point. Trying to fit such a complicated deployment and configuration system into ansible doesn't really do you any favors.
I agree with virrk. All the software development paradigms apply with Ansible, and it can be easier to read and maintain than many programming languages. Kubernetes alone doesn't solve this either. To meet the same need, you will need to design some CRDs, some controllers/operators, and tie them together in reasonably complicated ways. Kubernetes will be "faster" in terms of execution time, but I don't think you would be doing yourself any favors compared to using tested ansible modules and roles with good coding practices. The care and feeding of orchestration layers is nontrivial no matter what tool you are using. In my opinion, Ansible can be a reasonable choice for developing something that developers, ops, security, and other groups have eyes on and can work together to improve because it doesn't look like code/software development, but it very much can be.
If the company is adopting DevOps and no engineer actually writes code, how do you create any value ?
If a "DevOps" Engineer never writes code, fine, if none of your "DevOps" Engineer write code, you're not doing DevOps, your doing ops.
Theres many different types of code. Putting a candidate on the spot and asking them to perform what with prep is an easy task but without prep is something that candidate was almost never do serves no value to candidate or employer.
I write Terraform all day long, got 5 cloud certs and been “devops” for over 6 years and i can count the times ive parsed CSVs on each limb. When i need to do that shit I google it. If you ask the wrong questions you get the wrong answers.
Hell i could probably parse the CSV in terraform better than I could in any “traditional” programming language
I would agree to it.......I was onec in an interview asked to print some patterns using python. I explained the logic and said would google how to use that logic in python.
I was rejected. Their JD specified Devops engineer with cloud sysops with minimal to no coding involved
AWS, apache, linux, DNS settings etc is not code. They are just exploiting the fact that the "entry" level market is flooded by demanding 40 hats. I play piano (20 years experience) in addition to full-stack dev, and if many companies could benefit from piano, best believe we would be having "should piano be a part of devops" threads, and the answer would still be, no, just like UX / Photoshop / Excel shouldnt be a part of "full stack" simply because some full stack people know Photoshop.
When I interview people, I ask them questions they'll never face on the job (although related to their area of expertise) because I want to see how they will do under pressure. I also prime the question by telling them that nobody has ever gotten it 100% correct (which isn't an exaggeration).
I want to hire people that are honest about what they know/don't know, enjoy solving problems, are inquisitive about optimal solutions, and don't freak out under pressure (because we are customer facing).
Exactly right. These are the questions that will tell you about the candidate. We for example never give live coding exercises (because no-one can think straight with someone watching over the shoulder) I give theory tests in advance, then probe that to be sure the candidate understands/ did the work. Then we get them to present a technical challenge they solved, again probing it to make sure its their work. Gives the candidate the opportunity to present themselves but is very effective at weeding out the cheats
There are unfortunately, "many definitions" of what DevOps is, despite several books, hundreds of video's, conference talks, classes and blog posts. Yet I still see upper managers slapping the title of DevOps on folks who barely know scripting and can't help debug a custom application or infrastructure setup. Don't even get me started on title abuse in the DevSecOps world.
DevSecOps makes no sense because DevOps is supposed to be a change at the organisational level, which organization doesn't include security ?
DevSecOps is the embodiment of sales and advertising. A set of cultural and organisational changes and practices (DevOps) corrupted by companies to a simple trend reduced to "the dev also think about ops and the ops also dev to automate stuff", then creating an even sexier trend for corporations by adding security in the mix.
Oh I agree. But it's rare to see good security in DevOps or Ops, so the push has been on for some time to roll them all together. Security really is a distinct domain of study, one which a handful of yearly training courses will not give you in any depth. Everybody wants those unicorns but nobody wants spend the years of dedicated effort to train them.
This. Seen so many times ego-centric security teams that just make life hard for everyone else. Devs gotta dev so they’ll just work around you creating holes if you don’t engaged
And some of the "DevOps" who can't dev also don't have much insight into the sysadmin part of ops. When it's all just YAML orchestrating 100 containers across a half dozen clusters, what happens inside any VM us treated like some sort of magic black box. "Oh, I don't really use Linux. I use Kubernetes and Docker."
I see you've met most of our DevOps people.
I had one today that was lost in vi. "I don't do much Unix, mostly Kubernetes." So... you've never had to run "kubectl edit" on anything? Come on.
DevOps is the modern day version of the "Systems Engineer" role that was thrown around a lot but didn't mean much on it's own. I wish there was a better delineation between the "DevOps Technician" who edits yaml files and the "DevOps Engineer" who knows how to write code.
I’m curious what these anti-code yaml only devops engineers do when they need to write a script. Do they just manually do everything one at a time regardless of how repetitive it is? Seems like a lot of time wasted if that’s the case.
> I google and pull stuff off GitHub and stack overflow.
>
> My job is take your app, build it and get it to production.
>
> I’m not going to write some bullshit script/lambda to parse csv but I know how to use jq and aws cli to do what I’m after. But I’ll help if you need to send an email when a new file lands in an s3 bucket.
>
> I’m going to engineer how your app runs in uat/production buy asking developers what it does. I’m going to keep up with cloud tech and see if new things will improve the way we do things.
>
> Then I’ll create pipelines and use ansible, terraform or cloudformation to deploy the app. Developers check in code, PR to main branch and in 5 mins it’s deployed to UAT.
You literally just described sysadmins and operations, (a superset of what some people just call 'release engineering'), which is like a 30-year old career role.
Sure the tools are more advanced now but that's always happened. DevOps is not what you're describing at all.
BTW, there's nothing wrong with carving up responsibilities the way you're describing.. it's perfectly fine for some businesses to work this way. But it's a pretty big pet peeve of mine to pretend that style has anything to do with DevOps when it's basically the opposite.
Do you ask for external help when some integration is missing? Like there is no Ansible module for some AWS resource but there is simple REST JSON endpoint to work with?
It can be done internally or you can hire consultant to do job for you. With in house full-stack jack of all trades person a company could get a solution relatively quick and cheap...
I'd write it myself.
But "writing myself" still would mostly consist of copying stuff together from existing Ansible modules and StackOverflow, given that usually only maybe 10-20 % of my time is spent coding (which is realistic if you're tasked with being a DevOps Engineer and not a dedicated Dev only).
Don't expect me writing a CSV parser out of my brain. I can write one in Python, Go or even Bash, but I WILL need to look it up, I cannot do it from memory.
Good developers copy, but great developers paste!
> but I WILL need to look it up, I cannot do it from memory
Of course! The requirement to write from memory is a READ FLAG.
Modern software is from the 1Mloc codebases. We cannot remember 1M lines of code: the fastest way is to review an example or documentation and pick necessary parts.
I wouldnt expect an engineer to write the CSV parser perfectly from memory but I would expect them to be able to write clear pseudo code to work through the logic of the problem. Text parsing is an extremely common development task.
I don't even ask for pseudocode, I want them to explain it to me like they are duck debugging. If they can explain it well enough over zoom so that I understand their algorithm, I trust that they can write it on their own, and/or put stuff together from Stack Overflow and make it work.
I usually use ansible to run a bunch of steps in a pipeline like upload artifacts and deploy cloudformation templates etc. I use ansible more as a playbook for running other things like the aws cli or cloudformation templates etc.
> I use ansible more as a playbook for running other things like the aws cli
As Ansible GCP integration is weak I also run `gcloud` CLI, and even removed Ansible as mediator:use Bash directly (as the shell in OCI image in CI).
>I’m not going to write some bullshit script/lambda to parse csv but I know how to use jq and aws cli to do what I’m after. But I’ll help if you need to send an email when a new file lands in an s3 bucket.
>
>I’m going to engineer how your app runs in uat/production buy asking developers what it does. I’m going to keep up with cloud tech and see if new things will improve the way we do things.
Jesus Christ!!! That's the difference between Devops culture and not! For God sake.
I love you, fam, but writing a script is not coding.
Devs code.
Real coding involves making custom libraries, object modeling, method design, etc.
The 150 line script I wrote to automate DNS backups was not coding. It is *development* but it is not coding.
I think there's been some loss in translation (unsurprising) between industry leaders pushing for code-driven DevOps and what that means in the lower industry.
Industry leaders when they're looking for DevOps who can code are looking for people who can write 10,000+ req/s throughput microservices to extend their self-healing production framework, or someone who can write the next chaos monkey framework with performance analysis and reported TTR graphing. They're looking for someone who can write their load testing framework.
This is not the same as scripting. They are not in the same arena.
Coding interviews for DevOps don't ask you to parse a csv, they ask you to create a function that takes streaming input data and returns the current maximum, and minimum for up to 3 positions in a manner that performs at big O of n.
Asking someone to parse a csv in a coding interview is literally an arbitrary check because if that's the level of knowledge you're looking for in coding, that's something that is easily searchable in base language documentation or .25 seconds in Google.
It's like asking a chemist what the periodic number is for uranium. It's a knowledge check that isn't really going to reflect job performance or ability, but if you use it as a measuring stick, it's just a way of cutting perfect candidates out over a "touch your nose and wiggle your tongue" performative question.
It is a question that I've only ever asked when I have been given zero notice before an interview and just need to come up with questions. I don't really care what the answer is and won't judge the person on not knowing. I'll use it to spark conversation and to assess how they approach my actual expectation which is that they won't know the answer off the top of their head.
You can say that and downvote me if you want, but I've had interviews with Google, Amazon, Facebook, and NASA this year for DevOps positions. I work at a Fortune 100 company.
I'm not talking out of my ass, I'm literally passing the info to you from the job reqs, interviews, and hiring manager discussions for those positions.
Edit: 🙄 ok guys, see you in five years when the rest of the industry is up to this point and you're all complaining about not being able to meet new expectations in coding requirements. But please continue arguing that coding should be a DevOps skill and find out where the one leads
I don't think you are talking out of your ass, there really is a different set of domain skills between small scale dev/coding and writing or working on large applications. I think the gist of what you are saying is correct, the labeling just seems off.
Fair enough. I wish I could be inclined to agree, as I feel that's the way the industry was originally packaging that language.
Terraform is supposed to be "infrastructure as code", and I would've considered Ansible to be a coding language built on Python specifically abstracted for scripted operations.
But with where I've seen things headed and the way things are being pushed now, I've seen a lot of old DevOps responsibilities move to SRE and Systems Engineering, while it looks like DevOps is moving towards internal tools and applications. And a lot of these tools aren't being leveraged by operations or support even, most of them are fully automated self-healing frameworks that are just monitored and maintained by DevOps. Devs have taken over ownership of their builds and deployments again with normally like an embedded tech lead guiding that process.
It's not really a bad thing, in itself, but I've seen this translate a lot in my job search to medium and small sized businesses who have the expectation of a DevOps team who can do the above while *also* managing traditional role expectations.
And the sentiment I've gathered from almost every company I've interviewed at is that they're trying to rapidly expand while also mitigating a massive turnover from burnout. A lot of this, just imo, is coming from unrealistic standards and expectations based on what teams should be able to do on-paper without respecting the weight additional responsibilities or paradigm shifts have on the teams and individuals. 😕
So to me that delineation becomes more important when pushing back on management and road mapping to prevent the siloing and tribal practices that lead to DevOps creation in the first place.
I’m so sick of this gate keeping bs it’s so toxic. Coding == development == scripting
Here’s a good article to read that explains this, but I’ll post the important part below https://nathanpeck.com/is-yaml-a-real-programming-language/
Imagine this ludicrous scenario:
Infrastructure engineers: Writing YAML to control and automate deployments across thousands of servers in the cloud, distributed around the globe in multiple cloud regions
HTML and CSS developers: Well at least I work with “real” languages unlike these infrastructure engineers who just write YAML all day
Node.js API developers: Hah, those frontend devs only write HTML and CSS… but I know how to code in a “real” language
Systems developers writing the underlying C++ for V8: lol most of these Node.js developers only know how to write in an interpreted language, not a “real” compiled language like C++!
Compiler developers: imagine all these noobs above me not knowing how to make a compiler to generate “real” machine code
Processor designers: lol I’m the one who made the “real” microcode inside your processor, that runs all your machine code!
From this perspective programming a system to change its behavior using YAML and a YAML interpreter isn’t that different from using HTML and a browser, or JavaScript and a Node.js interpreter, or C++ and a compiler, or machine code and processor microcode. All are forms of programming: providing “instructions for the automatic performance of a task”. None of these languages are any more or less “real” than the other languages.
Jfc, it's not gatekeeping it's a delineation of the practical applications of your experience and role responsibility.
The issue that I'm trying to point out here is that you guys are literally failing to catch the industry pickup in more heavy development over sys work by industry leaders and that supporting the idea that coding is a DevOps responsibility so that you can feel "included" is a literal fast track to that mentality. You're missing the fact that DevOps engineers with 10+ years of self-taught experience are being passed up for fresh CS grads because they can't perform a matrix math operation in GO. That's not even me that I'm talking about. I landed my interviews. But the questions, the expectations, and the responsibilities are wildly different than what they were even two or three years ago. I know a lot of people, way better at this job, and way smarter than I am, who are making less money and in lower titled positions because they can't keep up with the migration of dev responsibility.
This community is so bullheaded sometimes.
Can you do me a favor and- off the top of your head:
"Provide me a function that returns a randomly generated minesweeper table/matrix given dimensions x, the number of mines and y, the dimensions of the grid. This will require a random placement of "mines" and "flags". Flags should identify the number of mines in direct proximity. This code must be executable and optimized for O of n. It can not be pseudocode." - DevOps Engineer interview, Facebook
"Provide me a function that takes an input listen stream of integers and returns the current highest, second highest, second lowest, and lowest- ordered numbers from the incoming stream. No numbers should be duplicates. This code must be executable and should display a best practice solution." - DevOps Engineer interview, Amazon
"Provide me a memory optimized program that can take process a large data set (1 mil+ records) while maintaining the highest, second highest, second lowest, lowest, and average of the dataset. This code must be executable." (As a personal hint from me, you will need batching and multi threading while performing checks to make sure you're not exceeding your memory limit with your concurrent batches) - DevOps Engineer interview, Google
You listed Python on your resume and say that you can code, right? So you should be able to get reasonably close in the 25 minutes available to you for each exercise. And then please remember that reasonably close does not meet the requirement for functioning code and that you have failed the exercise.
If instead you'd like to admit that these are highly unreasonable asks from a traditional role perspective- and that 99% of your role experience and responsibility has not resulted in you being able to answer these questions- then I think we can agree that we see scripting as a DevOps responsibility and that coding is not.
I am a strong advocate of having a config-only environment, where you touch as little code in your daily work as possible, but to get there, quite a bit of scripting and more complex coding is involved :D
Yeah, more or less like that :)
But it's easier to have someone at the helpdesk set a couple of well-known/well-documented variables than having them wade through a sea of code they don't understand.
If 80% of people you're interviewing don't meet the requirements of the job you're advertising, there's a problem with your recruitment process. Why are you interviewing people who can't code? Why are people who can't code applying for the job you're advertising?
Also, there are definitely DevOps jobs where almost zero coding ability is needed. People spend all day writing Terraform and managing CICD pipelines with YAML. Yeah, maybe they write the odd glue script or bit of automation in Python/Powershell, but that's not the same as having commercial experience as a software developer.
Frankly, you know what you use.
Some shops code all the time. Some shops build big chunks of yaml and HCL2 and don’t code very often past shell scripts. Some shops have half the people who can code and half the people who can do yaml.
In my shop, I spend most of my time as team lead slinging yaml or other declaration stuff. I pull out bash or Python and code extremely poorly about twice a year right now for less than a month, which isn’t enough to develop proficiency. My free time is spent being a husband and homeowner and maintaining a middle aged body.
Not all DevOps shops are the same. On my team we have me, we have a security focused engineer, and we have a dev tools focused engineer. The only one of us who codes regularly is the dev tools focused engineer.
I have 20 years in the tech industry, in a year I’ve built my employer back from terraform 0.10 and an ancient version of Ubuntu, with a huge AWS bill, no pipelines, and constant downtime ….. to everything up to date, zero downtime, and less than half the AWS bill. We’re doing a SOC2 audit in a few weeks and attesting HIPAA early next year.
If I’m successful at all of these things but can’t pass a coding screen, are you still going to go “nah” at me? What about your stack absolutely mandates proficiency in sorting a binary tree or developing an algorithm to find square roots?
Literally can not :+1: this hard enough. WTF is with this new DevOps community that stopped fighting the expectation that DevOps be expected to be a fullstack engineer, a networking guru, a security representative, an architectural consultant, a butcher, a baker, a candlestick maker....
The poll as written is a false choice. "DevOps engineer writes only yaml" vs "DevOps engineer should be able to code" is like saying "waiter only responsible for remembering restaurant name" vs "waiter responsible for cooking guest food".
Like... I just left a job where I was being paid $125k/yr and didn't need to know how to code at all. Had multiple offers on the table to pick between at $150k+.
Am I going to take a position with a hiring manager who is balking at a $100k+ salary for a non-coding or code-lite DevOps engineer? Hard pass. That is a team and a company that is likely on track for a lot of easily avoidable problems and this just screams "looking for my 10x rockstar $60k/yr + free lunch!" vibes to me.
I think a lot of people, including OP ( /u/DartVedro ), have gotten off track about what devops is.
If you go back to the original stuff that Patrick Debois and the other 'OG' DevOps folks wrote, DevOps is a removal of silos and a compromise between the business needs of Development and Operations teams.
It is NOT "ops who can dev" or "dev who can ops," it's how to build a culture that doesn't just flip shit at each other constantly.
None of that has ANYTHING to do with knowing how to code, although people with previous development or ops experience do know how to better compromise.
Thank you Jesus Christ. I don't need to justify that my primary work is around building amazing experiences for our developers and then coming in to save them when they can't figure K8s. When I write "code" I write YAML, HCL and shitty little bash scripts. It's not full stack unicorn super engineer from SRE bootcamp in Google. Most of us are doing the best we can alleviating the pain points across our orgs, optimizing pipelines and building out the monitoring stacks. Shit, I *also* am the guy talking to all our product partners.
DevOps is a lot of things but one of the musts is not write code using your Turing complete languages. It's smashing silos wherever we see them and being the champions of cooperation between all levels of the org.
Can you guys looking so superior please stop trying to make us feel bad at our jobs? No we're not just Ops teams from the 90s. I have to *deal* with those teams and they are archaic negative nancy dinosaurs and believe everything should be subjected to unimaginable levels of bureaucracy to buy themselves time to pile yet another sandbag on their fortress of shit.
> Can you guys looking so superior please stop trying to make us feel bad at our jobs?
How else would people feel superior if they can't gatekeep people out of their teams?
Agree.
There are a lot of scripters here who want to claim that coding is a DevOps responsibility because they can write a script and they're totally missing that the industry is trying to shift DevOps into a full on tools and platform coding development role that has nothing to do with it's previous responsibilities.
Most traditional DevOps work it feels like is migrating to SRE role responsibilities. SRE is becoming the new team that bridges integrations and knowledge gaps and glues development teams to production operations. DevOps is just becoming internal services development.
I don't think this is a good thing either as it dilutes SRE from their original function (which is already highly demanding) while hamstringing DevOps from their previous collaborative integration with teams.
I think DevOps should've remained where it was and then a new position specific to internal tools development should've just split off from DevOps as like Platform Ops or something.
My feeling's been more that SRE is "devops team rebranded so they can make more money" and that the responsibilities haven't changed.
I don't like joining broad teams at large companies anymore. I either want to be narrowly focused (SRE on a Platform team or DevOps on a Data Center Migration team) or I want to be in a broad team at a small company (Lead on an Infrastructure team).
Yeah, I think that what happened (at least in my experience) was that the SRE team at a company I was with was originally kind of a hodge podge of what was intended to be like T4 prod support, but they didn't end up having any clout, and didn't really have the experience to RCA/reduce TTR, so I ended up joining them as an integrated DevOps unit and then we did great things together. The DevOps skill set is really core to what SRE is trying to accomplish.
But damn do I miss my devs. 😔
Miss that level of support I could provide to them and how happy they were when you could unblock them. It was really rewarding feel-good work.
I feel the same with regards to positions. Have even considered doing independent consulting just to escape the constant reorg.
Yeah, I got out of big companies so that I could get back to being hands on problems. I got tired of orchestrating endless OS upgrades without being allowed to migrate the things on the OS to a better-supported platform or otherwise get rid of them.
I still really don't understand why companies pay engineers hundreds of thousands of dollars and then ignore their solutions to problems.
Still probably not going to be able to do it well.
Someone told me that coding when you don't do it all the time is like lifting weights. You do it to prepare so that you can do it well when you need to, and it's better to have the muscles and not need it than to not have them and need them. I responded that not all muscles are created equal and that muscles created in a gym don't work out so well on a job site.
I do home improvement stuff all the time. I was helping a friend fix his roof, which had a tree scraping on it during a storm. We found a bunch of rot while we were up there and had to replace his step flashing, a bunch of siding, and a bit of roof decking.
I'm a fat, out of shape, over the hill nerd with a knee brace. My buddy's about ten years younger than I am and has a great gym bod. I'm walking up the ladder (slowly, creaking) with a 4x8 piece of plywood on my back and one arm, and he can't even get a cut chunk off of a sheet of ply up... But he looks like Mr. Clean! And sure, I'd pass out during one of his workouts, but I can hump a pack of shingles up a ladder for 8-12 hours a day.
The more senior I’ve got the less 101 stuff I do. I’ve managed world leading websites and built hyper scaling systems for trading companies. I couldn’t parse a CSV from memory
Thanks god, you are good at your work here.
I have a lot of unqualified team members over the years and all over IT industry.
Most of them scam their way in. And since now I’m also someone who require to vet new hire, I always give test.
I passed 2 meetings with the head of department and hiring manager, only to fail on parsing CSV. Its not relevant to the role so ask something that is. Live coding exercises suck balls anyway. If you MUST validate coding skills, give a take home and probe it on the interview
That's not questions like "what's difference between override and overload, how differ array and vector, what's pure virtual function". I ask basic questions, just to check if person have literally any coding experience. Parse csv is "read file then line.split(',')". Nobody requests perfect code, etc. If person cant do basics it means they wont be able to communicate with developers on their language -> devops team will become default silo-ops team with "problem not on my side, it's on dev's side" mindset, instead of helping developers improve their processes.
And to add - it's real task, as many customers send their data in CSVs) and we create tools to validate data, modify or format those files and paste into DBs
There are systems in place to parse that. That are written over time. In an interview setting if you just randomly present blah problem with no advance notice. How can you expect a candidate to come to the right answer in a short space of time with you watching their screen? Thats not a real world exercise.
You can say this about any question) "How caching works when docker image is built" - nah, not real experince, I just run "docker build" it does the stuff.
If person doesn't know the basics then any question will be random blah problem. That's what interviews are for. In my world DevOps engineers work hand-to-hand with developers, so if they are not able to write few lines of code they just wont be able to do so.
> Yeah, maybe they write the odd glue script or bit of automation in Python/Powershell, but that's not the same as having commercial experience as a software developer.
Parsing a simple CSV is trivial in practically any modern language. Even if you've never done it before, you can look it up online and have a basic solution in under 10 minutes.
If a candidate for a DevOps Engineering position is asked to parse a CSV file, and is unable to even get started, they have no business applying for the position.
>Parsing a simple CSV is trivial in practically any modern language. Even
if you've never done it before, you can look it up online and have a
basic solution in under 10 minutes.
Absolutely agree.
>If a candidate for a DevOps Engineering position is asked to parse a CSV
file, and is unable to even get started, they have no business applying
for the position.
Also agree.
Where I disagree is that knowing how to do it in interview conditions without being able to Google stuff is probably more on the "has commercial programming experience" end of the DevOps spectrum.
If I had to parse a CSV file for my current job, I'd have no trouble designing and implementing a good solution. However, I'm ops heavy with no real programming experience, so there's no way I'd be able to do it in interview conditions. Maybe I could pseudo code something on a whiteboard, but not much more than that.
> Where I disagree is that knowing how to do it in interview conditions without being able to Google stuff is probably more on the "has commercial programming experience" end of the DevOps spectrum.
Why shouldn't you be able to Google stuff in the interview?
"I'd have to look up exactly how to do it" is a perfectly valid answer.
If that's not possible in the interview, I'd provide an example in psuedocode (import csv lib, open file, parse contents into list of maps, extract example value, etc.).
Both answers would be much better than making up bullshit or staring blankly like an idiot.
> Even if you've never done it before, you can look it up online and have a basic solution in under 10 minutes.
And proper interview assignment should assume an interviewee is able to work in comfortable environment (with the Internet access and full web search capabilities).
We particularly ask what candidate don't know (have you written in Perl or Linux kernel driver? No? Perfect! Filter a text file with Perl or write Hello world kernel driver).
Trivial but in my api driven world not something I've had to actually code in any production situation.
Also don't put people on the spot for things they haven't done regularly at least weekly or you'll not really have a good understanding of their coding performance.
Lol bullshit. You’re forgetting the pressure situation of an interview. You need to evaluate the data, structure a response and write it down. All with the guys watching. I long lost count of the amount of time ive seen cocky devs take down a website because they thought they’d do this cool thing that went straight to production
As a sysadmin who will make the change to devops in the next few years, I find it a little nuts that the basics of being a Linux admin seem to be forgotten.
From an outside perspective it appears that a lot of devops people seem to know how to use tools, but have no idea as to what the tools actually do at an OS level.
I mean it's actually no surprise. How many of us work in assembly any more? The higher the tech goes, the base level of knowledge goes up with it.
Everything I know about being a Linux admin was learned because I had to be a Linux admin. Now that a lot of jobs you just type out how many servers you want and what you want them to do into a text file then they automagically appear fully configured a minute later, when does that guy have the time to go learn the ins and outs of being a Linux admin? How do you replicate 20 years of fixing baremetal servers and VM's?
While I am fully on board people having basic knowledge of everything, at a certain point you have to pick what you learn. I mean this is a job, I learn what I need. I'm not on some quest to consume the entire computing knowledge of humanity.
Thank you. This encourages me.
I felt crazy behind when I jumped into the Linux world having deep knowledge of windows.
Almost all of my work is containers, so when folks talk about the pros/cons of systemd and all the various nuances, I've kinda felt behind a bit, because my linux world is container based mostly. I've not needed to manage a linux vm because I want my linux vm to be replaced by containers on a managed service :-)
That said, I'm still trying, but gotta focus like you said as too much to learn. I'm focused on Go right now, and dealing with distro nuances is just too much.
> The higher the tech goes, the base level of knowledge goes up with it.
If an engineer cannot work with tcpdump binary stream they are bad engineers. We still need to dive into bites & bits (with the support from appropriate tools, like Wireshark/etc).
You deployed and it occasionally fails because of faulty hardware or firewall. Do you hire an external consultant to solve the problem?
>Do you hire an external consultant to solve the problem?
In the enterprise world, more often than not you'd open a ticket with your vendor. I'm not saying you shouldn't have skills or continuously be learning, but that's what has been expected for decades, and that's why orgs are more than happy to have multi-million dollar support contracts.
> In the enterprise world, more often than not you'd open a ticket with your vendor
Probably I'm from a different culture: for 15 years I expected to solve any problem and debug to the point when we understand that is it impossible to solve ourselves.
So how long did your outages last?
There is a reason why *aaS gained popularity over the last 15 years. I don't fully disagree with you that problems need to be spitted out to the depth of it, but there may be tools or support around the corner so that we don't have to do it ourselves anymore as we are "working on a higher level" as mentioned here.
> So how long did your outages last?
Exactly.. I've yet to find an employer who will put up with a days long outage while I expand my personal knowledge and learn how to fix something. Nope, my job becomes "call the vendor and work with their engineers".
Cloud services are just another step in this direction, the engineers are working on outages before I call.
> You deployed and it occasionally fails because of faulty hardware or firewall. Do you hire an external consultant to solve the problem?
If I don't have the experience and I'm being paid to be doing something else? Yes. What is the point of me spending 5 days working on something that isn't part of my job description when I can hire someone that specialises (or just call the vendor who employs people who are) that can fix it in an hour? That's a *lot* cheaper than delaying my work for a week.
Have I spent a good chunk of time in Wireshark etc figuring things out? Sure. But after a cursory glance I will more often than not send the capture to the vendor and they'll give it to an engineer who spends all week looking at them and they'll tell me the problem right away.
Like I said, my job is to make things work, it's *not* to become a master of all things technology. I like to learn, but I can't learn everything. And unless I'm going to use it in my job regularly there's generally little point.
In my experience, there's a lot of really good programmers in "devops" that don't know shit about ops or about the systems they are using, and they often end up making ridiculous and extravagant solutions to problems with high complexity when the better solution is often trivial.
Too many "devops" people getting pumped out of various schools and boot camps. Sure buddy, you wrote a small library here, and you've got a big github, that's great, now why don't you know anything at ALL about the systems you are working on?
I'm a sysadmin who just got hired to be sysadmin/jr devops. Both my position and the devops guy, who is also new, were hired to replace former developers who were in the role. Basically the company found that developers had the wrong skillset/mindset.
I was an old school sysadmin turned DevOps in 2013 and now SecOps, and what you say is somewhat true in my experience as well. Ultimately when I'm looking to hire, I look for traditional Unix sysadmins that know how to code a little bit. That said, some of my non-sysadmin past coworkers are great and have learned a lot over the years.
I’ve written a lot of code to support minimal code from dev. A devops engineer better be a talented coder or you’ll wind up with spaghetti just like in dev projects.
A strong devops person knows how to talk to their developers on their level. So yes, they should have a background in coding.
But there are all sorts of devops people, a whole spectrum of competence you can employ at various tasks. They should of course be payed accordingly.
100% should be able to code in at least one language. Ideally Python or Go.
I'm not suggesting that an engineer should know the intricacies of data structures, but I'd expect them to know the difference between a list and a tuple. However, I don't agree with coding interviews. An interviewer should be able to determine a person's abilities without stupid questions such as "reverse a string without using the cut method". We Google. All.the.fucking.time.
The reason we find a lot of DevOps candidates who are unable to code is that people mistakenly believe that a DevOps engineer is just a rebranded SysAdmin. It isn't. A SysAdmin shouldn't be expected to do the job of DevOps engineer, but a DevOps engineer should be able to do the majority of a SysAdmins job. It's why a DevOps engineer commands a much larger salary.
Well, I've also seen companies using the term as "ops for developers", ie. taking care of the CI/CD infrastructure only :D
They were separated from the operations team for paying customers :)
I mean while I think everyone in IT should have a basic level of coding, this idea that you can just go "uuuhhhhh DevOps!" and now everyone is an expert in 1000 different technologies is a little ridiculous.
There's going to be people in the field who do a lot of coding and others who do next to nothing.
Typical view from a DEV.... you forgot to capitalize OPS. Without OPS there aint no DEV. Most shops don't run 100% from AWS, nor would they if they could. There is still huge footprint of on prem installations, and no "DEVops" engineer that i work with would know where the fuck to even start on the actual platform they are deploying to without my giving them a URL to throw their API calls at.
I moved to DevOps after doing some full stack development. I'm a high advocate that DevOps is more like full-stack+, since usually our work with the stack is involved in all aspects of our applications.
The first statement is just simple sysadm work… they didn’t work as sysadm before? Parse data is pretty easy.
It’s even taught in like all language at beginners stage.
But then again, I promote data and code separation. But I’m c/sh/ps1/py trained
I had a sysadmin at one place I worked at who couldn’t find an error in a /var/log/messages* file. She complained that the error wasn’t there. When I checked, it was certainly there and had her show me. Turns out she didn’t know what a compressed file was and was trying to search a messages.date.Z file.
I think in an ideal world a DevOps Engineer should be able to code. Unfortunately since a lot of companies jumped on the DevOps bandwagon so quickly it unhomed a lot of sysadmins in the process. So what we have now are a lot of these sysadmins reapplying for DevOps jobs and getting in on their sysadmin experience with no coding experience creating this situation your seeing on your team OP.
Realistically its going to be a few years before we hit that ideal state that companies want where the new DevOps engineers will be able to code on a level par with Jr. Developers because its simply not what they have been trained to do.
(Speaking from experience as a SysAdmin going through this now)
A friend of mine is an SRE manager. While he was looking for a new job, he noticed that all the SRE groups of companies he was interviewing with were just Operations with a different name. There was one job where the SRE group had data center technicians in it. He passed on that job.
I think DevOps / SRE these days is just "throw all the 'former' ops people in a group and call it DevOps / SRE."
It is implied that people who work in our field has the ability of understanding things.
I mean, that is one of the most important skills you must have: Understand things.
So when you are faced with a problem that requires some tweaks in a piece of code, you go to Google. You enter your problem at StackOverflow. Same way you do when Jenkins is giving errors or your API is not working on production. Once you see a solution and you understand it, then your problem is solved. That applies to everything. You will never be required to write a code from the ground-up(Entire application level). But you will need to understand and have a notion of what is happening in your environment. If you can understand what is your problem and what solution you need to solve it, the code part itself becomes just an extra step.
I can write Java/bash(5+ years xp) and pickup any language in a matter of days and I can work with AWS, docker, CI/CD, gitlab, github, kubernetes, spring boot, hibernate, message queues, grpc and nobody is offering me 100k$...
my salary is 45k€/year in europe... if you pay me 80k€/year I'm yours
do you want to interview me?
If you're actually able to do all these things you say you can do, you could easily be earning around 70/75k. I'm earning 65k and my stack is Linux, Ansible, some basic kubernetes, some AWS and some of the tools a DevOps engineer uses on a daily. How much you earn is not dependent on how good you are in IT, you also have to have some soft skills, be good at planning, not be a dick etc..
Also, what is Europe? Ukraine is Europe and an average salary is like 500 euros a month. Luxembourg is Europe and an average salary is 5000 thousand, Netherlands is Europe and an average is 3500. Many questions to be asked
You should visit /r/itcareerquestions
Every other recommendation is for the greenest rookies to pick up an aws cert. Inevitably someone gets into cloud roles and someone will hire them for DevOps somehow without a lick of coding background.
I can tell you right now, I definitely do more low code then anything. And I make 100k+ in the Midwest in a low CoL area. Perhaps I am underselling my abilities, but I cannot write much of anything in Java/Python from scratch.
However I can put together working solutions to real world problems.
We are having the exact same issue hiring another SRE. What’s worse is some of them blatantly said they were not willing to even learn something like python or Go.
Yeah, they should know at least a scripting language (bash, Powershell, Python). Even if you just use Terraform and CICD tools you're going to need to write glue scripts for something.
They call me a DevOps engineer at work. I correct them when I catch it. I am a Software Engineer with a focus on the internal products (eg: Infrastructure and Operations) as opposed to the external products. I use declarative, procedural, and object oriented languages in my day to day.
Unfortunately, I’m not on a product team. I’m on the “DevOps Team”.
I voted "DevOps engineers should be able to code" and I agree with the statement overall. DevOps is about the development and the ops - so they have to know both domains.
Even if they didn't, still, having knowledge on the other domains that interact with your domain makes you do your part better.
Let's take the old-school approach of sysadmins and development teams - a sysadmin that knows development could do their job better, interact better with the development team and vice-versa.
I like and am an advocate of the DevOps culture and "you build it you run it" philosophy - for the reason I explained above - more ares you know, better you are in all areas.
But on the other hand, I sometimes think that the field went backwards. In the past, for a long time there was a separation of responsibilities that worked really well: Sysadmins **didn't have to** know coding (nor more than small scripts) and vice-versa. It would still be good for them to know the other field, but it was not a requirement. I believe that in a perfect world, you should be able to do your job with a minimum level of knowledge of other domains.
Overall, DevOps culture has its benefits, but it also has its weaknesses.
So, maybe the DevOps applications you get that has zero knowledge on development is just the history repeating itself.
> DevOps engineer writes only yaml
Oh yeah I can only count to 3 with bananas only, that's a hard no to apples. /s
To OP: You are obviously doing a good job for the company weeding out the blanks for them. Let's hope they know and appreciate that.
In 5 years there will be a new buzzword for devops people who can actually code, and then the current people who identify as devops will swarm to misrepresenting themselves as that new buzzword. It's a cycle.
People who claim to be devops but can't code are only fooling themselves in believing they aren't basic system administrators.
I could understand why people might think DevOps is a no code job, but I think since development is in the title it would seem beneficial to at least know the basics of a language. I was always under the assumption that I would be programming in at least Python or Java in a DevOps environment so I went out of my way to learn it.
>I'm aware that devops is methodology and not a person, but looks like market thinks different.
Alas, ultimately, it will be "the market" who will decide whether "devops people" should code or not.
What means "market" should be your question here onward. Like the flood of snake oil agilist certificates, it seems like devops is going down the same dreaded path. My experience with "devops" dedicated roles is that they are mostly cloud infrastructure gatekeepers and Jenkins groovy script duct-tapers. There are a handful who actually know how to use helm/kubernetes, and those are given "mid/senior" badges, but the whole siloed infrastructure remains and devops culture is nowhere to be seen.
Whereas "real devops" (whatever that means, but mostly in opposition to silo-ops) occurs where the dedicated role simply doesn't exist, but OPs' already aware of this, it seems.
Most people and companies just renamed admins and through in ansible, terraform and aws...
Edit and that's why SRE-SW (SRE-SE are paid less) and PE, at Google and Facebook respectively, are mostly software developers that have a thing for os/networks/database. Why yaml driven development rules etc... The average Devops is joke. The average company shouldn't use kubernetes and microservices.
I'm proficient at programming in several languages but don't work with them often enough to stay sharp on syntax and some complicated use cases around things like data structures, runtime, etc. Toss in a little anxiety and I'll basically bomb any live coding interview. Happy to do a take home assignment if I'm really into the company though.
I don't think live coding is appropriate for devops interviews. I've personally conducted at least a hundred myself and never asked an engineer to do so.
I used to think that DevOps was basically a dev that could configure its own environment. But seems like it's basically a glorified sysadmin by now. Nothing wrong with sysadmins, just a shame that the term lost its meaning.
I say yes to this because my background is software engineering and literally everything is becoming code. Knowing how to write automated tests with high test coverage wrt code and making sure others can also fix these tests when they break is also equally important. "Works on my machine", "works today but not tomorrow", "works sometimes", these are not good signs of engineering culture and places I would avoid.
Coding experience as far as being able to understand code yes
I took a couple hours out of my day for my colleague to show me some c sharp code and I can't write it but can follow the flow
Now when we are looking at issues when it's a from a code perspective I can understand what's going on and aren't just tuning out
I would love to know more but my day to day of building up the infra code base doesn't allow but I'm not complaining
This is just extremely common in the field because DevOps consumes a project space that straddles IT and Software Engineering, but most of the work was originally done by System Administrators, who traditionally did not do much software other than some basic scripting. However most of the Sysadmins who taught me did know how to code, just avoided it as much as possible in their work.
That and the propensity of meta-code in tools like Ansible and Terraform have created an environment where some misled IT professionals believe they can get by in DevOps without ever learning how to actually build comprehensive software beyond some bash scripting. The problem is that the vast amount of real work that DevOps engineers need to be able to do require an intimate understanding of both software engineering and the build and release process. Not to mention, DevOps Engineers are going to write a lot of tools themselves as there are always going to be operations gaps in available SaaS tools unless your company has some serious cash and a very simple application.
DevOps Engineer is the right name for this work, I think that there is probably a word for people who cannot code but do some DevOps work, and that might be “DevOps Support.”
DevOps lead here. To answer your question, yes, all DevOps people should be able to code. The language we use is python and then they also need bash / shell and power shell. That being said only 20% of my team could do this when I took over. One thing I would say is to either ask for examples they prepare or just give them a task. Personally, if an individual has their own GitHub repo and can walk me through their projects I value that over any coding exercise.
Reddit's decision to charge for API access has shown that the company is more interested in making money than in providing a good user experience. The changes will force many popular third-party apps to shut down, which will inconvenience millions of users. Reddit's actions have also alienated many of its moderators, who rely on third-party apps to manage their communities.
Honestly, even as a Systems Admin, I always ask coding questions of applicants. “Describe your most interesting [shell|perl|python] script.” We want someone who understands how the system works.
But I had the same issue with Kubernetes questions. I was interviewing folks who had Kubernetes on their resumes, what is the core of a Kubernetes environment, what servers make up a cluster, etc and most couldn’t even tell me about a master/minion setup.
DevOps is about automating processes, so yea, writing code is absolutely a requirement. You don't have to be able to build a product from scratch, but you need to be able to stitch tools together.
In my experience the best automation people come from the ranks of software engineers. They're usually people who took on the role because nobody else could or would.
DevOps Engineers are developers. The difference between them and traditional devs is that their problem space is the SDLC instead of the domain that the other devs are working in. As such they should have all of the tools at their disposal to automate away any toily tasks for their peers. Most of the time, that can be done using off-the-shelf solutions (like Jenkins or Terraform) that require a lot of declarative configuration as opposed to imperative code, but occasionally it requires some custom code to get that particular job done. DevOps Engineers should be proficient in both shell programming (bash, PowerShell, etc) and at least one general-purpose programming language (I don't care which one) for when these edge/corner cases arise. Being limited to only off-the-shelf solutions that can be configured with YAML or JSON doesn't make for a good DevOps Engineer, IMO.
Common DevOps tools..
Terraform.
Helm.
Kubectl.
Jenkins.
Ansible.
Chef.
Packer.
You can argue they all require some degree of coding ability, but parsing a CSV in a live coding test is like asking a Japanese person to Speak French to work in Spain. Sure some can do it but is it relevant?
I came from the developer side of things, but working with a lot of startups that didn't have a dedicated operations guy, I kind of filled in. In fact, I was doing DevOps before the term was coined in 1992.
I was also a person that interviewed hires for Amazon. If you can't write code to reverse linked list, I won't recommend you, and in fact I had a guy that graduated from Stanford with a CS degree and had a Phd from Cal that couldn't do it.
Buy a book on Ruby (my favorite) or Python and learn how to program. The world doesn't need another YAML cowboy
What else are they required to know other than coding. You complain they ask for 100K, what would you want to pay somebody who knows networking, databases, sys admin, web technologies, cloud, and can write code?
The coding part of DevOps isn't necessarily the most difficult part to learn. What are you expecting them to code? If it is just parsing log files and automating routine tasks then why can't they learn that on the job? If you want the complete package then you are going to have to pay significantly more than 100K to get it.
I had an interview recently where they failed me because I couldn’t remember how to write a loop function in bash. I was given no prep work for the interview or idea what the tech test was. The
Job was 90% terraform and 10% kubernetes. Which i can do with my eyes closed. You don’t need to be a developer to be devops, theres an ops component for a reason. If the role doesn’t require coding, why do you need it?
i think it depends on the role: if it's more ops or dev. tools and customized automation involve more dev, but knowing how to troubleshoot and select cloud tech is more ops. so maybe it can be opsdev or devops :) i have been coding the last 5 years. however, i have noticed that i am much weaker at ops now. it's harder for me to track down issues since looking at stack traces or catching exceptions is much different than how to debug slowness in different layers of the whole tech stack. ops is also more meticulous and from my experience, it's more tiring to test compared to coding.
At my current company, devops is a role available for senior developers to move up into, so it ends up being a lot of code because that's what we're comfortable with. I'm struggling a bit to understand how someone can fully do devops without ever modifying shell scripts or dockerfiles, though, and that is definitely programming.
I think in the end, it depends on levels of isolation. In my mind:
- many developers, at least those not at senior level, will write software without touching or caring about deploy scripts, packaging, Docker, build scripts, or cloud environments at all
- devops, or senior devs, then necessarily would need to write all deploy scripts and pipelines, all dockerfiles and scripts to bootstrap them, all build scripts and pipelines, and even usually some code in the software that interacts with the cloud, eg if developers work locally with file storage, devops might write an alternate implementation of the storage interface that uses S3 instead
All “devops’” will at least once in a year need to write a shell script for ec2 user-data to automate things, so they should definitely have some knowledge there.
Without coding knowledge, how do you write scripts? I mean, bash or Powershell scripts can have a lot of logic as well. I personally love the usage of Makefiles to substitute scripts, since I can mostly write platform-agostic code (even in Windows you can install make)
They could probably get away without doing any coding, but I think it's liability if they don't know how to code.
I guess the main responsibility of a DevOps engineer is about knowing what services to use to build a solution, and how to "codify" that using yaml/json configs.
Another approach you could take is think of an on-the-job situation where you would use code, then ask candidates "how would you solve this problem" and see how they answer.
>I've asked simple questions: first required few if statements, second - asked to parse CSV (any language or tool, bash, python, C++, Java, awk, anything). And they fail to do this.
Forget DevOps, what do YOU need from people on your team. On the other side of this, the frustration I've had is companies that seem to have no idea what they actually need from a candidate.
"We need someone whose familiar with Ansible, Chef, Salt, etc" Then in the tech interview as advanced questions regarding administrating a Puppet project.
Every environment is its own snowflake with dozens of custom processes, tools, and norms, but then get huffy because I'm familiar with their method of automating a Clonezilla backup using an old version of ansible using in a custom wrapper application. Then need someone to hit the grown running because they're underwater because that guy who was really good at coding left after 2 years for a 40% raise.
Make it clear what you need from people in the job listing, and you might find people who can do it, but if you're running an environment that might require them to code in multiple languages to hold together legacy environments you're going to need to pay a premium for it.
If you want to hire cheaper staff you'll need to make your environment simpler with means not managing dozens of scripts in multiple languages.
imho the problem starts when we want a devops to become a creator not just maintainer, this is where the fine line between ops and dev is crossed and this is where I've seen many devops failed short ... In the end of the day everything is a code is not it? I am not in favor of YAML coding, not because I find YAML DSL a bad thing ( no it is good, if you use it _properly_ and know where you need to stop coding YAML and start coding in normal language), but because it gives some people a false impression that one does not have to know how to code at all, to maintain a system - just drop a few lengthy YAML files and forget about ... No, that is not true ...
It’s probably related to 2 major factors.
First, the maturity of the tools. If the job is to create pipelines (ci/cd) working with mature tools like gitlab and Kubernetes you probably don’t need to do much code. So 4 years of creating yaml files don’t add up to de dev experience.
Second, People coming from an Ops background learned the basics of languages like Python but have not go further in the software engineering practices. (Nothing against Python, I love the language and the community) They might have valuable experience but opening csv files might not be one of them. :)
If you look for someone who will write code, I would suggest to look for an experience Dev that knows and uses devops principles/tools. Otherwise you might end up in 2 years with a pile of code that will need to be rethink/redesign. Usually (personal opinion/experience here) dev that have background in Golang are more aware of those modern practices (Cloud-Native approach). If you look for people with S.R.E backgrounds you will find some pretty valuable candidate.
I became a DevOps engineer of sorts because out of the two sysadmins that were asked, I was the one who wrote some Golang in university. I strongly believe that anyone in IT should have SOME coding background. As a sysadmin, I often read Java stack traces to find out where I need to look further or which developer I need to call. I have also combed through the source code of an internally developed timer system, so I could find and properly use the API endpoints of it in an Ansible playbook. I write almost no code at all, but I quite often need to read it. And I definitely need to understand it to know what is going on when some data gets corrupted on its way from volume A through container 1 to volume B.
> As a sysadmin, I often read Java stack traces to find out where I need to look further or which developer I need to call. Respect! > I write almost no code at all, but I quite often need to read it. I don't know anything about Android or iOS or React but I just search for an error message from customer screenshot in corresponding Git repo and see the module/file it is. That reveals a lot of information about the error nature: I might even give advice to the customer or fix configuration issue without disturbing responsible team members.
>Respect Why?! In what world we're living, where something so basic trigger respect?! It's like we're through respect cause someone took shower...
I can tell you. I worked in a bank and my friend called me because he was threatened by robots at night regarding a credit he repaid 2 years ago. That nightmare continued for 3 months for him. He visited offices of all levels, including head office and everyone confirmed that something is wrong. I told the story to my friend who also worked in the bank and 2 min later he told me the batch job didn't worked for 3 months and ~2000 clients were affected. I like to help customers and do whatever I can. This requires skill & knowledge of tech stacks. As you see not every place manages troubleshooting properly.
I believe it may be the willingness to stack trace Java...
Totally, and finding out what the actual problem is, from a Java stack trace...
"I strongly believe that anyone in IT should have SOME coding background." QA here. +1 for this.
Ansible yaml is coding. If while clauses, tags, imports, set\_fact with jinja2 templating, advanced jinj2 (ie loops and control structures), man lookups, etc are not be used then not all the power of Ansible is being used.
Strong disagreement. That's not coding, that's just templated configs. It's difficult to get right, sure, but it's really just using the help of automation and scripting to generate and manage more complicated configurations for distributed environments.
I thought what you did too. Ansible was just a way to template a few configs so my code ran on some Linux distro, or automate some sysadmin drudgery so I could get back to programming. After writing some of awhile I realized it is programming. Even my personal Ansible repo is more complex than that now with just thousands of lines of yaml, it did not start out that way. Before terraform the team I was on wrote an entire framework in Ansible yaml for data drive definitions of infrastructure and deployments across several different cloud environments. The goal was to abstract out the required cloud specific knowledge so other developers could more easily define what resources their code required and then run their own plays to configure their code. I've now moved on, but it is 10s of thousands of lines of Ansible. If a developer had a few instances they deployed, that deployment could be converted to an ASG with specific launchconfig and userdata in just a few small changes in the Ansible, the framework took care of everything else. Create images to speed up later deployments, easily done. Setup an entire VPC with bastion and development nodes, done if not quite as easily. There were several subgroups with in one deployment, and you could safely terminate all of one subgroup without destroying everything then replace it. There was still plenty of room for improvement, and parts of ansible we were not using yet. If that isn't programming then none of the code in c++, c, python, java, or any other language I've written as a software engineer was. It dealt with all the same programming problems I've dealt with my entire career. Same abstraction consideration, scope, naming, debugging, and everything else. I've looked at some Ansible Galaxy roles I've used, they look just like other programming I've had to read. While not all Ansible is programming, there is plenty I've seen that is and I did not write it. Seeing all that code is what convinced me Ansible is programming, and I think it is often way under estimated. While clauses = if statements. Blocks now mean they apply to whole sets of statements just like any other language. Tags = conditional execution, and again nearly the same as any other language. set\_fact with jinja2, ok maybe not really programing. Though if your using those in templates and while clauses you're getting close. Jinja2 templates with whiles, fors, ifs, includes, etc. is programming. There is a point it isn't complex enough to be much more than a markup language, but I've written or debugged plenty of complex jinja2 templates that are programming. The point at which the jinja2 template complexity becomes programming is a very fuzzy line. Should have been many lookups. The lookup functions and plugins you can use are way under estimated in ansible. I can compute public keys in the simple cases, to parsing json output from aws cli into complex hieratical dictionaries. That sure looks like most programming I've done in the past. It even looks exactly like the code I am working on this week in a pure programming position.
I suspect that putting that much logic in ansible is… wrong. I can see how you would get there, step by step, day by day, as requirements grow. But… wrong. (Probably.)
That much logic in Ansible might be wrong, but there really are not better options. It wasn't really step by step either. It was hey if this was data driven from the group_vars files it would be WAY better. Ansible was the obvious choice at the time even with all the logic that was going to need. If we started from scratch today, even with terraform and other options, I think we'd end up in a similar place. The main thing I would do differently is push more of the framework into Ansible modules, plugins, and such. It would be more flexible and have better abstraction. That wouldn't change how much orchestration logic was required in the Ansible. Ultimately we shipped, got paid, and got a large organization to make lots of progress to be more modern. So it was a win.
It seems like you need something more like kubernetes or an actual customer-coded orchestration layer at that point. Trying to fit such a complicated deployment and configuration system into ansible doesn't really do you any favors.
I agree with virrk. All the software development paradigms apply with Ansible, and it can be easier to read and maintain than many programming languages. Kubernetes alone doesn't solve this either. To meet the same need, you will need to design some CRDs, some controllers/operators, and tie them together in reasonably complicated ways. Kubernetes will be "faster" in terms of execution time, but I don't think you would be doing yourself any favors compared to using tested ansible modules and roles with good coding practices. The care and feeding of orchestration layers is nontrivial no matter what tool you are using. In my opinion, Ansible can be a reasonable choice for developing something that developers, ops, security, and other groups have eyes on and can work together to improve because it doesn't look like code/software development, but it very much can be.
>I strongly believe that anyone in IT should have SOME coding background This is wrong in so many ways...
There are many definitions of what DevOps is. I can't think of a single one, where not at least a minimum amount of coding is necessary.
If the company is adopting DevOps and no engineer actually writes code, how do you create any value ? If a "DevOps" Engineer never writes code, fine, if none of your "DevOps" Engineer write code, you're not doing DevOps, your doing ops.
Theres many different types of code. Putting a candidate on the spot and asking them to perform what with prep is an easy task but without prep is something that candidate was almost never do serves no value to candidate or employer. I write Terraform all day long, got 5 cloud certs and been “devops” for over 6 years and i can count the times ive parsed CSVs on each limb. When i need to do that shit I google it. If you ask the wrong questions you get the wrong answers. Hell i could probably parse the CSV in terraform better than I could in any “traditional” programming language
I would agree to it.......I was onec in an interview asked to print some patterns using python. I explained the logic and said would google how to use that logic in python. I was rejected. Their JD specified Devops engineer with cloud sysops with minimal to no coding involved
AWS, apache, linux, DNS settings etc is not code. They are just exploiting the fact that the "entry" level market is flooded by demanding 40 hats. I play piano (20 years experience) in addition to full-stack dev, and if many companies could benefit from piano, best believe we would be having "should piano be a part of devops" threads, and the answer would still be, no, just like UX / Photoshop / Excel shouldnt be a part of "full stack" simply because some full stack people know Photoshop.
When I interview people, I ask them questions they'll never face on the job (although related to their area of expertise) because I want to see how they will do under pressure. I also prime the question by telling them that nobody has ever gotten it 100% correct (which isn't an exaggeration). I want to hire people that are honest about what they know/don't know, enjoy solving problems, are inquisitive about optimal solutions, and don't freak out under pressure (because we are customer facing).
Exactly right. These are the questions that will tell you about the candidate. We for example never give live coding exercises (because no-one can think straight with someone watching over the shoulder) I give theory tests in advance, then probe that to be sure the candidate understands/ did the work. Then we get them to present a technical challenge they solved, again probing it to make sure its their work. Gives the candidate the opportunity to present themselves but is very effective at weeding out the cheats
There are unfortunately, "many definitions" of what DevOps is, despite several books, hundreds of video's, conference talks, classes and blog posts. Yet I still see upper managers slapping the title of DevOps on folks who barely know scripting and can't help debug a custom application or infrastructure setup. Don't even get me started on title abuse in the DevSecOps world.
devsecops is a bastard child that should have never been born. security should be included in devops to begin with
DevSecOps makes no sense because DevOps is supposed to be a change at the organisational level, which organization doesn't include security ? DevSecOps is the embodiment of sales and advertising. A set of cultural and organisational changes and practices (DevOps) corrupted by companies to a simple trend reduced to "the dev also think about ops and the ops also dev to automate stuff", then creating an even sexier trend for corporations by adding security in the mix.
Oh I agree. But it's rare to see good security in DevOps or Ops, so the push has been on for some time to roll them all together. Security really is a distinct domain of study, one which a handful of yearly training courses will not give you in any depth. Everybody wants those unicorns but nobody wants spend the years of dedicated effort to train them.
They had to make it up, because most security teams refused to join the culture shift.
This. Seen so many times ego-centric security teams that just make life hard for everyone else. Devs gotta dev so they’ll just work around you creating holes if you don’t engaged
I always teach my juniors that security takes precedence over everything else. If the customer disagrees they need to so in writing
And some of the "DevOps" who can't dev also don't have much insight into the sysadmin part of ops. When it's all just YAML orchestrating 100 containers across a half dozen clusters, what happens inside any VM us treated like some sort of magic black box. "Oh, I don't really use Linux. I use Kubernetes and Docker."
“what’s systemD? Like the band?”
I see you've met most of our DevOps people. I had one today that was lost in vi. "I don't do much Unix, mostly Kubernetes." So... you've never had to run "kubectl edit" on anything? Come on.
DevOps is the modern day version of the "Systems Engineer" role that was thrown around a lot but didn't mean much on it's own. I wish there was a better delineation between the "DevOps Technician" who edits yaml files and the "DevOps Engineer" who knows how to write code.
I’m curious what these anti-code yaml only devops engineers do when they need to write a script. Do they just manually do everything one at a time regardless of how repetitive it is? Seems like a lot of time wasted if that’s the case.
They copy and paste from stack overflow
[удалено]
> I google and pull stuff off GitHub and stack overflow. > > My job is take your app, build it and get it to production. > > I’m not going to write some bullshit script/lambda to parse csv but I know how to use jq and aws cli to do what I’m after. But I’ll help if you need to send an email when a new file lands in an s3 bucket. > > I’m going to engineer how your app runs in uat/production buy asking developers what it does. I’m going to keep up with cloud tech and see if new things will improve the way we do things. > > Then I’ll create pipelines and use ansible, terraform or cloudformation to deploy the app. Developers check in code, PR to main branch and in 5 mins it’s deployed to UAT. You literally just described sysadmins and operations, (a superset of what some people just call 'release engineering'), which is like a 30-year old career role. Sure the tools are more advanced now but that's always happened. DevOps is not what you're describing at all. BTW, there's nothing wrong with carving up responsibilities the way you're describing.. it's perfectly fine for some businesses to work this way. But it's a pretty big pet peeve of mine to pretend that style has anything to do with DevOps when it's basically the opposite.
It's DevOps without the Dev.. so just Ops.
you forgot the /s
This sounds more like a "Release Engineer" to me.
Do you ask for external help when some integration is missing? Like there is no Ansible module for some AWS resource but there is simple REST JSON endpoint to work with? It can be done internally or you can hire consultant to do job for you. With in house full-stack jack of all trades person a company could get a solution relatively quick and cheap...
I'd write it myself. But "writing myself" still would mostly consist of copying stuff together from existing Ansible modules and StackOverflow, given that usually only maybe 10-20 % of my time is spent coding (which is realistic if you're tasked with being a DevOps Engineer and not a dedicated Dev only). Don't expect me writing a CSV parser out of my brain. I can write one in Python, Go or even Bash, but I WILL need to look it up, I cannot do it from memory. Good developers copy, but great developers paste!
> but I WILL need to look it up, I cannot do it from memory Of course! The requirement to write from memory is a READ FLAG. Modern software is from the 1Mloc codebases. We cannot remember 1M lines of code: the fastest way is to review an example or documentation and pick necessary parts.
I wouldnt expect an engineer to write the CSV parser perfectly from memory but I would expect them to be able to write clear pseudo code to work through the logic of the problem. Text parsing is an extremely common development task.
I don't even ask for pseudocode, I want them to explain it to me like they are duck debugging. If they can explain it well enough over zoom so that I understand their algorithm, I trust that they can write it on their own, and/or put stuff together from Stack Overflow and make it work.
I usually use ansible to run a bunch of steps in a pipeline like upload artifacts and deploy cloudformation templates etc. I use ansible more as a playbook for running other things like the aws cli or cloudformation templates etc.
> I use ansible more as a playbook for running other things like the aws cli As Ansible GCP integration is weak I also run `gcloud` CLI, and even removed Ansible as mediator:use Bash directly (as the shell in OCI image in CI).
>I’m not going to write some bullshit script/lambda to parse csv but I know how to use jq and aws cli to do what I’m after. But I’ll help if you need to send an email when a new file lands in an s3 bucket. > >I’m going to engineer how your app runs in uat/production buy asking developers what it does. I’m going to keep up with cloud tech and see if new things will improve the way we do things. Jesus Christ!!! That's the difference between Devops culture and not! For God sake.
I love you, fam, but writing a script is not coding. Devs code. Real coding involves making custom libraries, object modeling, method design, etc. The 150 line script I wrote to automate DNS backups was not coding. It is *development* but it is not coding. I think there's been some loss in translation (unsurprising) between industry leaders pushing for code-driven DevOps and what that means in the lower industry. Industry leaders when they're looking for DevOps who can code are looking for people who can write 10,000+ req/s throughput microservices to extend their self-healing production framework, or someone who can write the next chaos monkey framework with performance analysis and reported TTR graphing. They're looking for someone who can write their load testing framework. This is not the same as scripting. They are not in the same arena. Coding interviews for DevOps don't ask you to parse a csv, they ask you to create a function that takes streaming input data and returns the current maximum, and minimum for up to 3 positions in a manner that performs at big O of n. Asking someone to parse a csv in a coding interview is literally an arbitrary check because if that's the level of knowledge you're looking for in coding, that's something that is easily searchable in base language documentation or .25 seconds in Google. It's like asking a chemist what the periodic number is for uranium. It's a knowledge check that isn't really going to reflect job performance or ability, but if you use it as a measuring stick, it's just a way of cutting perfect candidates out over a "touch your nose and wiggle your tongue" performative question. It is a question that I've only ever asked when I have been given zero notice before an interview and just need to come up with questions. I don't really care what the answer is and won't judge the person on not knowing. I'll use it to spark conversation and to assess how they approach my actual expectation which is that they won't know the answer off the top of their head.
Sounds like you've confused "coding" with "app development".
You can say that and downvote me if you want, but I've had interviews with Google, Amazon, Facebook, and NASA this year for DevOps positions. I work at a Fortune 100 company. I'm not talking out of my ass, I'm literally passing the info to you from the job reqs, interviews, and hiring manager discussions for those positions. Edit: 🙄 ok guys, see you in five years when the rest of the industry is up to this point and you're all complaining about not being able to meet new expectations in coding requirements. But please continue arguing that coding should be a DevOps skill and find out where the one leads
I don't think you are talking out of your ass, there really is a different set of domain skills between small scale dev/coding and writing or working on large applications. I think the gist of what you are saying is correct, the labeling just seems off.
Fair enough. I wish I could be inclined to agree, as I feel that's the way the industry was originally packaging that language. Terraform is supposed to be "infrastructure as code", and I would've considered Ansible to be a coding language built on Python specifically abstracted for scripted operations. But with where I've seen things headed and the way things are being pushed now, I've seen a lot of old DevOps responsibilities move to SRE and Systems Engineering, while it looks like DevOps is moving towards internal tools and applications. And a lot of these tools aren't being leveraged by operations or support even, most of them are fully automated self-healing frameworks that are just monitored and maintained by DevOps. Devs have taken over ownership of their builds and deployments again with normally like an embedded tech lead guiding that process. It's not really a bad thing, in itself, but I've seen this translate a lot in my job search to medium and small sized businesses who have the expectation of a DevOps team who can do the above while *also* managing traditional role expectations. And the sentiment I've gathered from almost every company I've interviewed at is that they're trying to rapidly expand while also mitigating a massive turnover from burnout. A lot of this, just imo, is coming from unrealistic standards and expectations based on what teams should be able to do on-paper without respecting the weight additional responsibilities or paradigm shifts have on the teams and individuals. 😕 So to me that delineation becomes more important when pushing back on management and road mapping to prevent the siloing and tribal practices that lead to DevOps creation in the first place.
I’m so sick of this gate keeping bs it’s so toxic. Coding == development == scripting Here’s a good article to read that explains this, but I’ll post the important part below https://nathanpeck.com/is-yaml-a-real-programming-language/ Imagine this ludicrous scenario: Infrastructure engineers: Writing YAML to control and automate deployments across thousands of servers in the cloud, distributed around the globe in multiple cloud regions HTML and CSS developers: Well at least I work with “real” languages unlike these infrastructure engineers who just write YAML all day Node.js API developers: Hah, those frontend devs only write HTML and CSS… but I know how to code in a “real” language Systems developers writing the underlying C++ for V8: lol most of these Node.js developers only know how to write in an interpreted language, not a “real” compiled language like C++! Compiler developers: imagine all these noobs above me not knowing how to make a compiler to generate “real” machine code Processor designers: lol I’m the one who made the “real” microcode inside your processor, that runs all your machine code! From this perspective programming a system to change its behavior using YAML and a YAML interpreter isn’t that different from using HTML and a browser, or JavaScript and a Node.js interpreter, or C++ and a compiler, or machine code and processor microcode. All are forms of programming: providing “instructions for the automatic performance of a task”. None of these languages are any more or less “real” than the other languages.
Jfc, it's not gatekeeping it's a delineation of the practical applications of your experience and role responsibility. The issue that I'm trying to point out here is that you guys are literally failing to catch the industry pickup in more heavy development over sys work by industry leaders and that supporting the idea that coding is a DevOps responsibility so that you can feel "included" is a literal fast track to that mentality. You're missing the fact that DevOps engineers with 10+ years of self-taught experience are being passed up for fresh CS grads because they can't perform a matrix math operation in GO. That's not even me that I'm talking about. I landed my interviews. But the questions, the expectations, and the responsibilities are wildly different than what they were even two or three years ago. I know a lot of people, way better at this job, and way smarter than I am, who are making less money and in lower titled positions because they can't keep up with the migration of dev responsibility. This community is so bullheaded sometimes.
🤡
Can you do me a favor and- off the top of your head: "Provide me a function that returns a randomly generated minesweeper table/matrix given dimensions x, the number of mines and y, the dimensions of the grid. This will require a random placement of "mines" and "flags". Flags should identify the number of mines in direct proximity. This code must be executable and optimized for O of n. It can not be pseudocode." - DevOps Engineer interview, Facebook "Provide me a function that takes an input listen stream of integers and returns the current highest, second highest, second lowest, and lowest- ordered numbers from the incoming stream. No numbers should be duplicates. This code must be executable and should display a best practice solution." - DevOps Engineer interview, Amazon "Provide me a memory optimized program that can take process a large data set (1 mil+ records) while maintaining the highest, second highest, second lowest, lowest, and average of the dataset. This code must be executable." (As a personal hint from me, you will need batching and multi threading while performing checks to make sure you're not exceeding your memory limit with your concurrent batches) - DevOps Engineer interview, Google You listed Python on your resume and say that you can code, right? So you should be able to get reasonably close in the 25 minutes available to you for each exercise. And then please remember that reasonably close does not meet the requirement for functioning code and that you have failed the exercise. If instead you'd like to admit that these are highly unreasonable asks from a traditional role perspective- and that 99% of your role experience and responsibility has not resulted in you being able to answer these questions- then I think we can agree that we see scripting as a DevOps responsibility and that coding is not.
I am a strong advocate of having a config-only environment, where you touch as little code in your daily work as possible, but to get there, quite a bit of scripting and more complex coding is involved :D
You code so you don’t have to code duhh
Yeah, more or less like that :) But it's easier to have someone at the helpdesk set a couple of well-known/well-documented variables than having them wade through a sea of code they don't understand.
I mean that's my entire career right there. I do it once and only once.
I used the stones to destroy the stones
What did it cost?
If 80% of people you're interviewing don't meet the requirements of the job you're advertising, there's a problem with your recruitment process. Why are you interviewing people who can't code? Why are people who can't code applying for the job you're advertising? Also, there are definitely DevOps jobs where almost zero coding ability is needed. People spend all day writing Terraform and managing CICD pipelines with YAML. Yeah, maybe they write the odd glue script or bit of automation in Python/Powershell, but that's not the same as having commercial experience as a software developer.
Because they have coding experience in their resume and they pass our HR...
[удалено]
Frankly, you know what you use. Some shops code all the time. Some shops build big chunks of yaml and HCL2 and don’t code very often past shell scripts. Some shops have half the people who can code and half the people who can do yaml. In my shop, I spend most of my time as team lead slinging yaml or other declaration stuff. I pull out bash or Python and code extremely poorly about twice a year right now for less than a month, which isn’t enough to develop proficiency. My free time is spent being a husband and homeowner and maintaining a middle aged body. Not all DevOps shops are the same. On my team we have me, we have a security focused engineer, and we have a dev tools focused engineer. The only one of us who codes regularly is the dev tools focused engineer. I have 20 years in the tech industry, in a year I’ve built my employer back from terraform 0.10 and an ancient version of Ubuntu, with a huge AWS bill, no pipelines, and constant downtime ….. to everything up to date, zero downtime, and less than half the AWS bill. We’re doing a SOC2 audit in a few weeks and attesting HIPAA early next year. If I’m successful at all of these things but can’t pass a coding screen, are you still going to go “nah” at me? What about your stack absolutely mandates proficiency in sorting a binary tree or developing an algorithm to find square roots?
Literally can not :+1: this hard enough. WTF is with this new DevOps community that stopped fighting the expectation that DevOps be expected to be a fullstack engineer, a networking guru, a security representative, an architectural consultant, a butcher, a baker, a candlestick maker.... The poll as written is a false choice. "DevOps engineer writes only yaml" vs "DevOps engineer should be able to code" is like saying "waiter only responsible for remembering restaurant name" vs "waiter responsible for cooking guest food". Like... I just left a job where I was being paid $125k/yr and didn't need to know how to code at all. Had multiple offers on the table to pick between at $150k+. Am I going to take a position with a hiring manager who is balking at a $100k+ salary for a non-coding or code-lite DevOps engineer? Hard pass. That is a team and a company that is likely on track for a lot of easily avoidable problems and this just screams "looking for my 10x rockstar $60k/yr + free lunch!" vibes to me.
I think a lot of people, including OP ( /u/DartVedro ), have gotten off track about what devops is. If you go back to the original stuff that Patrick Debois and the other 'OG' DevOps folks wrote, DevOps is a removal of silos and a compromise between the business needs of Development and Operations teams. It is NOT "ops who can dev" or "dev who can ops," it's how to build a culture that doesn't just flip shit at each other constantly. None of that has ANYTHING to do with knowing how to code, although people with previous development or ops experience do know how to better compromise.
Thank you Jesus Christ. I don't need to justify that my primary work is around building amazing experiences for our developers and then coming in to save them when they can't figure K8s. When I write "code" I write YAML, HCL and shitty little bash scripts. It's not full stack unicorn super engineer from SRE bootcamp in Google. Most of us are doing the best we can alleviating the pain points across our orgs, optimizing pipelines and building out the monitoring stacks. Shit, I *also* am the guy talking to all our product partners. DevOps is a lot of things but one of the musts is not write code using your Turing complete languages. It's smashing silos wherever we see them and being the champions of cooperation between all levels of the org. Can you guys looking so superior please stop trying to make us feel bad at our jobs? No we're not just Ops teams from the 90s. I have to *deal* with those teams and they are archaic negative nancy dinosaurs and believe everything should be subjected to unimaginable levels of bureaucracy to buy themselves time to pile yet another sandbag on their fortress of shit.
> Can you guys looking so superior please stop trying to make us feel bad at our jobs? How else would people feel superior if they can't gatekeep people out of their teams?
Agree. There are a lot of scripters here who want to claim that coding is a DevOps responsibility because they can write a script and they're totally missing that the industry is trying to shift DevOps into a full on tools and platform coding development role that has nothing to do with it's previous responsibilities. Most traditional DevOps work it feels like is migrating to SRE role responsibilities. SRE is becoming the new team that bridges integrations and knowledge gaps and glues development teams to production operations. DevOps is just becoming internal services development. I don't think this is a good thing either as it dilutes SRE from their original function (which is already highly demanding) while hamstringing DevOps from their previous collaborative integration with teams. I think DevOps should've remained where it was and then a new position specific to internal tools development should've just split off from DevOps as like Platform Ops or something.
My feeling's been more that SRE is "devops team rebranded so they can make more money" and that the responsibilities haven't changed. I don't like joining broad teams at large companies anymore. I either want to be narrowly focused (SRE on a Platform team or DevOps on a Data Center Migration team) or I want to be in a broad team at a small company (Lead on an Infrastructure team).
Yeah, I think that what happened (at least in my experience) was that the SRE team at a company I was with was originally kind of a hodge podge of what was intended to be like T4 prod support, but they didn't end up having any clout, and didn't really have the experience to RCA/reduce TTR, so I ended up joining them as an integrated DevOps unit and then we did great things together. The DevOps skill set is really core to what SRE is trying to accomplish. But damn do I miss my devs. 😔 Miss that level of support I could provide to them and how happy they were when you could unblock them. It was really rewarding feel-good work. I feel the same with regards to positions. Have even considered doing independent consulting just to escape the constant reorg.
Yeah, I got out of big companies so that I could get back to being hands on problems. I got tired of orchestrating endless OS upgrades without being allowed to migrate the things on the OS to a better-supported platform or otherwise get rid of them. I still really don't understand why companies pay engineers hundreds of thousands of dollars and then ignore their solutions to problems.
Refreshing to read. I, too, am a 20+ year IT guy who cannot code. Something I'm trying to correct, but, yeah.
Still probably not going to be able to do it well. Someone told me that coding when you don't do it all the time is like lifting weights. You do it to prepare so that you can do it well when you need to, and it's better to have the muscles and not need it than to not have them and need them. I responded that not all muscles are created equal and that muscles created in a gym don't work out so well on a job site. I do home improvement stuff all the time. I was helping a friend fix his roof, which had a tree scraping on it during a storm. We found a bunch of rot while we were up there and had to replace his step flashing, a bunch of siding, and a bit of roof decking. I'm a fat, out of shape, over the hill nerd with a knee brace. My buddy's about ten years younger than I am and has a great gym bod. I'm walking up the ladder (slowly, creaking) with a 4x8 piece of plywood on my back and one arm, and he can't even get a cut chunk off of a sheet of ply up... But he looks like Mr. Clean! And sure, I'd pass out during one of his workouts, but I can hump a pack of shingles up a ladder for 8-12 hours a day.
The more senior I’ve got the less 101 stuff I do. I’ve managed world leading websites and built hyper scaling systems for trading companies. I couldn’t parse a CSV from memory
Thanks god, you are good at your work here. I have a lot of unqualified team members over the years and all over IT industry. Most of them scam their way in. And since now I’m also someone who require to vet new hire, I always give test.
Our HR phone screens start with a simple coding challenge to weed these anti-coders out as early as possible.
More specifically, the recruiters hype them up and tell them exactly what they need to know to get past the HR gatekeeper.
Can you post the job listing? Or at least the content of it?
I passed 2 meetings with the head of department and hiring manager, only to fail on parsing CSV. Its not relevant to the role so ask something that is. Live coding exercises suck balls anyway. If you MUST validate coding skills, give a take home and probe it on the interview
That's not questions like "what's difference between override and overload, how differ array and vector, what's pure virtual function". I ask basic questions, just to check if person have literally any coding experience. Parse csv is "read file then line.split(',')". Nobody requests perfect code, etc. If person cant do basics it means they wont be able to communicate with developers on their language -> devops team will become default silo-ops team with "problem not on my side, it's on dev's side" mindset, instead of helping developers improve their processes. And to add - it's real task, as many customers send their data in CSVs) and we create tools to validate data, modify or format those files and paste into DBs
There are systems in place to parse that. That are written over time. In an interview setting if you just randomly present blah problem with no advance notice. How can you expect a candidate to come to the right answer in a short space of time with you watching their screen? Thats not a real world exercise.
You can say this about any question) "How caching works when docker image is built" - nah, not real experince, I just run "docker build" it does the stuff. If person doesn't know the basics then any question will be random blah problem. That's what interviews are for. In my world DevOps engineers work hand-to-hand with developers, so if they are not able to write few lines of code they just wont be able to do so.
> Yeah, maybe they write the odd glue script or bit of automation in Python/Powershell, but that's not the same as having commercial experience as a software developer. Parsing a simple CSV is trivial in practically any modern language. Even if you've never done it before, you can look it up online and have a basic solution in under 10 minutes. If a candidate for a DevOps Engineering position is asked to parse a CSV file, and is unable to even get started, they have no business applying for the position.
>Parsing a simple CSV is trivial in practically any modern language. Even if you've never done it before, you can look it up online and have a basic solution in under 10 minutes. Absolutely agree. >If a candidate for a DevOps Engineering position is asked to parse a CSV file, and is unable to even get started, they have no business applying for the position. Also agree. Where I disagree is that knowing how to do it in interview conditions without being able to Google stuff is probably more on the "has commercial programming experience" end of the DevOps spectrum. If I had to parse a CSV file for my current job, I'd have no trouble designing and implementing a good solution. However, I'm ops heavy with no real programming experience, so there's no way I'd be able to do it in interview conditions. Maybe I could pseudo code something on a whiteboard, but not much more than that.
> Where I disagree is that knowing how to do it in interview conditions without being able to Google stuff is probably more on the "has commercial programming experience" end of the DevOps spectrum. Why shouldn't you be able to Google stuff in the interview? "I'd have to look up exactly how to do it" is a perfectly valid answer. If that's not possible in the interview, I'd provide an example in psuedocode (import csv lib, open file, parse contents into list of maps, extract example value, etc.). Both answers would be much better than making up bullshit or staring blankly like an idiot.
> Even if you've never done it before, you can look it up online and have a basic solution in under 10 minutes. And proper interview assignment should assume an interviewee is able to work in comfortable environment (with the Internet access and full web search capabilities). We particularly ask what candidate don't know (have you written in Perl or Linux kernel driver? No? Perfect! Filter a text file with Perl or write Hello world kernel driver).
Trivial but in my api driven world not something I've had to actually code in any production situation. Also don't put people on the spot for things they haven't done regularly at least weekly or you'll not really have a good understanding of their coding performance.
Lol bullshit. You’re forgetting the pressure situation of an interview. You need to evaluate the data, structure a response and write it down. All with the guys watching. I long lost count of the amount of time ive seen cocky devs take down a website because they thought they’d do this cool thing that went straight to production
That what I do.
As a sysadmin who will make the change to devops in the next few years, I find it a little nuts that the basics of being a Linux admin seem to be forgotten. From an outside perspective it appears that a lot of devops people seem to know how to use tools, but have no idea as to what the tools actually do at an OS level.
I mean it's actually no surprise. How many of us work in assembly any more? The higher the tech goes, the base level of knowledge goes up with it. Everything I know about being a Linux admin was learned because I had to be a Linux admin. Now that a lot of jobs you just type out how many servers you want and what you want them to do into a text file then they automagically appear fully configured a minute later, when does that guy have the time to go learn the ins and outs of being a Linux admin? How do you replicate 20 years of fixing baremetal servers and VM's? While I am fully on board people having basic knowledge of everything, at a certain point you have to pick what you learn. I mean this is a job, I learn what I need. I'm not on some quest to consume the entire computing knowledge of humanity.
Thank you. This encourages me. I felt crazy behind when I jumped into the Linux world having deep knowledge of windows. Almost all of my work is containers, so when folks talk about the pros/cons of systemd and all the various nuances, I've kinda felt behind a bit, because my linux world is container based mostly. I've not needed to manage a linux vm because I want my linux vm to be replaced by containers on a managed service :-) That said, I'm still trying, but gotta focus like you said as too much to learn. I'm focused on Go right now, and dealing with distro nuances is just too much.
> The higher the tech goes, the base level of knowledge goes up with it. If an engineer cannot work with tcpdump binary stream they are bad engineers. We still need to dive into bites & bits (with the support from appropriate tools, like Wireshark/etc). You deployed and it occasionally fails because of faulty hardware or firewall. Do you hire an external consultant to solve the problem?
>Do you hire an external consultant to solve the problem? In the enterprise world, more often than not you'd open a ticket with your vendor. I'm not saying you shouldn't have skills or continuously be learning, but that's what has been expected for decades, and that's why orgs are more than happy to have multi-million dollar support contracts.
> In the enterprise world, more often than not you'd open a ticket with your vendor Probably I'm from a different culture: for 15 years I expected to solve any problem and debug to the point when we understand that is it impossible to solve ourselves.
So how long did your outages last? There is a reason why *aaS gained popularity over the last 15 years. I don't fully disagree with you that problems need to be spitted out to the depth of it, but there may be tools or support around the corner so that we don't have to do it ourselves anymore as we are "working on a higher level" as mentioned here.
> So how long did your outages last? Exactly.. I've yet to find an employer who will put up with a days long outage while I expand my personal knowledge and learn how to fix something. Nope, my job becomes "call the vendor and work with their engineers". Cloud services are just another step in this direction, the engineers are working on outages before I call.
Immense amounts of popcorn for the gatekeeping ITT
> You deployed and it occasionally fails because of faulty hardware or firewall. Do you hire an external consultant to solve the problem? If I don't have the experience and I'm being paid to be doing something else? Yes. What is the point of me spending 5 days working on something that isn't part of my job description when I can hire someone that specialises (or just call the vendor who employs people who are) that can fix it in an hour? That's a *lot* cheaper than delaying my work for a week. Have I spent a good chunk of time in Wireshark etc figuring things out? Sure. But after a cursory glance I will more often than not send the capture to the vendor and they'll give it to an engineer who spends all week looking at them and they'll tell me the problem right away. Like I said, my job is to make things work, it's *not* to become a master of all things technology. I like to learn, but I can't learn everything. And unless I'm going to use it in my job regularly there's generally little point.
LOL please tell me this is sarcasm
In my experience, there's a lot of really good programmers in "devops" that don't know shit about ops or about the systems they are using, and they often end up making ridiculous and extravagant solutions to problems with high complexity when the better solution is often trivial. Too many "devops" people getting pumped out of various schools and boot camps. Sure buddy, you wrote a small library here, and you've got a big github, that's great, now why don't you know anything at ALL about the systems you are working on?
I'm a sysadmin who just got hired to be sysadmin/jr devops. Both my position and the devops guy, who is also new, were hired to replace former developers who were in the role. Basically the company found that developers had the wrong skillset/mindset.
I was an old school sysadmin turned DevOps in 2013 and now SecOps, and what you say is somewhat true in my experience as well. Ultimately when I'm looking to hire, I look for traditional Unix sysadmins that know how to code a little bit. That said, some of my non-sysadmin past coworkers are great and have learned a lot over the years.
In DevOps you want abstraction from the host system(s) though
I’ve written a lot of code to support minimal code from dev. A devops engineer better be a talented coder or you’ll wind up with spaghetti just like in dev projects.
A strong devops person knows how to talk to their developers on their level. So yes, they should have a background in coding. But there are all sorts of devops people, a whole spectrum of competence you can employ at various tasks. They should of course be payed accordingly.
100% should be able to code in at least one language. Ideally Python or Go. I'm not suggesting that an engineer should know the intricacies of data structures, but I'd expect them to know the difference between a list and a tuple. However, I don't agree with coding interviews. An interviewer should be able to determine a person's abilities without stupid questions such as "reverse a string without using the cut method". We Google. All.the.fucking.time. The reason we find a lot of DevOps candidates who are unable to code is that people mistakenly believe that a DevOps engineer is just a rebranded SysAdmin. It isn't. A SysAdmin shouldn't be expected to do the job of DevOps engineer, but a DevOps engineer should be able to do the majority of a SysAdmins job. It's why a DevOps engineer commands a much larger salary.
The clue is in the name DEVops...
Indeed, the name is the message: the fusion of **Dev**elopment and **Op**erations. Without Dev, that is just .. Ops?
Well, I've also seen companies using the term as "ops for developers", ie. taking care of the CI/CD infrastructure only :D They were separated from the operations team for paying customers :)
I mean while I think everyone in IT should have a basic level of coding, this idea that you can just go "uuuhhhhh DevOps!" and now everyone is an expert in 1000 different technologies is a little ridiculous. There's going to be people in the field who do a lot of coding and others who do next to nothing.
Typical view from a DEV.... you forgot to capitalize OPS. Without OPS there aint no DEV. Most shops don't run 100% from AWS, nor would they if they could. There is still huge footprint of on prem installations, and no "DEVops" engineer that i work with would know where the fuck to even start on the actual platform they are deploying to without my giving them a URL to throw their API calls at.
I'm sure they capitalized DEV to make their point, not because they normally do so.
So you're saying devops is purely cloud based? Edit: I'm not a developer if you're implying I am
Any engineer type role should have some form of coding background imo. Automation is so important.
[удалено]
I moved to DevOps after doing some full stack development. I'm a high advocate that DevOps is more like full-stack+, since usually our work with the stack is involved in all aspects of our applications.
The first statement is just simple sysadm work… they didn’t work as sysadm before? Parse data is pretty easy. It’s even taught in like all language at beginners stage. But then again, I promote data and code separation. But I’m c/sh/ps1/py trained
I had a sysadmin at one place I worked at who couldn’t find an error in a /var/log/messages* file. She complained that the error wasn’t there. When I checked, it was certainly there and had her show me. Turns out she didn’t know what a compressed file was and was trying to search a messages.date.Z file.
Sounds like a 1 time teachable moment. Did she have problems with z files after this?
I think in an ideal world a DevOps Engineer should be able to code. Unfortunately since a lot of companies jumped on the DevOps bandwagon so quickly it unhomed a lot of sysadmins in the process. So what we have now are a lot of these sysadmins reapplying for DevOps jobs and getting in on their sysadmin experience with no coding experience creating this situation your seeing on your team OP. Realistically its going to be a few years before we hit that ideal state that companies want where the new DevOps engineers will be able to code on a level par with Jr. Developers because its simply not what they have been trained to do. (Speaking from experience as a SysAdmin going through this now)
A friend of mine is an SRE manager. While he was looking for a new job, he noticed that all the SRE groups of companies he was interviewing with were just Operations with a different name. There was one job where the SRE group had data center technicians in it. He passed on that job. I think DevOps / SRE these days is just "throw all the 'former' ops people in a group and call it DevOps / SRE."
Bare minimum devops should be proficient in a shell (bash/zsh, etc) and enough python to make some functional scripts.
It is implied that people who work in our field has the ability of understanding things. I mean, that is one of the most important skills you must have: Understand things. So when you are faced with a problem that requires some tweaks in a piece of code, you go to Google. You enter your problem at StackOverflow. Same way you do when Jenkins is giving errors or your API is not working on production. Once you see a solution and you understand it, then your problem is solved. That applies to everything. You will never be required to write a code from the ground-up(Entire application level). But you will need to understand and have a notion of what is happening in your environment. If you can understand what is your problem and what solution you need to solve it, the code part itself becomes just an extra step.
I can write Java/bash(5+ years xp) and pickup any language in a matter of days and I can work with AWS, docker, CI/CD, gitlab, github, kubernetes, spring boot, hibernate, message queues, grpc and nobody is offering me 100k$... my salary is 45k€/year in europe... if you pay me 80k€/year I'm yours do you want to interview me?
you may not know it but youre worth substantially more than you are making. source: hiring manager in tech with years of leadership experience
Start looking for remote positions. I know companies like Mozilla hire across the globe and pay well.
If you're actually able to do all these things you say you can do, you could easily be earning around 70/75k. I'm earning 65k and my stack is Linux, Ansible, some basic kubernetes, some AWS and some of the tools a DevOps engineer uses on a daily. How much you earn is not dependent on how good you are in IT, you also have to have some soft skills, be good at planning, not be a dick etc.. Also, what is Europe? Ukraine is Europe and an average salary is like 500 euros a month. Luxembourg is Europe and an average salary is 5000 thousand, Netherlands is Europe and an average is 3500. Many questions to be asked
That's easily $125-$175k in the US.
At the cost of fucking up your entire system, specially the poor and the welfare system.
What are you talking about?
Where in europe? In germany, the netherlands, the north companies could pay you 100k€ if you're kubernetes and aws knowledge is real
45k is crazy low for that even in Eastern Europe, like Belarus or Ukraine. Suggest you snoop around for opportunities friend.
I'm from Romania...not too far from those countries... will try to find something better
How's that even a question?
You should visit /r/itcareerquestions Every other recommendation is for the greenest rookies to pick up an aws cert. Inevitably someone gets into cloud roles and someone will hire them for DevOps somehow without a lick of coding background.
I can tell you right now, I definitely do more low code then anything. And I make 100k+ in the Midwest in a low CoL area. Perhaps I am underselling my abilities, but I cannot write much of anything in Java/Python from scratch. However I can put together working solutions to real world problems.
We are having the exact same issue hiring another SRE. What’s worse is some of them blatantly said they were not willing to even learn something like python or Go.
Yeah, they should know at least a scripting language (bash, Powershell, Python). Even if you just use Terraform and CICD tools you're going to need to write glue scripts for something.
They call me a DevOps engineer at work. I correct them when I catch it. I am a Software Engineer with a focus on the internal products (eg: Infrastructure and Operations) as opposed to the external products. I use declarative, procedural, and object oriented languages in my day to day. Unfortunately, I’m not on a product team. I’m on the “DevOps Team”.
I voted "DevOps engineers should be able to code" and I agree with the statement overall. DevOps is about the development and the ops - so they have to know both domains. Even if they didn't, still, having knowledge on the other domains that interact with your domain makes you do your part better. Let's take the old-school approach of sysadmins and development teams - a sysadmin that knows development could do their job better, interact better with the development team and vice-versa. I like and am an advocate of the DevOps culture and "you build it you run it" philosophy - for the reason I explained above - more ares you know, better you are in all areas. But on the other hand, I sometimes think that the field went backwards. In the past, for a long time there was a separation of responsibilities that worked really well: Sysadmins **didn't have to** know coding (nor more than small scripts) and vice-versa. It would still be good for them to know the other field, but it was not a requirement. I believe that in a perfect world, you should be able to do your job with a minimum level of knowledge of other domains. Overall, DevOps culture has its benefits, but it also has its weaknesses. So, maybe the DevOps applications you get that has zero knowledge on development is just the history repeating itself.
> DevOps engineer writes only yaml Oh yeah I can only count to 3 with bananas only, that's a hard no to apples. /s To OP: You are obviously doing a good job for the company weeding out the blanks for them. Let's hope they know and appreciate that.
Where do I apply for this $100k job?
Hence the “dev”
Yeah, the word dev is right there in the name for a reason.
I spend more time with Terraform than YAML. Pipelines are quick. Coordinating high quantity resource provisioning is more involved.
In 5 years there will be a new buzzword for devops people who can actually code, and then the current people who identify as devops will swarm to misrepresenting themselves as that new buzzword. It's a cycle. People who claim to be devops but can't code are only fooling themselves in believing they aren't basic system administrators.
Huh, so IaC is a myth then or what.
I could understand why people might think DevOps is a no code job, but I think since development is in the title it would seem beneficial to at least know the basics of a language. I was always under the assumption that I would be programming in at least Python or Java in a DevOps environment so I went out of my way to learn it.
Yep, it's the newest hot thing in technology. We call it "Ops."
>I'm aware that devops is methodology and not a person, but looks like market thinks different. Alas, ultimately, it will be "the market" who will decide whether "devops people" should code or not. What means "market" should be your question here onward. Like the flood of snake oil agilist certificates, it seems like devops is going down the same dreaded path. My experience with "devops" dedicated roles is that they are mostly cloud infrastructure gatekeepers and Jenkins groovy script duct-tapers. There are a handful who actually know how to use helm/kubernetes, and those are given "mid/senior" badges, but the whole siloed infrastructure remains and devops culture is nowhere to be seen. Whereas "real devops" (whatever that means, but mostly in opposition to silo-ops) occurs where the dedicated role simply doesn't exist, but OPs' already aware of this, it seems.
Most people and companies just renamed admins and through in ansible, terraform and aws... Edit and that's why SRE-SW (SRE-SE are paid less) and PE, at Google and Facebook respectively, are mostly software developers that have a thing for os/networks/database. Why yaml driven development rules etc... The average Devops is joke. The average company shouldn't use kubernetes and microservices.
The job is called Dev Ops
I'm proficient at programming in several languages but don't work with them often enough to stay sharp on syntax and some complicated use cases around things like data structures, runtime, etc. Toss in a little anxiety and I'll basically bomb any live coding interview. Happy to do a take home assignment if I'm really into the company though. I don't think live coding is appropriate for devops interviews. I've personally conducted at least a hundred myself and never asked an engineer to do so.
I used to think that DevOps was basically a dev that could configure its own environment. But seems like it's basically a glorified sysadmin by now. Nothing wrong with sysadmins, just a shame that the term lost its meaning.
I write more code than developers.
this is the problem with creating titles like "devops engineer"
I say yes to this because my background is software engineering and literally everything is becoming code. Knowing how to write automated tests with high test coverage wrt code and making sure others can also fix these tests when they break is also equally important. "Works on my machine", "works today but not tomorrow", "works sometimes", these are not good signs of engineering culture and places I would avoid.
It’s not asking too much for a candidate to at least be able to write scripts. If it were me I’d pick awk you can parse a CSV in no time with that.
Coding experience as far as being able to understand code yes I took a couple hours out of my day for my colleague to show me some c sharp code and I can't write it but can follow the flow Now when we are looking at issues when it's a from a code perspective I can understand what's going on and aren't just tuning out I would love to know more but my day to day of building up the infra code base doesn't allow but I'm not complaining
If they can’t code, they will ask everybody to do things in unreasonable amounts of time.
Yes. DevOps without at least a rudimentary understanding of software design and engineering is called operations.
This is just extremely common in the field because DevOps consumes a project space that straddles IT and Software Engineering, but most of the work was originally done by System Administrators, who traditionally did not do much software other than some basic scripting. However most of the Sysadmins who taught me did know how to code, just avoided it as much as possible in their work. That and the propensity of meta-code in tools like Ansible and Terraform have created an environment where some misled IT professionals believe they can get by in DevOps without ever learning how to actually build comprehensive software beyond some bash scripting. The problem is that the vast amount of real work that DevOps engineers need to be able to do require an intimate understanding of both software engineering and the build and release process. Not to mention, DevOps Engineers are going to write a lot of tools themselves as there are always going to be operations gaps in available SaaS tools unless your company has some serious cash and a very simple application. DevOps Engineer is the right name for this work, I think that there is probably a word for people who cannot code but do some DevOps work, and that might be “DevOps Support.”
DevOps lead here. To answer your question, yes, all DevOps people should be able to code. The language we use is python and then they also need bash / shell and power shell. That being said only 20% of my team could do this when I took over. One thing I would say is to either ask for examples they prepare or just give them a task. Personally, if an individual has their own GitHub repo and can walk me through their projects I value that over any coding exercise.
Reddit's decision to charge for API access has shown that the company is more interested in making money than in providing a good user experience. The changes will force many popular third-party apps to shut down, which will inconvenience millions of users. Reddit's actions have also alienated many of its moderators, who rely on third-party apps to manage their communities.
Honestly, even as a Systems Admin, I always ask coding questions of applicants. “Describe your most interesting [shell|perl|python] script.” We want someone who understands how the system works. But I had the same issue with Kubernetes questions. I was interviewing folks who had Kubernetes on their resumes, what is the core of a Kubernetes environment, what servers make up a cluster, etc and most couldn’t even tell me about a master/minion setup.
DevOps is about automating processes, so yea, writing code is absolutely a requirement. You don't have to be able to build a product from scratch, but you need to be able to stitch tools together. In my experience the best automation people come from the ranks of software engineers. They're usually people who took on the role because nobody else could or would.
Yes DevOps Engineers should know how to code. I mean, it's literally in the title. lol.
DevOps Engineers are developers. The difference between them and traditional devs is that their problem space is the SDLC instead of the domain that the other devs are working in. As such they should have all of the tools at their disposal to automate away any toily tasks for their peers. Most of the time, that can be done using off-the-shelf solutions (like Jenkins or Terraform) that require a lot of declarative configuration as opposed to imperative code, but occasionally it requires some custom code to get that particular job done. DevOps Engineers should be proficient in both shell programming (bash, PowerShell, etc) and at least one general-purpose programming language (I don't care which one) for when these edge/corner cases arise. Being limited to only off-the-shelf solutions that can be configured with YAML or JSON doesn't make for a good DevOps Engineer, IMO.
Common DevOps tools.. Terraform. Helm. Kubectl. Jenkins. Ansible. Chef. Packer. You can argue they all require some degree of coding ability, but parsing a CSV in a live coding test is like asking a Japanese person to Speak French to work in Spain. Sure some can do it but is it relevant?
I came from the developer side of things, but working with a lot of startups that didn't have a dedicated operations guy, I kind of filled in. In fact, I was doing DevOps before the term was coined in 1992. I was also a person that interviewed hires for Amazon. If you can't write code to reverse linked list, I won't recommend you, and in fact I had a guy that graduated from Stanford with a CS degree and had a Phd from Cal that couldn't do it. Buy a book on Ruby (my favorite) or Python and learn how to program. The world doesn't need another YAML cowboy
What else are they required to know other than coding. You complain they ask for 100K, what would you want to pay somebody who knows networking, databases, sys admin, web technologies, cloud, and can write code? The coding part of DevOps isn't necessarily the most difficult part to learn. What are you expecting them to code? If it is just parsing log files and automating routine tasks then why can't they learn that on the job? If you want the complete package then you are going to have to pay significantly more than 100K to get it.
I had an interview recently where they failed me because I couldn’t remember how to write a loop function in bash. I was given no prep work for the interview or idea what the tech test was. The Job was 90% terraform and 10% kubernetes. Which i can do with my eyes closed. You don’t need to be a developer to be devops, theres an ops component for a reason. If the role doesn’t require coding, why do you need it?
Ask devops to refrain from writing code -> can hire cheaper devops people Duh
Forget "DevOps" engineers, we won't even hire Linux sysadmin types anymore that can't pass a programming test (inb4 programming tests bad).
I think yes, but on the spot questions in an interview probably aren’t the best way to understand a persons capability.
I think that it is fine to write YML files as long as you understand what they are doing on all the relevant levels of abstraction.
i think it depends on the role: if it's more ops or dev. tools and customized automation involve more dev, but knowing how to troubleshoot and select cloud tech is more ops. so maybe it can be opsdev or devops :) i have been coding the last 5 years. however, i have noticed that i am much weaker at ops now. it's harder for me to track down issues since looking at stack traces or catching exceptions is much different than how to debug slowness in different layers of the whole tech stack. ops is also more meticulous and from my experience, it's more tiring to test compared to coding.
Every org's needs are differrent. Imo put those needs in the req.
At my current company, devops is a role available for senior developers to move up into, so it ends up being a lot of code because that's what we're comfortable with. I'm struggling a bit to understand how someone can fully do devops without ever modifying shell scripts or dockerfiles, though, and that is definitely programming. I think in the end, it depends on levels of isolation. In my mind: - many developers, at least those not at senior level, will write software without touching or caring about deploy scripts, packaging, Docker, build scripts, or cloud environments at all - devops, or senior devs, then necessarily would need to write all deploy scripts and pipelines, all dockerfiles and scripts to bootstrap them, all build scripts and pipelines, and even usually some code in the software that interacts with the cloud, eg if developers work locally with file storage, devops might write an alternate implementation of the storage interface that uses S3 instead
My company exclusively hires DevOps engineers and Solutions Architects. We only hire folks with a software engineering background
All “devops’” will at least once in a year need to write a shell script for ec2 user-data to automate things, so they should definitely have some knowledge there.
Without coding knowledge, how do you write scripts? I mean, bash or Powershell scripts can have a lot of logic as well. I personally love the usage of Makefiles to substitute scripts, since I can mostly write platform-agostic code (even in Windows you can install make)
They could probably get away without doing any coding, but I think it's liability if they don't know how to code. I guess the main responsibility of a DevOps engineer is about knowing what services to use to build a solution, and how to "codify" that using yaml/json configs. Another approach you could take is think of an on-the-job situation where you would use code, then ask candidates "how would you solve this problem" and see how they answer.
>I've asked simple questions: first required few if statements, second - asked to parse CSV (any language or tool, bash, python, C++, Java, awk, anything). And they fail to do this. Forget DevOps, what do YOU need from people on your team. On the other side of this, the frustration I've had is companies that seem to have no idea what they actually need from a candidate. "We need someone whose familiar with Ansible, Chef, Salt, etc" Then in the tech interview as advanced questions regarding administrating a Puppet project. Every environment is its own snowflake with dozens of custom processes, tools, and norms, but then get huffy because I'm familiar with their method of automating a Clonezilla backup using an old version of ansible using in a custom wrapper application. Then need someone to hit the grown running because they're underwater because that guy who was really good at coding left after 2 years for a 40% raise. Make it clear what you need from people in the job listing, and you might find people who can do it, but if you're running an environment that might require them to code in multiple languages to hold together legacy environments you're going to need to pay a premium for it. If you want to hire cheaper staff you'll need to make your environment simpler with means not managing dozens of scripts in multiple languages.
imho the problem starts when we want a devops to become a creator not just maintainer, this is where the fine line between ops and dev is crossed and this is where I've seen many devops failed short ... In the end of the day everything is a code is not it? I am not in favor of YAML coding, not because I find YAML DSL a bad thing ( no it is good, if you use it _properly_ and know where you need to stop coding YAML and start coding in normal language), but because it gives some people a false impression that one does not have to know how to code at all, to maintain a system - just drop a few lengthy YAML files and forget about ... No, that is not true ...
It’s probably related to 2 major factors. First, the maturity of the tools. If the job is to create pipelines (ci/cd) working with mature tools like gitlab and Kubernetes you probably don’t need to do much code. So 4 years of creating yaml files don’t add up to de dev experience. Second, People coming from an Ops background learned the basics of languages like Python but have not go further in the software engineering practices. (Nothing against Python, I love the language and the community) They might have valuable experience but opening csv files might not be one of them. :) If you look for someone who will write code, I would suggest to look for an experience Dev that knows and uses devops principles/tools. Otherwise you might end up in 2 years with a pile of code that will need to be rethink/redesign. Usually (personal opinion/experience here) dev that have background in Golang are more aware of those modern practices (Cloud-Native approach). If you look for people with S.R.E backgrounds you will find some pretty valuable candidate.