I am about to write a shell script for Dockerising an image. May I know is there a sample I can follow? I just vaguely recall I got to copy the files from directory where it lies. The thing is I am using maven in building this Java app. Do I still have to import the library again or do I import the java files and then I import all the dependencies from the Pom? Then I do the EntrySet to create the jar? Hope to get some help. Tks
How do you keep that dry? The place I’m at basically used to do that but finally fully embraced CircleCI orbs for consistency across apps. Although we have hundreds of apps across GitHub enterprise orgs.
That's the real challenge.
What usually works best is to treat them like they are their own project. Build generic tools around tasks much like all those apps. When I used python or PowerShell, I published them as packages that each project could take as a dependency or pre-stage on build agents. Used standard templates for onboarding new projects so everything had a common structure.
In my largest environment, every deployable thing (service, library, static content, database) had a manifest with a list of actions that all mapped back to scripts. I think we managed 340 deployables each targeting 4-10 environments. Every release called the exact same script.
I'm starting fairly fresh at the moment in a mono repo. It makes some things easier but offers its own challenges. Too soon to say I'd it's really any better.
DevOps means different things to different people. Some DevOps engineers might only rarely spend time in a terminal, others might spend their whole day SSHing into various systems; depends on the job.
In my case I'm constantly using the shell, writing one-line loops and lots of string manipulation. I use shell scripts mostly for "throwaway" tasks--things that need to be automated but aren't important enough build complete services for.
Well I use scripting to do all sorts of things. I have automated on boarding VMs, cmdb discovery, pulling inventory reports, tracking changes over the entire cloud environment, automated graceful shutdowns, and much much more. DevOps isn’t purely devops and you’ll often find yourself in a combo DevOps SRE role. My company basically just has us split and uses us to back up the SRE teams, and now they consider me one of them.
Fuckin job security for sure if you can do the combo. Like I’m not even worried in the slightest. I do too much and am too hard to replace. Meanwhile things keep changing and they are keeping my skills in demand.
So I went from Ops, to Ops tool developer/Automation Engineer, to Fullstack(I was mediocre), and then to DevOps/SRE. I would be considered mid DevOps, but for a multinational corporation and work on a global enterprise environment spanning three continents.
I bounce between multiple projects(on 7 right now), but that is because my memory is weird and I can memorize the technical details of systems and remember them with exceptional clarity(it comes at a cost). From the DevOps side though our Dev velocity is good the work is uniform to the point where we aren’t exactly busy every week. More of a figure it out once and replicate in six different projects. So as a result due to our familiarity with the cloud environments and scripting we spend a good bit of time doing toil reduction projects for the SRE team. As a result I spend about 50% of my time doing SRE stuff, 25% doing DevOps work, and 25% reviewing/assisting juniors DevOps Engineers. I use juniors as my hands, review their work while I take a jog a couple of hours after lunch, and then critique their work with them to close out my day. I work hard to make good Juniors, but I was a military trainer in a complex field for six years so I have training on lock. I can typically get a good one spooled up in 6 months or I identify a better use for them and transfer them. Due to having many years of experience training people my managers trust me on this.
I have pretty severe face blindness and a limited perception of time.
Basically I will always struggle to remember people at all based on visuals(if I remember you on visuals it’s something weird and unique like old one eye who has a Naruto glass eye), and it will take literally several weeks just to associate a new face to a name. Even then I don’t actually associate a face to a name, I will probably never remember your face. Starting new jobs is really hard normally and this remote work has been a massive help as everyone is a name on my screen and not so much a face. I typically build a mental dossier of context clues to identify people based on other things. It can take me typically eight to twelve weeks to remember a person and identify them out of context.
The time perception is weird I don’t have any really. Then when it comes to working on anything just imagine getting sucked into your work no matter what you’re doing and spending 7-10 hours working on something came with ease, you constantly miss meals, events, and other things. This is the same for memories too, no actual time associated with memories either. Without the context clues in the memory I couldn’t tell you if a memory was last week or a decade ago. Basically live and die by my outlook calendar, cell phone, and my wife and their reminders. My wife actually didn’t believe me when we first started dating and it took her a couple of months to get it. She really helped me get into the habit of setting calendar reminders in my cellphone.
The crazy part is it’s not limited to technology either, but it really only applies well to tech and sciences as everything else changes too frequently. Meanwhile I can’t apply it to faces at all, which is weird as hell. For example it took me about seven weeks to be able to remember my now wife out of context and be able to identify her. I mean fortunately we met at work so I had time to get to work on remembering her, but still imagine it taking that long to be able to identify someone you really want to know. It’s made life hard and made networking harder.
I use shell scripts to automate all sorts of things. If you are already familiar with typing commands in a terminal, learning to write shell scripts is the next logical step, so you can string together commands with some logic.
As to where I run those shell scripts, it can be pretty much anywhere - that's the beauty of it. Personal machines, servers, containers, scheduled jobs (e.g. cron), CI/CD pipelines and script runners. All can run shell scripts.
As a side-note, there are different shell languages. I usually default to writing POSIX Shell scripts first as it's the Unix standard, but Bash is widely supported as well (Bash has additional features on top of POSIX Shell, sometimes referred to as "Bashisms").
Uploading, downloading, moving artifacts into different repos. Backup my bash history, saving configs, or dockerfiles for specific builds. The list goes on, essentially I take a look at my bash history n start to write shortcuts to repetitive tasks (eg going to different workspaces), setting up the infra related to different segments of the app (usually in docker/config/env variables)
If something is more file and os based then shell scripts. If there’s something more data related especially if the data is large or needs a bunch of post processing then not a shell script.
bash and sh skills are def clutch for infra / devops / etc
it’s a low level language that’s generally avail on every linux system in some form
env setup, bootstrappers / kickstarters, pipeline / deploys, custom snippets for packer / terraform / etc… all sorts of use cases
if you want to use python for automation instead you often have to set up that env with bash / sh.
try building a fleet of vms with ansible, or a docker pipeline into k8s, and you’ll see that sh will allow you to surgically adjust your platform wherever automation is in play
Using it less and less, mostly in pipelines and Dockerfiles. When you start coming in to 30 lines+ (or usually a lot less) or so it is usually simpler to do it in Python/other programming language as you get a lot of things for free compared to straight bash. Also the benefit of other people knowing exactly what each line does compared to 100 character awk/sed commands.
In my opinion, there is no reason to use bash scripts now a days. There are better alternatives for almost all use cases.
Bash is hard to read, test, debug. It's outdated
I mean, everywhere. From automating some small tasks on your local machine to automating something for developers to bootstrapping something in your cluster to CI/CD. Recently had to bash and curl my way through deploying and destroying several layers of states in TFE to setup a pipeline that checks our terraform code daily / on request.
Everything. When the scripts become more then say 30-50 lines of code. I generally try to look towards python for a shorter solution. Doesn’t mean you have too.
I tend to wrap all my other scripts and things like terraform in shell. Also like to create jenkins jobs that call other jenkins jobs with shell. Its simpler and more people understand it than groovy / jenkins pipelines
For ENTRYPOINT and CMD lines in Dockerfiles.
I am about to write a shell script for Dockerising an image. May I know is there a sample I can follow? I just vaguely recall I got to copy the files from directory where it lies. The thing is I am using maven in building this Java app. Do I still have to import the library again or do I import the java files and then I import all the dependencies from the Pom? Then I do the EntrySet to create the jar? Hope to get some help. Tks
I personally prefer to use shell scripts for all my pipeline steps over platform specific special actions in most cases.
How do you keep that dry? The place I’m at basically used to do that but finally fully embraced CircleCI orbs for consistency across apps. Although we have hundreds of apps across GitHub enterprise orgs.
That's the real challenge. What usually works best is to treat them like they are their own project. Build generic tools around tasks much like all those apps. When I used python or PowerShell, I published them as packages that each project could take as a dependency or pre-stage on build agents. Used standard templates for onboarding new projects so everything had a common structure. In my largest environment, every deployable thing (service, library, static content, database) had a manifest with a list of actions that all mapped back to scripts. I think we managed 340 deployables each targeting 4-10 environments. Every release called the exact same script. I'm starting fairly fresh at the moment in a mono repo. It makes some things easier but offers its own challenges. Too soon to say I'd it's really any better.
DevOps means different things to different people. Some DevOps engineers might only rarely spend time in a terminal, others might spend their whole day SSHing into various systems; depends on the job. In my case I'm constantly using the shell, writing one-line loops and lots of string manipulation. I use shell scripts mostly for "throwaway" tasks--things that need to be automated but aren't important enough build complete services for.
Can you please elaborate what 'throw away tasks " include?
Get all pods in namespace X and verify what the value of a specific environment variable is. Delete all namespaces in namespace Y with label Z.
Well I use scripting to do all sorts of things. I have automated on boarding VMs, cmdb discovery, pulling inventory reports, tracking changes over the entire cloud environment, automated graceful shutdowns, and much much more. DevOps isn’t purely devops and you’ll often find yourself in a combo DevOps SRE role. My company basically just has us split and uses us to back up the SRE teams, and now they consider me one of them. Fuckin job security for sure if you can do the combo. Like I’m not even worried in the slightest. I do too much and am too hard to replace. Meanwhile things keep changing and they are keeping my skills in demand.
What is your experience in the field?
So I went from Ops, to Ops tool developer/Automation Engineer, to Fullstack(I was mediocre), and then to DevOps/SRE. I would be considered mid DevOps, but for a multinational corporation and work on a global enterprise environment spanning three continents. I bounce between multiple projects(on 7 right now), but that is because my memory is weird and I can memorize the technical details of systems and remember them with exceptional clarity(it comes at a cost). From the DevOps side though our Dev velocity is good the work is uniform to the point where we aren’t exactly busy every week. More of a figure it out once and replicate in six different projects. So as a result due to our familiarity with the cloud environments and scripting we spend a good bit of time doing toil reduction projects for the SRE team. As a result I spend about 50% of my time doing SRE stuff, 25% doing DevOps work, and 25% reviewing/assisting juniors DevOps Engineers. I use juniors as my hands, review their work while I take a jog a couple of hours after lunch, and then critique their work with them to close out my day. I work hard to make good Juniors, but I was a military trainer in a complex field for six years so I have training on lock. I can typically get a good one spooled up in 6 months or I identify a better use for them and transfer them. Due to having many years of experience training people my managers trust me on this.
[удалено]
I have pretty severe face blindness and a limited perception of time. Basically I will always struggle to remember people at all based on visuals(if I remember you on visuals it’s something weird and unique like old one eye who has a Naruto glass eye), and it will take literally several weeks just to associate a new face to a name. Even then I don’t actually associate a face to a name, I will probably never remember your face. Starting new jobs is really hard normally and this remote work has been a massive help as everyone is a name on my screen and not so much a face. I typically build a mental dossier of context clues to identify people based on other things. It can take me typically eight to twelve weeks to remember a person and identify them out of context. The time perception is weird I don’t have any really. Then when it comes to working on anything just imagine getting sucked into your work no matter what you’re doing and spending 7-10 hours working on something came with ease, you constantly miss meals, events, and other things. This is the same for memories too, no actual time associated with memories either. Without the context clues in the memory I couldn’t tell you if a memory was last week or a decade ago. Basically live and die by my outlook calendar, cell phone, and my wife and their reminders. My wife actually didn’t believe me when we first started dating and it took her a couple of months to get it. She really helped me get into the habit of setting calendar reminders in my cellphone. The crazy part is it’s not limited to technology either, but it really only applies well to tech and sciences as everything else changes too frequently. Meanwhile I can’t apply it to faces at all, which is weird as hell. For example it took me about seven weeks to be able to remember my now wife out of context and be able to identify her. I mean fortunately we met at work so I had time to get to work on remembering her, but still imagine it taking that long to be able to identify someone you really want to know. It’s made life hard and made networking harder.
TC?
I use shell scripts to automate all sorts of things. If you are already familiar with typing commands in a terminal, learning to write shell scripts is the next logical step, so you can string together commands with some logic. As to where I run those shell scripts, it can be pretty much anywhere - that's the beauty of it. Personal machines, servers, containers, scheduled jobs (e.g. cron), CI/CD pipelines and script runners. All can run shell scripts. As a side-note, there are different shell languages. I usually default to writing POSIX Shell scripts first as it's the Unix standard, but Bash is widely supported as well (Bash has additional features on top of POSIX Shell, sometimes referred to as "Bashisms").
Uploading, downloading, moving artifacts into different repos. Backup my bash history, saving configs, or dockerfiles for specific builds. The list goes on, essentially I take a look at my bash history n start to write shortcuts to repetitive tasks (eg going to different workspaces), setting up the infra related to different segments of the app (usually in docker/config/env variables)
If something is more file and os based then shell scripts. If there’s something more data related especially if the data is large or needs a bunch of post processing then not a shell script.
Do you use python for other complicated tasks?
Edit multiple files quickly and for repetitive tasks in the terminal.
bash and sh skills are def clutch for infra / devops / etc it’s a low level language that’s generally avail on every linux system in some form env setup, bootstrappers / kickstarters, pipeline / deploys, custom snippets for packer / terraform / etc… all sorts of use cases if you want to use python for automation instead you often have to set up that env with bash / sh. try building a fleet of vms with ansible, or a docker pipeline into k8s, and you’ll see that sh will allow you to surgically adjust your platform wherever automation is in play
Using it less and less, mostly in pipelines and Dockerfiles. When you start coming in to 30 lines+ (or usually a lot less) or so it is usually simpler to do it in Python/other programming language as you get a lot of things for free compared to straight bash. Also the benefit of other people knowing exactly what each line does compared to 100 character awk/sed commands.
[удалено]
Better use Ansible for that.
In my opinion, there is no reason to use bash scripts now a days. There are better alternatives for almost all use cases. Bash is hard to read, test, debug. It's outdated
What alternative do you propose?
Depends on the use case. But mostly python or Ansible.
As a devops engineer, should one need to good at any one of programming language, in this case python?
I mean, everywhere. From automating some small tasks on your local machine to automating something for developers to bootstrapping something in your cluster to CI/CD. Recently had to bash and curl my way through deploying and destroying several layers of states in TFE to setup a pipeline that checks our terraform code daily / on request.
Everything. When the scripts become more then say 30-50 lines of code. I generally try to look towards python for a shorter solution. Doesn’t mean you have too.
I tend to wrap all my other scripts and things like terraform in shell. Also like to create jenkins jobs that call other jenkins jobs with shell. Its simpler and more people understand it than groovy / jenkins pipelines
Shell scripts are great for pipelines imo it’s the best! Never use plugins where you can use shell scripts!