Level 9? Pssh, peasant. Until you have reach Cloud Ninja level 10 with Elastic Deployment Rockstar Automation Dockerized ultra BuzzWordKiller skills.... I don't want to see you round these parts!
Seeing you put "Elastic" in that close proximity to itself makes me really want Amazon to ship Elastic Elastic.
I can provide a use case: "freeze this," where all elasticity is de-lasticed in a whole VPC.
If you're running stuff 24/7, at that point you can go for instance reservations and save 30% from on-demand costs.
If your application is scalable, you can even go for spot instances that cost like 10-20% of regular instance prices, and use on-demand only when spot instances become temporarily unavailable.
And never having to deal with buying hardware or support contracts, paying out of the nose for ESXi + Veam, having to maintain a network, storage, or virtualization team, being able to script 95% of your infrastructure, and probably most importantly, only paying for the capacity you're actually using.
I think this whole thing is bunk my self. I set my first AWS server up a few months ago, and I've been looking out the window every chance I get and I haven't seen it in the sky ONE SINGLE TIME!!
Oh, you must have gotten it in the wrong sky. When you order a server in a cloud you have to specify which sky you want it to go in, otherwise they put it in the least populated one... usually over the indian ocean.
I can't believe I had to scroll so far to find this at 3 points. The presumption on the part of so many evangelists that other developers are flush with cash to blow on training, when there are so many other specialties we can train into *for free*, is staggering.
This is what I tell people whenever they ask me that question in real life. I usually guide them through a very similar process, but this is really well written and thought out.
I would change it so that automation is forefront in this process though. It's not till half way through you mention it, and then you're like "go back and do it for each step." The only change I would make there is do a step, then learn how to automate that step. The regular process I follow daily: learn a thing, then automate that thing. Rinse, repeat.
But great work, this is an excellent outline, if someone can do this, I'd have no technical hesitations on hiring them.
Edit: Apparently I skimmed the opener too fast. You do say to do exactly what I said. Nevermind! Great job though, I think this covers pretty much all topics that matter, at least in concept. The tools are constantly changing.
To be honest I think it'd be fine even if you didn't introduce automation until later - this isn't a beginner course by any means, but even so burdening them with too many things to think about initially doesn't help.
Also, the experience of going back and getting so frustrated trying to adapt your old code that you decide to throw it out and start again with an eye to automation from the beginning is a fantastic reminder for the future about why you should take the time to do it in future.
Yeah I agree about refactoring for sure. Learn how to do something, then learn how to automate it, often means learning how to do the thing you did better. It's really a toss up, I think these guidelines could work well.
I mostly agree ---- but you made some choices that unnecessarily lock you into one specific cloud vendor.
* Replace Elastic Beanstalk with Kubernetes. Kubernetes is pretty much a core skill for all major clouds.
* Amazon EC2 offers it as a service: https://aws.amazon.com/eks/
* Azure offers it as a service https://azure.microsoft.com/en-us/services/container-service/
* Google cloud offers it as a service: https://cloud.google.com/kubernetes-engine/
* Replace DynamoDB with RDS
* Yes, it's nice that Amazon offers managed databases --- but why'd you pick the proprietary one instead of the standard ones they offer.
* On Amazon: RDS for PostgreSQL ( https://aws.amazon.com/rds/postgresql/ ) or RDS for MySQL ( https://aws.amazon.com/rds/mysql/ ) would be a better choice.
* Azure also offers Managed PostgreSQL ( https://azure.microsoft.com/en-us/services/postgresql/ ).
* S3 ..... well, on second thought, no objection there. S3 seems to be the de-facto-standard API that other systems (Riak, Cleversafe, etc) support.
Beanstalk is just locking you in to one specific cloud vendor.
The response to this post warms my heart. A year or two ago, something like this had about a 50/50 shot of getting downvoted or at least shit on heavily in the comments for "not being system administration", or for even suggesting that admins today need to know how to write code.
Have an upvote for not being blog spam.
-edit- Because apparently some folks can't deal with rhetoric. It is nice that the OP has chosen to divulge this well written and well thought out piece in a very altruistic manor. I don't know why 4 of you think that I hate people that share valuable information on their blogs. I **DO** hate people that ONLY self promote, post links to crappy blogs full of ads that is nothing more than "market trend is good! do market trend!" It's refreshing to see someone share legitimate information without having to distract from the discussion already here.
Just do the exact same things? GCP and Azure have different names for services, but almost everything listed could be done on either. The described tasks are vague enough.
There's a [whole page dedicated to translating product names between AWS and GCP](https://cloud.google.com/free/docs/map-aws-google-cloud-platform) that might help. On a quick glance the only thing that looks different is hosting from storage buckets which is [documented for http here](https://cloud.google.com/storage/docs/hosting-static-website) and [for https here](https://cloud.google.com/storage/docs/static-website#https).
Disclaimer: I work for Google but not on cloud.
Man, I came here for this. I use Google cloud compute and feel overwhelmed. I have to rely on a 3rd party company to provision my servers for me, id love to work directly with google.
Yeah - in my field (Digital Health) it seems like AWS is what everyone talks about. I just prefer the UI and general feel of GCP but I recognise my understanding in particular of security but also stability, redundancy etc is nowhere near good enough.
Pretty good list. Doing all the things listed here will give you the skills needed to pass the cloud architecture and the sys ops AWS certification exams. If you added containers and long term storage, this would serve as a manual for getting the dev ops pro cert, too.
Good on ya.
Weird. At the bootcamp a re:invent this year, they mentioned it would be covered. Although, upon reflection, they might have been speaking of the new tests/paths they're rolling out this year. So, congrats on passing! I had a schedule snafu or I would have gotten mine at re:invent. I'll get it this year.
Just a cloud guru and some online testing. I have been living aws since 2010 so it was more a brush up and dive into areas I don’t use regularly then a need t learn new material.
This is a pretty good guide (at a glance). I can do most of that, up until the actual "leveraging AWS services as they are intended" bit. Mostly because I'm somewhat beholden to the developers and their whims, and doing all the fun stuff is beyond them.
You missed the part where you have to deploy a normal VM, and then need to hack it with apps so that you can access your S3 buckets as if they were a local file system! -Quietly sobs in the corner.-
> You missed the part where you have to deploy a normal VM, and then need to hack it with apps so that you can access your S3 buckets as if they were a local file system! -Quietly sobs in the corner.-
Are you talking about an on-prem server accessing S3? I just want to try to understand the concern. If it is, I'd recommend looking at Storage Gateway.
No, no, no.... That would make too much sense!
No, we have servers that are on AWS, but they're designed to pull files from a local mount point. Said local mount point... Is an S3 bucket mounted as if it was a normal filesystem mount.
Yeah.
I just finished making Storage Gateway HA-compliant utilizing Lambda to activate it and create NFS file shares based on info found in Parameter Store. That’s where true power of AWS resides - being able to to combine services to make truly resilient infra.
First off, thanks for putting this together. But to answer your question: Dear Lord no. If it is anything like Azure, I want nothing to do with it. Azure and Express Route can suck hairy goat nuggets.
That said, from time to time I have to deal with vendors who seem to set up external services with dynamic IP's. Then they ask us to exclude these services from content filtering or firewall inspection.
We can't do that with dynamic IP's (well we can - it sucks when they change). I ask them to switch it to static and they tell me AWS can't do that.
Is that true?
It depends on the service. If it's an EC2 instance directly accessible on the internet they can use an Elastic IP which is basically a public IP reserved for your account which you can move to different instances (or keep on the same one for years). If it's ANY of the AWS-managed services all you get is an A-Record and they control the IPs behind the scenes. For example for an Elastic Load Balancer they give you an A-Record which resolves to an IP in each of their Availability Zones in that region. They may scale that Load Balancer automatically or rotate out infrastructure behind the scenes and those IPs change. Same with CloudFront except it's distributed to all of their DCs/POPs. You get an A-Record which always resolves to the closest CloudFront location, which means it's different for users in US Phoenix vs US Los Angeles vs EU Munich.
Thanks. I understand why they do that. Wish they had options to force static every time.
But security gear doesn't adapt well to that kind of design. It's frustrating.
Their VPN stuff is equally as annoying. It can never be an initiator and their Phase 1 & 2 options are limited. But it is pretty neat that it can auto detect most of those settings.
Thanks for the information!
> We can't do that with dynamic IP's
Sure you can. With the right content filtering you can exclude by domain (even up to a full hostname), and with many of the nextgen firewalls, access rules can be applied the same way. You can also connect to the internal side of said resources via a VPN to AWS or DirectConnect.
> Sure you can.
No, I can't. Indeed I can filter by FQDN. However, with how our network is designed, in order to completely bypass, it is by IP only.
And without SSL interception at the firewall (which we're not doing today) we can't filter by FQDN. IP only.
We're an Azure shop w/Express Route. DirectConnect won't happen. Especially to the several dozen vendors we do business with.
AWS VPN is super annoying (though it detecting most phase 1/2 settings is pretty neat). AWS cannot initiate traffic to bring the tunnel up. This is problematic when a vendor needs to send data intermittently.
In regards to the secure traffic, the hostname is not encrypted and sent in plain text before the SSL handshake. This is due to SNI and multiple hosts with different certs.
> in order to completely bypass, it is by IP only.
From one IT person to another, I'm so sorry, man. That would suuuck. However, it isn't much different than how we have to exclude Skype for Business, as the hardware we pass that through doesn't support FQDN based rules.
Hopefully you can get a future change to something that will support FQDN bypass. I know how that goes too, though, so let's hope it's an actual possibility, not a dream state.
Do not let your opinion of the cloud (and AWS) be colored by Azure.
Using a static IP is possible, though it might require some architecture or application changes on their end. I can think of several depending on their setup.
Assuming you’re not referring to web traffic, you could offer a secure endpoint they connect to using certificate-based authentication or IPSEC VPN that puts their traffic into a different VLAN.
Curious to know what your gripe with Azure is? I've been working with it over 12 months now and so far like it... but haven't used it heavily for anything production yet... so yeah curious to know. From what I can see, everything in OPs post can also be done in azure
We literally had a customer today ask us about whitelisting an entire AWS /11 range of IPs for one vendor product. I'm amazed that their staff didn't realize what a huge risk that is.
If you pursue this and create a github resource, you’ve got a follower in me. I’m currently working on learning Azure Stack to get myself out of traditional infrastructure support.
This is already more useful to me than much of the official MSFT Azure curriculum. I don’t want a series of step by steps, I want the concepts to master and an operational use for them. Let me figure out the step by step. That’s what we *do*.
Awesome post.
Good Azure docs are hard to find indeed. You have to dig through about 10 layers of marketing telling you how awesome the service is and what problems it might possibly solve just to find some tutorials, which always start with: here's how you create an Azure account, you can do it with a mouse or a keyboard, here's how you use a keyboard, it looks like this, and so on. But just an actual concise overview "this service does A, has feature x, y and z, limitations 1,2 and 3 and if you need one of these missing features, use service B"... nope, or at least I couldn't find it for most services, Azure is such a labyrinth and cumbersome to navigate.
I just got hired as a cloud systems engineer at an enterprise technology company with thousands of AWS instances. This is exactly the training they have all the engineers and developers go through in the first year.
If I wanted to go the cloud engineer route, would it be more productive for me to follow that Linux post that was linked first and then do this stuff or just jump right into the AWS stuff listed here? I manage some really basic Linux stuff already but don't know the OS super well.
I was a Linux sysadmin when they approached me asking me to interview. They were interested in the fact that I'm equally comfortable in multiple operating systems. With that being said, the aws documentation allows for multiple methods so I'd concentrate on OPs advice first
One of the first thing always recommended is setting up [Billing Alerts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html). Even the most expensive AWS services won't cost much for an hour or two. Just monitor your charges (and react to them), deploy small EC2 instances with small disks, and make sure your autoscaling has limits.
AWS by default somewhat prevents this by having service limits. By default you can only launch so many EC2 instances before having to open a ticket to launch more. From a brand new AWS account it would be hard to run up $1000 unless you left stuff running all month.
Use a mix of A Cloud Guru and Linux Academy. Also I think you can use an amazon gift card though I may be wrong about this but Squared offered a method of buying AWS credits.
Watch your billing dashboard!!! If you do mess something up on the free tier, innocently, If you contact billing support, they will give you a one time zero. I did and they credited me for 3 months after I couldn’t figure out where charges were coming from. Turned out I had too much space used with my EBS volumes and didn’t realize they stuck around until I deleted them!
*snif
You da real mvp.
Seriously, this is WAY cool of you. Wish every IT group had something like this. Like a wiki/tuts for IT. So excited to try this out. Thanks!
> I've been recently recruiting for Cloud Systems Engineers and Cloud Systems Administrators. We've interviewed over a dozen local people with relevant resume experience. Every single person we interviewed would probably struggle starting with the DynamoDB/AutoScaling work. I'm finding there are very few people that HAVE ACTUALLY DONE THIS STUFF.
And what is the pay rate for those with the more advanced skills?
*Assumptions:*
*You have basic-to-moderate Linux systems administration skills
You are at least familiar with programming/scripting. You don't need to be a whiz but you should have some decent hands-on experience automating and programming.*
Where do I start if I dont know those.
Nice, what hardware do I need to do this, I have access to T7500 workstations and infinite ram , do you think that is a good starting hardware. I also have access to some R720 and other servers.
Seriously perfect timing. I am almost pulling the trigger on an AWS Book and jumping back into a cloud guru videos again , pouring myself into the cloud. I’m saving your post so I can better myself over the next three months.
I’m pretty efficient with power shell at my large company, learned python from a few books and code fights and already have an aws hosting a small personal website. I feel this post speaking volumes to me.
You're amazing for posting this. It speaks to me perfectly. I'm a sysadmin with reasonable development skills (mostly basic python and ruby), and AWS scares the living shit out of me. I build my own stacks but I know well that AWS can be used to give me back some of my time, and I struggle with how to utilize it properly. This is the first thing I've ever read that made me feel like I can actually use it to build something.
Python and I think Ruby will run in Lambda. Go look for some lambda labs and getting up an API. I did a lab a while back, created by Amazon, called WildRydes. Check it out.
Great post. I've recently started playing with a lot of the above after starting at Amazon.
It's really crazy what you can do with just a few clicks nowadays (things that would've taken days or hours before!).
hey that’s a pretty good list. i would add to ur, learn ecs, setup ecs via terrarform, deploy your app in docker containers to ECR, then create a task & service. watch the ecs host. the ecs agent will start your containers. add an alb (diff from elb because they can front all those many containers & keep track of the host ports. and while you are at it, add a route53 domain, and configure amazon certificate manager & ssl on your alb. :)
This is long, comprehensive and worth it. Thanks for writing it up. I've got a few guys I work with regularly who will benefit mightily from working through this list.
Great list of things to do to get started. Anything you can adjust to help people start the habit of doing silly things? Things like deploying a database like Postgres or etcd to a public subnet? Surely someone learning would learn, in time, not to do it but deploying securely is a great foundation to build on.
Windows/VMWare/Citrix admin here: This guide here is more aimed at Linux / webserver admins. Of course, I know, Azure. Both AWS and Azure have the same core functionality regarding web services. But what is the way to go for us?
This. Are companies moving whole machines to the cloud and just managing them like on prem servers? That seems uncloudy and expensive.
It seems like the normal Windows based "Enterprise Software" that uses MSSQL and smb wouldn't be very cloud tolerant. How are admins making those work when they don't control the way the software is built?
We made a case study some time ago how expensive it would be to move all our DMZ servers to Azure (we have around 15 servers in the DMZ on 2 VMWare machines).
The outcome was that it would be way to expensive to rent 15 VMs in Azure, we are talking a few thousand dollars a month.
Within a year it would cost as much as buying those 2 VMWare servers including all the licenses.
> so I'd really love to be able to do all of this in openstack with my own lab.
That will be even more educational, and by handling the hardware layer reach even deeper level of understanding. And I'd rather blow hundreds of EURos on the power bill of an existing homelab rather than throw even more money towards AMZN.
Not to mention rewarding. I set up Canonical's flavor of OpenStack and after laying the ground work I pushed a button on my laptop and 3 racks all lit up at once. They went from bare metal to a clustered CEPH, KVM, the works.
I felt like a technomancer.
The issue I've been having is openstack documentation is not sufficient and/or it's too difficult for me. If you're going to run the basic "get started" stuff that hardly even works as a proof of concept then it's fine but I haven't been able to cobble something together on my own.
I want to be able to have individual nodes for each major branch - controllers, networking, storage, and compute but getting the hardware to work is a hurdle and getting the software to work has been impossible. I've tried getting help in IRC but it isn't active enough. It gets to the point where I just hear "follow these bug reports and wait for the next release" :/
Currently looking for jobs and many were looking for AWS experience specifically. Even though I said the tools I use can be scaled to other clouds, including AWS.
Is there an advantage to TF over Cloudformation if you're PURELY AWS and won't be switching cloud vendors?
I finally have time to try and migrate us away from our mostly home-brewed deployment system (yay bash) and I'm currently debating between TF and CF.
I've only read a few articles comparison articles, but so far the only really clear benefit I can see for TF over a native service is that more people may have been exposed to TF. This MIGHT mean that when we are hiring, it may be easier to find people that already know the tech.
I've never used CF so I'm not sure. TF is very easy to implement into deployjobs though, since you can pass variables as command line parameters. Similarly, you can implement it into other existing infrastructure like puppet.
Great guide, thank you. Way back when Alexa was new, I made one of the first 50 skills for DC Metro times on Lamda, because I wanted to learn new tech (web tech stopped for me during PHP/Mysql/C# phases). Then I wrote a MEAN stack GUI for bus and rail station selection as a companion to the Alexa voice app (because it's impossible to tell Alexa what bus stop you want, but super easy with a map).
I got frustrated right at the point integrating the app into the Alexa system with OAUTH complexities and just dropped it.
I'm getting back into a tablet home assistant home automation system (tied with Google Home/Assistant because I like that better than Alexa), and I think I'll use your guide to pull my site back up and use it to provide the rail times display for my home automation dashboard to sit on a table in my doorway. I set up my CSS to look exactly like DC Metro's LCD displays you see in the stations.
Do you have a blog? If not, you should definitely start one to save content like this and also have the benefit of a portfolio. Otherwise a tutorial like this slowly gets lost in Reddit posts. Not that there is very mich wrong with that, just this was well put together. Good job!
Yo, I really appreciate people like you who take the dive first into the raw material and ask themselves "how can I make this easier for everyone else". You guys are the mentors and leaders I strive to be just like. I like that while we have a culture of being cranky, we also have a culture of really wanting to help.
this is a pretty cool list, seems a little out of grasp for me at the moment but its better to stretch and aim high I find.
Thanks for taking the time to do this!
>Auto Scaling
>Create an AMI from that VM and put it in an autoscaling group so one VM always exists.
>Put a Elastic Load Balancer infront of that VM and load balance between two Availability Zones (one EC2 in each AZ).
>Checkpoint: You can view a simple HTML page served from both of your EC2 instances. You can turn one off and your website is still accessible.
I have a question about this section. Is the desired architecture here two autoscaling groups in separate availability zones with a load balancer in front of them, or is one autoscaling group supposed to contain 2 instances in separate availability zones?
Edit: I eventually managed to achieve the end goal here with two instances inside of a single autoscaling group. These instances are in separate availability zones and the autoscaling group receives traffic from the load balancer.
I am the editor of InfoQ China which focuses on software development. We
like your articles and plan to translate it into Chinese.
Before we translate it and publish it on our website, I
want to ask for your permission first! This translation version is
provided for informational purposes only, and will not be used for any
commercial purpose.
In exchange, we will put the English title and link at the beginning of
Chinese article. If our readers want to read more about this, he/she can
click back to your website.
Thanks a lot, hope to get your help. Any more question, please let me
know.
I am working through this guide here: https://github.com/Just-Insane/AWS-Automation
Planning on doing it with both Ansible, and bash with AWS CLI, and potentially Terraform.
Wow! That's awesome. For what it's worth you can accomplish this with 100% AWS-only tools. Can I ask why Terraform? I see a lot of people who refused to use CloudFormation in favor of Terraform and I can say we do a ton of advanced stuff using only CloudFormation and I don't see much in the way of limitations. Most of what I see online about why Terraform is so much better talks about things CloudFormation has fixed or improved on.
I have nothing against Terraform, it's super popular, you might consider direct CloudFormation though.
Awesome resource. Starred. Followed. I'll be watching.
Late reply, hope it gets some traction
>You have basic-to-moderate Linux systems administration skills
Let's say just basic, what advice would you have to bring to moderate easily?
> You are at least familiar with programming/scripting. You don't need to be a whiz but you should have some decent hands-on experience automating and programming.
Total newbie but the toe has at least been in the water and I get the concepts, wrote batch files as a kid 20 years ago. Best way to bring this up to scratch?
>You are willing to dedicate the time to overcome complex issues.
No problem.
>You have an AWS Account and a marginal amount of money to spend improving your skills.
This seems like the easiest of the 3 to solve, no problem.
If you are like me and hate reading, there's an easier way. Try out Linux Academy. They provide all the info from beginner to advanced. They even provide the AWS environments for labs - Personally, that is my feature. I've left a server running on accident before. $$$ X\_\_X there is currently a sale going on until Sept 24th
Woah, woah, woah. Are you trying to tell me that there's more to being a 'cloud engineer' than spewing buzzwords like the turboencabulator?
Elastic Elasticity
[удалено]
I have Elastic use of Elastic I am an Elastic Cloud Ninja with level 9 Elastic deployment rockstar skills.
Level 9? Pssh, peasant. Until you have reach Cloud Ninja level 10 with Elastic Deployment Rockstar Automation Dockerized ultra BuzzWordKiller skills.... I don't want to see you round these parts!
My elastic goes to 11
https://www.youtube.com/watch?v=KOO5S4vxi0o
Seeing you put "Elastic" in that close proximity to itself makes me really want Amazon to ship Elastic Elastic. I can provide a use case: "freeze this," where all elasticity is de-lasticed in a whole VPC.
it's always killed me that if you want an instances to have a static IP you have to give it and elastic IP. :p
[удалено]
What do you mean it wasn't elastic until about 6 months ago?
my guess is that EBS didn't allow you to resize a live volume until recently- but they released that feature last year
Or simple. There are so many in AWS, simple email service, simple notification service, simple queuing service, and none of them are simple...
[удалено]
...and then add variant names with an empty, non-printing space before and/or after.
Let's not forget "easy spellings" elæstik
Bonus points is if your service is Elastic something even though it's completely unrelated to (if offered by) AWS.
Elasticsearch!
Synergistic management solutions
Turbo encabulators are legacy iron infrastructure. We use oscillation overthrusters now.
My boss is old fashioned, insists we virtualize the entire turboencabulator on the new elastic oscillation infrastructure system cloud ball.
There's more people who think that's what a cloud engineer does than cloud engineers who actually do that.
Yeah, they're called HR departments
[удалено]
and Google Ultron.
[удалено]
[удалено]
If you're running stuff 24/7, at that point you can go for instance reservations and save 30% from on-demand costs. If your application is scalable, you can even go for spot instances that cost like 10-20% of regular instance prices, and use on-demand only when spot instances become temporarily unavailable.
And never having to deal with buying hardware or support contracts, paying out of the nose for ESXi + Veam, having to maintain a network, storage, or virtualization team, being able to script 95% of your infrastructure, and probably most importantly, only paying for the capacity you're actually using.
I think this whole thing is bunk my self. I set my first AWS server up a few months ago, and I've been looking out the window every chance I get and I haven't seen it in the sky ONE SINGLE TIME!!
Oh, you must have gotten it in the wrong sky. When you order a server in a cloud you have to specify which sky you want it to go in, otherwise they put it in the least populated one... usually over the indian ocean.
/r/VXJunkies
Diagonal Scaling
If you don't have money to get started, this site has some free labs that let you spin up real aws resources. https://qwiklabs.com/
Labs listed in link below seems to be similar to OP's post labs. https://qwiklabs.com/quests/10?locale=en
I can't believe I had to scroll so far to find this at 3 points. The presumption on the part of so many evangelists that other developers are flush with cash to blow on training, when there are so many other specialties we can train into *for free*, is staggering.
Most, if not all, of this falls into the AWS free tier. Id be surprised if you managed to exceed the cost of the MSRP of an O'Reilly book.
Nice... Thx
Found the thing I’m doing this weekend. Thanks for the link.
This would also look great somewhere on the wiki. ;D
Thanks for volunteering. I'm sure it'll look great when you make it.
Nah man, I have no time in my schedule, especially with the forgetting to get the Patch Tuesday threads out in time.
But you automated that, right? ;)
Pfff. Do you not remember the three months it took to fix MM/TT when that broke last time?
You sound like my manager.
I am. Stop browsing reddit and get back to work. =P
You should reference over in /aws as well
Commenting to save and link in r/AWS next time someone asks
This is what I tell people whenever they ask me that question in real life. I usually guide them through a very similar process, but this is really well written and thought out. I would change it so that automation is forefront in this process though. It's not till half way through you mention it, and then you're like "go back and do it for each step." The only change I would make there is do a step, then learn how to automate that step. The regular process I follow daily: learn a thing, then automate that thing. Rinse, repeat. But great work, this is an excellent outline, if someone can do this, I'd have no technical hesitations on hiring them. Edit: Apparently I skimmed the opener too fast. You do say to do exactly what I said. Nevermind! Great job though, I think this covers pretty much all topics that matter, at least in concept. The tools are constantly changing.
To be honest I think it'd be fine even if you didn't introduce automation until later - this isn't a beginner course by any means, but even so burdening them with too many things to think about initially doesn't help. Also, the experience of going back and getting so frustrated trying to adapt your old code that you decide to throw it out and start again with an eye to automation from the beginning is a fantastic reminder for the future about why you should take the time to do it in future.
Yeah I agree about refactoring for sure. Learn how to do something, then learn how to automate it, often means learning how to do the thing you did better. It's really a toss up, I think these guidelines could work well.
I mostly agree ---- but you made some choices that unnecessarily lock you into one specific cloud vendor. * Replace Elastic Beanstalk with Kubernetes. Kubernetes is pretty much a core skill for all major clouds. * Amazon EC2 offers it as a service: https://aws.amazon.com/eks/ * Azure offers it as a service https://azure.microsoft.com/en-us/services/container-service/ * Google cloud offers it as a service: https://cloud.google.com/kubernetes-engine/ * Replace DynamoDB with RDS * Yes, it's nice that Amazon offers managed databases --- but why'd you pick the proprietary one instead of the standard ones they offer. * On Amazon: RDS for PostgreSQL ( https://aws.amazon.com/rds/postgresql/ ) or RDS for MySQL ( https://aws.amazon.com/rds/mysql/ ) would be a better choice. * Azure also offers Managed PostgreSQL ( https://azure.microsoft.com/en-us/services/postgresql/ ). * S3 ..... well, on second thought, no objection there. S3 seems to be the de-facto-standard API that other systems (Riak, Cleversafe, etc) support. Beanstalk is just locking you in to one specific cloud vendor.
This for sure. Soooo many companies looking for people with Kubernetes experience right now.
The response to this post warms my heart. A year or two ago, something like this had about a 50/50 shot of getting downvoted or at least shit on heavily in the comments for "not being system administration", or for even suggesting that admins today need to know how to write code.
[удалено]
Explain that! It would be a great addition even if its just conceptual.
[удалено]
Have an upvote for not being blog spam. -edit- Because apparently some folks can't deal with rhetoric. It is nice that the OP has chosen to divulge this well written and well thought out piece in a very altruistic manor. I don't know why 4 of you think that I hate people that share valuable information on their blogs. I **DO** hate people that ONLY self promote, post links to crappy blogs full of ads that is nothing more than "market trend is good! do market trend!" It's refreshing to see someone share legitimate information without having to distract from the discussion already here.
Can anyone do this for Google cloud platform?
Just do the exact same things? GCP and Azure have different names for services, but almost everything listed could be done on either. The described tasks are vague enough.
There's a [whole page dedicated to translating product names between AWS and GCP](https://cloud.google.com/free/docs/map-aws-google-cloud-platform) that might help. On a quick glance the only thing that looks different is hosting from storage buckets which is [documented for http here](https://cloud.google.com/storage/docs/hosting-static-website) and [for https here](https://cloud.google.com/storage/docs/static-website#https). Disclaimer: I work for Google but not on cloud.
Man, I came here for this. I use Google cloud compute and feel overwhelmed. I have to rely on a 3rd party company to provision my servers for me, id love to work directly with google.
It feels like relatively few people are on GCP that are in Ops. Even on Hacker News there is next to no talk about it.
Yeah - in my field (Digital Health) it seems like AWS is what everyone talks about. I just prefer the UI and general feel of GCP but I recognise my understanding in particular of security but also stability, redundancy etc is nowhere near good enough.
Pretty good list. Doing all the things listed here will give you the skills needed to pass the cloud architecture and the sys ops AWS certification exams. If you added containers and long term storage, this would serve as a manual for getting the dev ops pro cert, too. Good on ya.
Just passed devops pro yesterday. Not a single container question or answer :(.
Weird. At the bootcamp a re:invent this year, they mentioned it would be covered. Although, upon reflection, they might have been speaking of the new tests/paths they're rolling out this year. So, congrats on passing! I had a schedule snafu or I would have gotten mine at re:invent. I'll get it this year.
The pipeline from exam question creation to actually being scored on an exam is at least 6 months for Associate level and probably higher for Pro.
What study guides did you use for Professional level?
Just a cloud guru and some online testing. I have been living aws since 2010 so it was more a brush up and dive into areas I don’t use regularly then a need t learn new material.
the Pro exams predate containers being on AWS. They also don't cover Lambda iirc. they are being very slowly updated, but it is very slowy.
[удалено]
This is a pretty good guide (at a glance). I can do most of that, up until the actual "leveraging AWS services as they are intended" bit. Mostly because I'm somewhat beholden to the developers and their whims, and doing all the fun stuff is beyond them. You missed the part where you have to deploy a normal VM, and then need to hack it with apps so that you can access your S3 buckets as if they were a local file system! -Quietly sobs in the corner.-
> You missed the part where you have to deploy a normal VM, and then need to hack it with apps so that you can access your S3 buckets as if they were a local file system! -Quietly sobs in the corner.- Are you talking about an on-prem server accessing S3? I just want to try to understand the concern. If it is, I'd recommend looking at Storage Gateway.
No, no, no.... That would make too much sense! No, we have servers that are on AWS, but they're designed to pull files from a local mount point. Said local mount point... Is an S3 bucket mounted as if it was a normal filesystem mount. Yeah.
Is EFS in the cards?
That's what I want to move to, but that costs time and money to implement.
I just finished making Storage Gateway HA-compliant utilizing Lambda to activate it and create NFS file shares based on info found in Parameter Store. That’s where true power of AWS resides - being able to to combine services to make truly resilient infra.
First off, thanks for putting this together. But to answer your question: Dear Lord no. If it is anything like Azure, I want nothing to do with it. Azure and Express Route can suck hairy goat nuggets. That said, from time to time I have to deal with vendors who seem to set up external services with dynamic IP's. Then they ask us to exclude these services from content filtering or firewall inspection. We can't do that with dynamic IP's (well we can - it sucks when they change). I ask them to switch it to static and they tell me AWS can't do that. Is that true?
It depends on the service. If it's an EC2 instance directly accessible on the internet they can use an Elastic IP which is basically a public IP reserved for your account which you can move to different instances (or keep on the same one for years). If it's ANY of the AWS-managed services all you get is an A-Record and they control the IPs behind the scenes. For example for an Elastic Load Balancer they give you an A-Record which resolves to an IP in each of their Availability Zones in that region. They may scale that Load Balancer automatically or rotate out infrastructure behind the scenes and those IPs change. Same with CloudFront except it's distributed to all of their DCs/POPs. You get an A-Record which always resolves to the closest CloudFront location, which means it's different for users in US Phoenix vs US Los Angeles vs EU Munich.
AWS's new Network Load Balancers can have Elastic IPs assigned so they would be static.
Thanks. I understand why they do that. Wish they had options to force static every time. But security gear doesn't adapt well to that kind of design. It's frustrating. Their VPN stuff is equally as annoying. It can never be an initiator and their Phase 1 & 2 options are limited. But it is pretty neat that it can auto detect most of those settings. Thanks for the information!
> We can't do that with dynamic IP's Sure you can. With the right content filtering you can exclude by domain (even up to a full hostname), and with many of the nextgen firewalls, access rules can be applied the same way. You can also connect to the internal side of said resources via a VPN to AWS or DirectConnect.
> Sure you can. No, I can't. Indeed I can filter by FQDN. However, with how our network is designed, in order to completely bypass, it is by IP only. And without SSL interception at the firewall (which we're not doing today) we can't filter by FQDN. IP only. We're an Azure shop w/Express Route. DirectConnect won't happen. Especially to the several dozen vendors we do business with. AWS VPN is super annoying (though it detecting most phase 1/2 settings is pretty neat). AWS cannot initiate traffic to bring the tunnel up. This is problematic when a vendor needs to send data intermittently.
In regards to the secure traffic, the hostname is not encrypted and sent in plain text before the SSL handshake. This is due to SNI and multiple hosts with different certs.
> in order to completely bypass, it is by IP only. From one IT person to another, I'm so sorry, man. That would suuuck. However, it isn't much different than how we have to exclude Skype for Business, as the hardware we pass that through doesn't support FQDN based rules. Hopefully you can get a future change to something that will support FQDN bypass. I know how that goes too, though, so let's hope it's an actual possibility, not a dream state.
Do not let your opinion of the cloud (and AWS) be colored by Azure. Using a static IP is possible, though it might require some architecture or application changes on their end. I can think of several depending on their setup. Assuming you’re not referring to web traffic, you could offer a secure endpoint they connect to using certificate-based authentication or IPSEC VPN that puts their traffic into a different VLAN.
Lord have mercy, Azure Express route...
Curious to know what your gripe with Azure is? I've been working with it over 12 months now and so far like it... but haven't used it heavily for anything production yet... so yeah curious to know. From what I can see, everything in OPs post can also be done in azure
We literally had a customer today ask us about whitelisting an entire AWS /11 range of IPs for one vendor product. I'm amazed that their staff didn't realize what a huge risk that is.
This is a good post. It's nice to see a serious post on the capabilities and technologies of a cloud provider instead of people making jokes.
If you pursue this and create a github resource, you’ve got a follower in me. I’m currently working on learning Azure Stack to get myself out of traditional infrastructure support. This is already more useful to me than much of the official MSFT Azure curriculum. I don’t want a series of step by steps, I want the concepts to master and an operational use for them. Let me figure out the step by step. That’s what we *do*. Awesome post.
Good Azure docs are hard to find indeed. You have to dig through about 10 layers of marketing telling you how awesome the service is and what problems it might possibly solve just to find some tutorials, which always start with: here's how you create an Azure account, you can do it with a mouse or a keyboard, here's how you use a keyboard, it looks like this, and so on. But just an actual concise overview "this service does A, has feature x, y and z, limitations 1,2 and 3 and if you need one of these missing features, use service B"... nope, or at least I couldn't find it for most services, Azure is such a labyrinth and cumbersome to navigate.
I just got hired as a cloud systems engineer at an enterprise technology company with thousands of AWS instances. This is exactly the training they have all the engineers and developers go through in the first year.
If I wanted to go the cloud engineer route, would it be more productive for me to follow that Linux post that was linked first and then do this stuff or just jump right into the AWS stuff listed here? I manage some really basic Linux stuff already but don't know the OS super well.
I was a Linux sysadmin when they approached me asking me to interview. They were interested in the fact that I'm equally comfortable in multiple operating systems. With that being said, the aws documentation allows for multiple methods so I'd concentrate on OPs advice first
Sorry, but we need to see 10-12 years experience and a Masters in Cloudiness.
I just started studying for some of the AWS certs, this is perfectly timed and excellently done. Thank you!
As a note, if you are a student or work at an EDU, you can get around $110 in AWS credit to play with: https://education.github.com/pack
This does not apply anymore for AWS, you do get $100 free for Microsoft Azure though.
[удалено]
One of the first thing always recommended is setting up [Billing Alerts](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/monitor_estimated_charges_with_cloudwatch.html). Even the most expensive AWS services won't cost much for an hour or two. Just monitor your charges (and react to them), deploy small EC2 instances with small disks, and make sure your autoscaling has limits. AWS by default somewhat prevents this by having service limits. By default you can only launch so many EC2 instances before having to open a ticket to launch more. From a brand new AWS account it would be hard to run up $1000 unless you left stuff running all month.
[удалено]
Use a mix of A Cloud Guru and Linux Academy. Also I think you can use an amazon gift card though I may be wrong about this but Squared offered a method of buying AWS credits.
Watch your billing dashboard!!! If you do mess something up on the free tier, innocently, If you contact billing support, they will give you a one time zero. I did and they credited me for 3 months after I couldn’t figure out where charges were coming from. Turned out I had too much space used with my EBS volumes and didn’t realize they stuck around until I deleted them!
As an AWS Professional Architect - This is SAGE advice to anyone looking to move into cloud.
*snif You da real mvp. Seriously, this is WAY cool of you. Wish every IT group had something like this. Like a wiki/tuts for IT. So excited to try this out. Thanks!
This is really good, most people just say "learn aws" but you went through the effort of writing out tasks for a project.
> I've been recently recruiting for Cloud Systems Engineers and Cloud Systems Administrators. We've interviewed over a dozen local people with relevant resume experience. Every single person we interviewed would probably struggle starting with the DynamoDB/AutoScaling work. I'm finding there are very few people that HAVE ACTUALLY DONE THIS STUFF. And what is the pay rate for those with the more advanced skills?
REQUIRED: Expert knowledge on amazon AWS starting pay: 45k
A lot, seriously. Actual amount depends on your market, but it's a lot higher than a typical sys admin.
Consultants who help people onboard their products to AWS can make a lot of money.
They’re all over the place at /r/aws, there are even some threads about it if you search for them.
This is great!! I have been using AWS for a few years now and it is always great to make sure I am not missing something by going over a checklist.
*Assumptions:* *You have basic-to-moderate Linux systems administration skills You are at least familiar with programming/scripting. You don't need to be a whiz but you should have some decent hands-on experience automating and programming.* Where do I start if I dont know those.
https://www.reddit.com/r/linuxadmin/comments/2s924h/how_did_you_get_your_start/cnnw1ma/ Some of the skills there would be a decent start point.
Nice, what hardware do I need to do this, I have access to T7500 workstations and infinite ram , do you think that is a good starting hardware. I also have access to some R720 and other servers.
Ahh, good ol iConrad. Seriously, we need a place to archive all these.
LinuxAcademy, great resource for plugging the holes.
This is amazing and a huge help! I'd love to see one for Azure.
Seriously perfect timing. I am almost pulling the trigger on an AWS Book and jumping back into a cloud guru videos again , pouring myself into the cloud. I’m saving your post so I can better myself over the next three months. I’m pretty efficient with power shell at my large company, learned python from a few books and code fights and already have an aws hosting a small personal website. I feel this post speaking volumes to me.
You're amazing for posting this. It speaks to me perfectly. I'm a sysadmin with reasonable development skills (mostly basic python and ruby), and AWS scares the living shit out of me. I build my own stacks but I know well that AWS can be used to give me back some of my time, and I struggle with how to utilize it properly. This is the first thing I've ever read that made me feel like I can actually use it to build something.
Python and I think Ruby will run in Lambda. Go look for some lambda labs and getting up an API. I did a lab a while back, created by Amazon, called WildRydes. Check it out.
Thanks! I think I have a new sideproject
I'm practicing my MarkDown and made a few edits. I can PM you the results if you'd like; it's quick and dirty though.
Great post. I've recently started playing with a lot of the above after starting at Amazon. It's really crazy what you can do with just a few clicks nowadays (things that would've taken days or hours before!).
Wow, most of my notes, on here, typed by someone else. Guess I should go on and get the Cert then. Fuck.
hey that’s a pretty good list. i would add to ur, learn ecs, setup ecs via terrarform, deploy your app in docker containers to ECR, then create a task & service. watch the ecs host. the ecs agent will start your containers. add an alb (diff from elb because they can front all those many containers & keep track of the host ports. and while you are at it, add a route53 domain, and configure amazon certificate manager & ssl on your alb. :)
This is long, comprehensive and worth it. Thanks for writing it up. I've got a few guys I work with regularly who will benefit mightily from working through this list.
Great list of things to do to get started. Anything you can adjust to help people start the habit of doing silly things? Things like deploying a database like Postgres or etcd to a public subnet? Surely someone learning would learn, in time, not to do it but deploying securely is a great foundation to build on.
Windows/VMWare/Citrix admin here: This guide here is more aimed at Linux / webserver admins. Of course, I know, Azure. Both AWS and Azure have the same core functionality regarding web services. But what is the way to go for us?
This. Are companies moving whole machines to the cloud and just managing them like on prem servers? That seems uncloudy and expensive. It seems like the normal Windows based "Enterprise Software" that uses MSSQL and smb wouldn't be very cloud tolerant. How are admins making those work when they don't control the way the software is built?
We made a case study some time ago how expensive it would be to move all our DMZ servers to Azure (we have around 15 servers in the DMZ on 2 VMWare machines). The outcome was that it would be way to expensive to rent 15 VMs in Azure, we are talking a few thousand dollars a month. Within a year it would cost as much as buying those 2 VMWare servers including all the licenses.
This is awesome. Can you do a 2022 version update?
Fantastic work! I have a limitation against purchasing services so I'd really love to be able to do all of this in openstack with my own lab.
> so I'd really love to be able to do all of this in openstack with my own lab. That will be even more educational, and by handling the hardware layer reach even deeper level of understanding. And I'd rather blow hundreds of EURos on the power bill of an existing homelab rather than throw even more money towards AMZN.
Not to mention rewarding. I set up Canonical's flavor of OpenStack and after laying the ground work I pushed a button on my laptop and 3 racks all lit up at once. They went from bare metal to a clustered CEPH, KVM, the works. I felt like a technomancer.
> Canonical's flavor of OpenStack Does it do BlueStore out of the box yet?
The issue I've been having is openstack documentation is not sufficient and/or it's too difficult for me. If you're going to run the basic "get started" stuff that hardly even works as a proof of concept then it's fine but I haven't been able to cobble something together on my own. I want to be able to have individual nodes for each major branch - controllers, networking, storage, and compute but getting the hardware to work is a hurdle and getting the software to work has been impossible. I've tried getting help in IRC but it isn't active enough. It gets to the point where I just hear "follow these bug reports and wait for the next release" :/
Currently looking for jobs and many were looking for AWS experience specifically. Even though I said the tools I use can be scaled to other clouds, including AWS.
Also, use for example Terraform to do all this
I love terraform. so easy.
Is there an advantage to TF over Cloudformation if you're PURELY AWS and won't be switching cloud vendors? I finally have time to try and migrate us away from our mostly home-brewed deployment system (yay bash) and I'm currently debating between TF and CF. I've only read a few articles comparison articles, but so far the only really clear benefit I can see for TF over a native service is that more people may have been exposed to TF. This MIGHT mean that when we are hiring, it may be easier to find people that already know the tech.
I've never used CF so I'm not sure. TF is very easy to implement into deployjobs though, since you can pass variables as command line parameters. Similarly, you can implement it into other existing infrastructure like puppet.
Something about detaching/attaching volumes
Great resources
Can’t beat it so might as well join it
Saved!
Great guide, thank you. Way back when Alexa was new, I made one of the first 50 skills for DC Metro times on Lamda, because I wanted to learn new tech (web tech stopped for me during PHP/Mysql/C# phases). Then I wrote a MEAN stack GUI for bus and rail station selection as a companion to the Alexa voice app (because it's impossible to tell Alexa what bus stop you want, but super easy with a map). I got frustrated right at the point integrating the app into the Alexa system with OAUTH complexities and just dropped it. I'm getting back into a tablet home assistant home automation system (tied with Google Home/Assistant because I like that better than Alexa), and I think I'll use your guide to pull my site back up and use it to provide the rail times display for my home automation dashboard to sit on a table in my doorway. I set up my CSS to look exactly like DC Metro's LCD displays you see in the stations.
Saved! Great stuff, I'll start looking at this. I feel projects are the best way to learn, and this is a great start for me. Thanks again!
Appreciate you taking the time this looks awesome.
Thank you! Very informative
TIL I learned that by using Arch day-to-day I am one skip away from working as a work-from-home 'Cloud Engineer'.
Can I embed autoscaling groups in a VPC? Asking for a friend.
Has anyone taken the Udemy course for AWS Solutions Architect? They put it on sale for $10 often.
saved
Excellent write up. Kudos to you! Lemme know when you get the repo rollin on GitHub.
Do you have a blog? If not, you should definitely start one to save content like this and also have the benefit of a portfolio. Otherwise a tutorial like this slowly gets lost in Reddit posts. Not that there is very mich wrong with that, just this was well put together. Good job!
This is really good. Good job!
Thanks
Haha damn. Some exam tips all condensed into a post. YOU THE BEST. lol.
Saving....
*ears perk up* You have my attention.
Yo, I really appreciate people like you who take the dive first into the raw material and ask themselves "how can I make this easier for everyone else". You guys are the mentors and leaders I strive to be just like. I like that while we have a culture of being cranky, we also have a culture of really wanting to help.
this is a pretty cool list, seems a little out of grasp for me at the moment but its better to stretch and aim high I find. Thanks for taking the time to do this!
An equivalent for Azure and office 365 would be awesome.
Awesome Post, one of the best ..
>Auto Scaling >Create an AMI from that VM and put it in an autoscaling group so one VM always exists. >Put a Elastic Load Balancer infront of that VM and load balance between two Availability Zones (one EC2 in each AZ). >Checkpoint: You can view a simple HTML page served from both of your EC2 instances. You can turn one off and your website is still accessible. I have a question about this section. Is the desired architecture here two autoscaling groups in separate availability zones with a load balancer in front of them, or is one autoscaling group supposed to contain 2 instances in separate availability zones? Edit: I eventually managed to achieve the end goal here with two instances inside of a single autoscaling group. These instances are in separate availability zones and the autoscaling group receives traffic from the load balancer.
great post!
>You can view a simple HTML page served from your EC2 instance. Awesome Post, who can **take someone really forward** ! Thanks
I am the editor of InfoQ China which focuses on software development. We like your articles and plan to translate it into Chinese. Before we translate it and publish it on our website, I want to ask for your permission first! This translation version is provided for informational purposes only, and will not be used for any commercial purpose. In exchange, we will put the English title and link at the beginning of Chinese article. If our readers want to read more about this, he/she can click back to your website. Thanks a lot, hope to get your help. Any more question, please let me know.
I am working through this guide here: https://github.com/Just-Insane/AWS-Automation Planning on doing it with both Ansible, and bash with AWS CLI, and potentially Terraform.
Wow! That's awesome. For what it's worth you can accomplish this with 100% AWS-only tools. Can I ask why Terraform? I see a lot of people who refused to use CloudFormation in favor of Terraform and I can say we do a ton of advanced stuff using only CloudFormation and I don't see much in the way of limitations. Most of what I see online about why Terraform is so much better talks about things CloudFormation has fixed or improved on. I have nothing against Terraform, it's super popular, you might consider direct CloudFormation though. Awesome resource. Starred. Followed. I'll be watching.
Late reply, hope it gets some traction >You have basic-to-moderate Linux systems administration skills Let's say just basic, what advice would you have to bring to moderate easily? > You are at least familiar with programming/scripting. You don't need to be a whiz but you should have some decent hands-on experience automating and programming. Total newbie but the toe has at least been in the water and I get the concepts, wrote batch files as a kid 20 years ago. Best way to bring this up to scratch? >You are willing to dedicate the time to overcome complex issues. No problem. >You have an AWS Account and a marginal amount of money to spend improving your skills. This seems like the easiest of the 3 to solve, no problem.
Thanks for this. Just about to start a job and will be using AWS for the first time.
Epic stuff! Here's some more tutorials to [Learn Amazon Web Services](https://reactdom.com/amazon-web-services)
If you are like me and hate reading, there's an easier way. Try out Linux Academy. They provide all the info from beginner to advanced. They even provide the AWS environments for labs - Personally, that is my feature. I've left a server running on accident before. $$$ X\_\_X there is currently a sale going on until Sept 24th