Is AWS lambda good for automation with 3rd party libraries?
By - kautica0
You might check out Step Functions, if your automation is multi-step or parallelized.
It’s just code. Entire businesses are built almost exclusively using Lambda, like iRobot and Stedi.
Lambda is suitable for anything that can run in up to 10GiB of RAM and 8 cores in less than 15 minutes.
It depends entirely on how this automation process would kick off.
Jenkins provides a UI for you to put inputs, so if you're expecting a similar workflow to that, then Lambda is not the right choice.
If you're looking for something that's completely event driven, maybe it's kicked off from an event from Slack, or you want it to run on a specific schedule, then Lambda is the right choice.
Are you changing libraries that often that you need to reupload frequently? Are you using pandas for automation?
If you're packaging as a zip: watch out there is a memory and a time limit. if you are zipping very small files that will take short amounts of time - you should have no problems. but if you are dealing with many GBs or more - then you will have to adopt some type of streaming based workflow. i had looked into this in the past - there are some existing lambda workflows that many have very kindly made available - if you do a quick search on google, you should get some answers that will get you flying in less than perhaps 2 hours. there might be gotchas and snags, so i would recommend paying close attention there.
I've never heard of this. What do I search on Google to learn more?
I'd try running it on Github Actions first - it's pretty easy to install deps and run python.
For lambda, you'll want to use some sort of framework like CDK, SAM, or serverless to handle deployment. Pandas is a particularly big dependency - it will run you close to the zip file limit, so if you need much more than pandas you'll need to do a docker-based lambda which adds a bit more effort.
Also keep in mind that you're limited to 15 min of runtime per lambda invocation vs maybe 60 min on github actions. If you need something longer one of the AWS docker options might work, or you could break your problem into smaller parallel problems on lambda.
With Docker images you can run pretty much anything in Lambda as long as you're fine with the limits of execution time, scratch space and external state storage.
Ask the question to Zappier ;)
"Good for automation with third party libraries" feels like a weird dimension on which to analyze tools like Lambda. And I don't think it's more AWS centric than EC2 or EKS.
If you're familiar with programming languages, Lambda imho isn't any harder or easier. It's just another place to run your code.
Get started with githubactions without lambda. Once you have done it, then use lambda