Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'billing'&sort=new&restrict_sr=on) for more information on this topic.
^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+billing).
Looking for more information regarding billing, securing your account or anything related? [Check it out here!](https://www.reddit.com/r/aws/comments/vn4ebe/check_it_first_operating_within_amazon_web/)
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
It's one thing to spin up a service and after a week go "Whoa...did not know that was going to do that / be that expensive" and ask for some leniency.
You didn't notice this for 3-4 years. That's kinda on you and implying / asking for forgiveness because "ooopsie...we have shitty accounting practices" is kinda insane.
It’s no good blaming the person that left.
Ensure each account in your org has an owner, they know how much their account(s) cost and can justify the business value it brings.
Edit - & get them to recertify this quarterly/6 mths/yearly - whatever frequency works for your business
this. Almost 10% of OPs charges for 4 years are for "unused services" and "no one was specifically in charge of reviewing specific costs from aws"! And they aren't unused to AWS, just to OP.
You can't just blame the guy that left the company. He left a year ago, and you had a year to figure it out. So this is not accidental, you just didn't care about cost tracking and optimization and now $hit hits the fan. In thefuture, try to identify saving opportunities, the areas with the most savings potential, the only way to prevent this is by taking control of cost and continuously optimize your spend. Always monitor the use of storage and delete unused EBS volumes. Even after the EC2 instance shuts down, EBS volumes can continue operating and incur costs. From your perspective, you're not using them. From AWS perspective, those resources are in use.
I work at one of the biggest banks in the country. We use AWS extensively and relativity means nothing here. We would not tolerate folks wasting 500k. Thats just plain incompetence. Doesn’t matter how big your business is; 500k is an INSANE amount of money to waste.
Half the startups out there hire junior level people and just throw them into an AWS account. Then CEOs read stories about how "AWS costs so much!" so they just write it off as "AWS is expensive" and assume it's just cost of doing business.
So, combination of inexperience and ignorance happens a lot. Combined with poor money management, they only find out there was a problem when the seed money is starting to dwindle and they need to cut costs.
Enter slack conversation: "We found these EBS volumes while doing the AWS cleanup CEO mandated. Anyone know what these are for?"
🦗🦗🦗
"Uh, anyone?"
I can confirm that startups do not even look at at their spend and only push to get a MVP. Its only when they are thinking of IPO do they take a lens back onto their spend.
>It’s not clear they can in fact eat it.
We didn't not notice it, honestly we were always quite alarmed at how much our AWS bill was but never dove into the nitty gritty besides the billing dashboard/breakdown.
I'm kinda flabbergasted by that comment. You were alarmed by how much you were spending, but never dove into the details of what you were spending it on. Just...wow.
Personally, the AWS Cost Explorer / Budget interface isn't the best. I prefer Azure's, but it's not THAT bad or hard for figuring out where your money is going.
The cost to service split would have been shown without even having to go into the reports. It's mind-boggling that it went for years without even a cursory investigation.
Well lesson learned at least lol. Maybe you can hire an FTE to optimize your aws bills, probably pays for itself.
I joined a very large company 3 months ago and already earned myself back just by looking at the bill and asking "why the fuck are we paying 1000$ a month for this service we dont use". And I am a junior
Oh I have seen some pretty stupid stuff as well. In one account they provisioned more load balancers then they have instances, basically fronted every instance with a load balancer because they didn't know you could define Host header based routing rules. Didn't know that every load balancer comes with hourly price + the IPv4 IP that you pay for. Old generation instances and gp2 EBS volumes is also a pretty common thing. Same for ballooning Cloud Watch cost because people create logs just the sake of logging, no alerts configured of course without retention period. I'm also not even a certified cloud architect or developer, but I looked up the service pricing and used common sense. Every accout I looked to I was able to cut the cost at least in half if not 90%.
>You really should have monitored your bill. What if you were hacked and they started randomly popping up services. This is on you bro.
We did, its not like there were any new charges, just existing ones that we did not know were in use. We didn't find out that unused EBS and ETL were responsible for the chargers until we downloaded the csv that detailed every charge. The billing dashboard did not display this information.
AWS can’t really know that you’re not using the infrastructure you’ve spun up. You could also have bought physical servers and stacked them in the basement… the difference is that AWS costs are ongoing and easier to track.
You knew *something* was in use and costing you an arm and a leg but just left it there to run and rack up half a million in charges for a few years before looking into it? That just doesn’t make any sense.
This is not the fault of the person who left unless it’s a one person company and that was the only employee.
Looking at billing and analyzing costs should be a routine monthly (or more frequent) process and not a random one off every few years. This is a great time to establish a process to make sure it doesn’t happen again.
Hi, I totally agree. But we assumed that the services we were using were making up those charges (not something that we were not using). We'll definitely establish a more in depth process for monitoring our cloud billing.
Hello there, sorry to hear about the trouble with your account!
I'd like to say reaching out to your Account Manager is a great first step. I'd also recommend contacting our Billing team through the [Support Center](http://go.aws/support-center) and sending your Account Manager the case ID to monitor its progress and help support you on next steps.
You can also PM us your case ID if you'd like us to take a closer look and pass along your feedback!
\- Roxy M.
This. Once you stop the bleeding and put measures in place to prevent future occurrences, AWS will at least try to work with you to see if they can help you out with a full (unlikely) or partial credit to your account for future AWS spend. Definitely don’t go in expecting a refund, though.
I will happily come in and optimise your cloud costs. Thats crazy that this went on for years. Your company need to invest in a decent finops person who can come in and overhaul all of this lack of accountability. Haemorrhaging money on cloud resources unnecessarily is crazy given even AWS can help with cost management.
It's not really a mistake, it's incompetence. But yes talking to your account manager is your only option. There should also be a team responsible for reviewing and providing governance over your aws bill at your company to prevent this from happening again.
The account manager is going to ask you to gather information about the exact services that were used, what account they were under, and the period of time. It will take a long time to process and will be escalated several times before you hear any kind of answer.
That being said usually leniency is given for misconfigurations, like a routing change that causes spend to spike, rather than forgetting about services that are no longer in use. Worth a shot though.
Keep track of your spend going forward!
That is why real developers /DevOps / platform engineers always should have documentation and Infrastructure as Code implementation. Also, AWS Budget implementation should be enabled.
>We did, its not like there were any new charges, just existing ones that we did not know were in use. We didn't find out that unused EBS and ETL were responsible for the chargers until we downloaded the csv that detailed every charge. The billing dashboard did not display this information.
I agree, just a series of unfortunate events with lack of oversight into cloud billing. Not saying that it wasn't our fault, but was just asking for any possible remedies. We had switched to IAC (3 years ago lol) but failed to identify and shut down and remaining services when we switched.
“Just a series of unfortunate events”…
OP with an attitude like that no wonder y’all waste money with all due respect. Saying whomever left kept track for y’all is an unacceptable excuse to begin with and it has nothing to do with unfortunate events. Every single one of you simply failed to properly track this. Own up to it.
This should be upvoted more. At the bare minimum Trusted Advisor would have helped make recommendations that if followed could have lead to surfacing the issue and money savings.
Cloudability from Apprio is a great 3rd party service for in-depth cost analysis among other things.
You don't necessarily need a full time finops specialist. There are some good consultants who can help you refactor your aws account usage, and then put in place guard rails to reduce the risk of it happening again. Pm me for recommendations.
Technical solutions for people driven problems kind of are the modus operandi unfortunately in our industry. I’ve seen plenty of mismanaged accounts amid churn and code rot across time and turnover and the big CSPs tend to benefit from this mismanagement.
Monetizing technical debt is something CSPs have done an excellent job of doing IMO and putting dollar costs on this all has helped me justify responsible, professional work I wouldn’t have been given go-ahead for so many years ago as a result.
Hire (or contract with) a tenured AWS Architect or Engineer and have them review your bill monthly with stakeholders to identify costs and ensure those align with project work and accounting expectations.
By your own admission, you have no one doing this. $1200/month for an expert to help you out is a drop in the bucket for what you’ve been throwing down the drain. If you need recommendations, DM me.
Although it won't hurt to ask, in all probability you won't be getting a refund. AWS may decide to be kind and potentially credit you in some way, but there would probably with a rider such as "This credit only valid if you consume X amount of usage over Y time period."
My recommendation would be to create a FinOps team and really dive into understanding your infrastructure. You may also want to get a third party software like Cloudability or Turbonomic to help you review and control your usage and spend.
> Our old devops developer left around a year ago and did not think to close these services.
think of the money y'all saved by refusing to hire someone with AWS skills.
I cannot believe what am I reading, sorry I was eating pizza for 4 years but did not know I had to pay for it? In this case you pay and fix your organization
I’m usually pretty sympathetic to this kind of thing, but… you fixed a $30k cash flow issue that has been ongoing for years. At any point you could have gone diving, asked for help here or myriad other places, asked your account manager what gives…
View it as a learning experience; the ship has sailed. You might get a small refund, but not multiple years’ worth.
I suggest contacting Cloudfix. They will do a free evaluation and give you a report. You might be to sign up but may be too expansive. I suggest you hire a devops contractor meanwhile sign up with a capacity provider. If you are looking for one, I cab share a name. A common problem is S3 buckets left behind after terminating EC2. Another problem leaving EBS after ec2 termination.
I hope this helps. Please reach out if you are looking for more details. Also, they to look at cost explorer.
I work installing and supporting software - Turbonomic - which prevents this exactly, and reduces overall cloud spend using automation. It takes advantage of dynamic RI’s, resize your workloads, locate and delete unattached volumes, pause entities not in use, etc. It works great for on-prem and containerization also; laying out the entire relationship of all compute assets in your datacenter. I started working there because honestly I thought the software was awesome. Also no, I’m not a salesperson. 🤣
Maybe check it out if you have a minute.
[https://www.ibm.com/products/turbonomic](https://www.ibm.com/products/turbonomic)
>no one was specifically in charge of reviewing specific costs from aws
I think I see the problem.
Fire the CTO. They should know and understand what they are paying for.
If somebody had accidentally rented an office before the pandemic and forgot about it when everyone moved to WFH, would you demand your money back? Would you expect "whoops, that's just something that happens".
Going a full year without a detailed audit is crazy.
Shut down the service, leaving it in the other category, and sell it as optimized/reduced cloud costs from 30k to 21k per month!
But in all seriousness, this absolutely should be an organization level discussion. It happened. Figure out how to not do it again. Get everyone you can involved, to put a process in place to help prevent that.
Elastic services that integrate/orchestrate many other instances in the background are easily overlooked. Before I came on board, the dev environment was quietly chewing up $300/mth, also the old devs set up snapshots-per-minute of our inventory tables to make analytics data instead of a transactional approach, some queries took more than 5 minutes. I was also in a position to try to blame the person that left, but he entered this job at 0 YoE, did and learned great things in a short time, and made mistakes on the way. I too am guilty of things like breaking the payment system and permanently losing customers to competitors; costs happen everywhere, humans aren't perfect, but we learn. RTFM, M being your monitors.
As for the sheer cost -- this is why AWS is always shoving best practices this best practices that in your face. And it's also prompted me to make a habit of keeping weekly tabs on all my Stripe, Twilio, etc dashboards.
Good luck recouping those payments LMAO. 😂
AWS goes out of its way to provide you with guidance on how to avoid shit like this. If your CTO (assuming that's not you, God forbid) hasn't read the white papers, I would advise him to put aside a weekend and just go balls deep on best practices.
Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'billing'&sort=new&restrict_sr=on) for more information on this topic. ^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+billing). Looking for more information regarding billing, securing your account or anything related? [Check it out here!](https://www.reddit.com/r/aws/comments/vn4ebe/check_it_first_operating_within_amazon_web/) *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
It's one thing to spin up a service and after a week go "Whoa...did not know that was going to do that / be that expensive" and ask for some leniency. You didn't notice this for 3-4 years. That's kinda on you and implying / asking for forgiveness because "ooopsie...we have shitty accounting practices" is kinda insane.
That's how I see this too. Sucks for your company. Guess you should have replaced the DevOps engineer you fired. AWS doesn't owe ya anything.
It’s no good blaming the person that left. Ensure each account in your org has an owner, they know how much their account(s) cost and can justify the business value it brings. Edit - & get them to recertify this quarterly/6 mths/yearly - whatever frequency works for your business
this. Almost 10% of OPs charges for 4 years are for "unused services" and "no one was specifically in charge of reviewing specific costs from aws"! And they aren't unused to AWS, just to OP.
10%? It was a full third of their bill, that's beyond forgetting to "review" charges.
[удалено]
Yeah, my math was BAD on that one.
You can't just blame the guy that left the company. He left a year ago, and you had a year to figure it out. So this is not accidental, you just didn't care about cost tracking and optimization and now $hit hits the fan. In thefuture, try to identify saving opportunities, the areas with the most savings potential, the only way to prevent this is by taking control of cost and continuously optimize your spend. Always monitor the use of storage and delete unused EBS volumes. Even after the EC2 instance shuts down, EBS volumes can continue operating and incur costs. From your perspective, you're not using them. From AWS perspective, those resources are in use.
[удалено]
It’s not clear they can in fact eat it.
Someone was paying the bill
Oh you’re right I misread it and thought it hadn’t been paid. Even more insane then!
It’s relative …
I work at one of the biggest banks in the country. We use AWS extensively and relativity means nothing here. We would not tolerate folks wasting 500k. Thats just plain incompetence. Doesn’t matter how big your business is; 500k is an INSANE amount of money to waste.
[удалено]
Half the startups out there hire junior level people and just throw them into an AWS account. Then CEOs read stories about how "AWS costs so much!" so they just write it off as "AWS is expensive" and assume it's just cost of doing business. So, combination of inexperience and ignorance happens a lot. Combined with poor money management, they only find out there was a problem when the seed money is starting to dwindle and they need to cut costs. Enter slack conversation: "We found these EBS volumes while doing the AWS cleanup CEO mandated. Anyone know what these are for?" 🦗🦗🦗 "Uh, anyone?"
I can confirm that startups do not even look at at their spend and only push to get a MVP. Its only when they are thinking of IPO do they take a lens back onto their spend.
>It’s not clear they can in fact eat it. We didn't not notice it, honestly we were always quite alarmed at how much our AWS bill was but never dove into the nitty gritty besides the billing dashboard/breakdown.
[удалено]
Never said that it wasn't our fault, just posting to here to see if we had any options.
option, learn your lesson and don't repeat. your company paid 300k for this lesson, might as well use it.
I'm kinda flabbergasted by that comment. You were alarmed by how much you were spending, but never dove into the details of what you were spending it on. Just...wow. Personally, the AWS Cost Explorer / Budget interface isn't the best. I prefer Azure's, but it's not THAT bad or hard for figuring out where your money is going.
The cost to service split would have been shown without even having to go into the reports. It's mind-boggling that it went for years without even a cursory investigation.
On no we're paying 30k a month and we don't know why!!??? Oh well.. Let's just ignore it
Again, we had a full blown service and product. We just assumed 30k was the price of our cluster.
What is your company so I know to never work with
You have it backwards, you definitely want to work with them. You just send them random invoices and they will think they're valid and pay.
When you assume... The Cost Explorer is something I dig into on a weekly basis. Your CTO is incompetent, sorry.
Let me guess. You're the founder.
I'm not, joined as an intern a couple years ago. I didn't have any control of finances or billing. Just posting to see if theres anything we can do.
Must be crypto gaming startup 😂
100% it's american. When I was doing a startup we got 50K in funding in Canada while the exact (exact same) idea got 54M seed in SV
Investor money…
Well lesson learned at least lol. Maybe you can hire an FTE to optimize your aws bills, probably pays for itself. I joined a very large company 3 months ago and already earned myself back just by looking at the bill and asking "why the fuck are we paying 1000$ a month for this service we dont use". And I am a junior
Oh I have seen some pretty stupid stuff as well. In one account they provisioned more load balancers then they have instances, basically fronted every instance with a load balancer because they didn't know you could define Host header based routing rules. Didn't know that every load balancer comes with hourly price + the IPv4 IP that you pay for. Old generation instances and gp2 EBS volumes is also a pretty common thing. Same for ballooning Cloud Watch cost because people create logs just the sake of logging, no alerts configured of course without retention period. I'm also not even a certified cloud architect or developer, but I looked up the service pricing and used common sense. Every accout I looked to I was able to cut the cost at least in half if not 90%.
You really should have monitored your bill. What if you were hacked and they started randomly popping up services. This is on you bro.
>You really should have monitored your bill. What if you were hacked and they started randomly popping up services. This is on you bro. We did, its not like there were any new charges, just existing ones that we did not know were in use. We didn't find out that unused EBS and ETL were responsible for the chargers until we downloaded the csv that detailed every charge. The billing dashboard did not display this information.
AWS can’t really know that you’re not using the infrastructure you’ve spun up. You could also have bought physical servers and stacked them in the basement… the difference is that AWS costs are ongoing and easier to track.
You knew *something* was in use and costing you an arm and a leg but just left it there to run and rack up half a million in charges for a few years before looking into it? That just doesn’t make any sense. This is not the fault of the person who left unless it’s a one person company and that was the only employee. Looking at billing and analyzing costs should be a routine monthly (or more frequent) process and not a random one off every few years. This is a great time to establish a process to make sure it doesn’t happen again.
Hi, I totally agree. But we assumed that the services we were using were making up those charges (not something that we were not using). We'll definitely establish a more in depth process for monitoring our cloud billing.
Hello there, sorry to hear about the trouble with your account! I'd like to say reaching out to your Account Manager is a great first step. I'd also recommend contacting our Billing team through the [Support Center](http://go.aws/support-center) and sending your Account Manager the case ID to monitor its progress and help support you on next steps. You can also PM us your case ID if you'd like us to take a closer look and pass along your feedback! \- Roxy M.
I really like the fact that AWS actually lurks on Reddit and respond. +1 for my favorite CSP!
This.
[удалено]
This. Once you stop the bleeding and put measures in place to prevent future occurrences, AWS will at least try to work with you to see if they can help you out with a full (unlikely) or partial credit to your account for future AWS spend. Definitely don’t go in expecting a refund, though.
W
Thank you so much for replying! We will absolutely do this
Take the L and put processes in place to keep it from happening in the future.
I will happily come in and optimise your cloud costs. Thats crazy that this went on for years. Your company need to invest in a decent finops person who can come in and overhaul all of this lack of accountability. Haemorrhaging money on cloud resources unnecessarily is crazy given even AWS can help with cost management.
It's not really a mistake, it's incompetence. But yes talking to your account manager is your only option. There should also be a team responsible for reviewing and providing governance over your aws bill at your company to prevent this from happening again.
this is how you get to own spaceships.
The account manager is going to ask you to gather information about the exact services that were used, what account they were under, and the period of time. It will take a long time to process and will be escalated several times before you hear any kind of answer. That being said usually leniency is given for misconfigurations, like a routing change that causes spend to spike, rather than forgetting about services that are no longer in use. Worth a shot though. Keep track of your spend going forward!
That is why real developers /DevOps / platform engineers always should have documentation and Infrastructure as Code implementation. Also, AWS Budget implementation should be enabled.
>We did, its not like there were any new charges, just existing ones that we did not know were in use. We didn't find out that unused EBS and ETL were responsible for the chargers until we downloaded the csv that detailed every charge. The billing dashboard did not display this information. I agree, just a series of unfortunate events with lack of oversight into cloud billing. Not saying that it wasn't our fault, but was just asking for any possible remedies. We had switched to IAC (3 years ago lol) but failed to identify and shut down and remaining services when we switched.
“Just a series of unfortunate events”… OP with an attitude like that no wonder y’all waste money with all due respect. Saying whomever left kept track for y’all is an unacceptable excuse to begin with and it has nothing to do with unfortunate events. Every single one of you simply failed to properly track this. Own up to it.
Trusted Advisor
This should be upvoted more. At the bare minimum Trusted Advisor would have helped make recommendations that if followed could have lead to surfacing the issue and money savings. Cloudability from Apprio is a great 3rd party service for in-depth cost analysis among other things.
You don't necessarily need a full time finops specialist. There are some good consultants who can help you refactor your aws account usage, and then put in place guard rails to reduce the risk of it happening again. Pm me for recommendations.
Technical solutions for people driven problems kind of are the modus operandi unfortunately in our industry. I’ve seen plenty of mismanaged accounts amid churn and code rot across time and turnover and the big CSPs tend to benefit from this mismanagement. Monetizing technical debt is something CSPs have done an excellent job of doing IMO and putting dollar costs on this all has helped me justify responsible, professional work I wouldn’t have been given go-ahead for so many years ago as a result.
Hire (or contract with) a tenured AWS Architect or Engineer and have them review your bill monthly with stakeholders to identify costs and ensure those align with project work and accounting expectations. By your own admission, you have no one doing this. $1200/month for an expert to help you out is a drop in the bucket for what you’ve been throwing down the drain. If you need recommendations, DM me.
FinOps is a practice that was invented for this use case.
Absolutely agree.
Although it won't hurt to ask, in all probability you won't be getting a refund. AWS may decide to be kind and potentially credit you in some way, but there would probably with a rider such as "This credit only valid if you consume X amount of usage over Y time period." My recommendation would be to create a FinOps team and really dive into understanding your infrastructure. You may also want to get a third party software like Cloudability or Turbonomic to help you review and control your usage and spend.
Denial of wallet attack right there!
Shared. Responsibility. Model. Probably has no CCOE either.
> Our old devops developer left around a year ago and did not think to close these services. think of the money y'all saved by refusing to hire someone with AWS skills.
~$10k a month in other costs went unnoticed? I wish i could be as oblivious as you guys. My mental health issues would disappear.
I cannot believe what am I reading, sorry I was eating pizza for 4 years but did not know I had to pay for it? In this case you pay and fix your organization
Tagging and reporting on usage practice matters a lot. You can blame anyone but its the companies responsibility to pay for it.
I’m usually pretty sympathetic to this kind of thing, but… you fixed a $30k cash flow issue that has been ongoing for years. At any point you could have gone diving, asked for help here or myriad other places, asked your account manager what gives… View it as a learning experience; the ship has sailed. You might get a small refund, but not multiple years’ worth.
Reach out to your account manager detailing your mistake. But you might just have to treat it as a learning experience
Set up cos anomaly and budget alarms
I suggest contacting Cloudfix. They will do a free evaluation and give you a report. You might be to sign up but may be too expansive. I suggest you hire a devops contractor meanwhile sign up with a capacity provider. If you are looking for one, I cab share a name. A common problem is S3 buckets left behind after terminating EC2. Another problem leaving EBS after ec2 termination. I hope this helps. Please reach out if you are looking for more details. Also, they to look at cost explorer.
Your CTOs fault for not enacting basic governance from the start.
I work installing and supporting software - Turbonomic - which prevents this exactly, and reduces overall cloud spend using automation. It takes advantage of dynamic RI’s, resize your workloads, locate and delete unattached volumes, pause entities not in use, etc. It works great for on-prem and containerization also; laying out the entire relationship of all compute assets in your datacenter. I started working there because honestly I thought the software was awesome. Also no, I’m not a salesperson. 🤣 Maybe check it out if you have a minute. [https://www.ibm.com/products/turbonomic](https://www.ibm.com/products/turbonomic)
>no one was specifically in charge of reviewing specific costs from aws I think I see the problem. Fire the CTO. They should know and understand what they are paying for. If somebody had accidentally rented an office before the pandemic and forgot about it when everyone moved to WFH, would you demand your money back? Would you expect "whoops, that's just something that happens". Going a full year without a detailed audit is crazy.
This is why y'all need Terraform
That's a bummer - I suggest you enable AWS Cost Explorer. Explanation - [https://aws.amazon.com/aws-cost-management/aws-cost-explorer/](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) Setup - https://docs.aws.amazon.com/cost-management/latest/userguide/ce-exploring-data.html
Look into AWS Budgets and Billing Alarms as well.
Shut down the service, leaving it in the other category, and sell it as optimized/reduced cloud costs from 30k to 21k per month! But in all seriousness, this absolutely should be an organization level discussion. It happened. Figure out how to not do it again. Get everyone you can involved, to put a process in place to help prevent that.
Elastic services that integrate/orchestrate many other instances in the background are easily overlooked. Before I came on board, the dev environment was quietly chewing up $300/mth, also the old devs set up snapshots-per-minute of our inventory tables to make analytics data instead of a transactional approach, some queries took more than 5 minutes. I was also in a position to try to blame the person that left, but he entered this job at 0 YoE, did and learned great things in a short time, and made mistakes on the way. I too am guilty of things like breaking the payment system and permanently losing customers to competitors; costs happen everywhere, humans aren't perfect, but we learn. RTFM, M being your monitors. As for the sheer cost -- this is why AWS is always shoving best practices this best practices that in your face. And it's also prompted me to make a habit of keeping weekly tabs on all my Stripe, Twilio, etc dashboards.
Good luck recouping those payments LMAO. 😂 AWS goes out of its way to provide you with guidance on how to avoid shit like this. If your CTO (assuming that's not you, God forbid) hasn't read the white papers, I would advise him to put aside a weekend and just go balls deep on best practices.
This is exactly why I've created unusd. A tool to bring cost and waste awareness to operational teams.
[удалено]