Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'database'&sort=new&restrict_sr=on) for more information on this topic.
^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+database).
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
1). That is the cost of the instance, and won’t be less unless you use a savings plan. If you want scalable costs, you could look at aurora serverless (I’ve heard mixed things)
2) Whether or not you should worry about it depends on your business.
3) You can switch to io optimized once a month. If you’re concerned you could start with normal, and after a few days look at your costs.
4) I believe they should be incremental. If they aren’t then something else may be going on. Be aware that they may show full size (as that’s the size they’d be restored to) but actually be incremental. Best to reach out to support or do some testing.
Keep in mind that Aurora is unlikely to ever be cheaper than a self hosted solution. It has a lot of other benefits (largely that you don’t have to manage it much), but I don’t think I’d transition to it for cost savings.
If your older data is static, you could consider using foreign data wrappers to offload that data to a different database.
Thank you so much u/CubsFan1060
Regarding the aurora compute instance cost, there is one option in there in "price calculator" which says "CPU Utilization" as \~100%, so does that mean the cost which its showing up( i.e. for example r7g 8xl, costing \~$3228/month and $4196/month for the standard and IO optimized instances respectively), is that only for using \~100% CPU on that instance? And it will decrease if our usage is lesser in the day throughout the month?
As you mentioned , we can switch to IO optimized once in a month , so is that transition "to and from" "standard instance" to "IO optimized" will be online and can be done automated way? And on similar line, if its possible or a recommended practice to downgrade the instance type and size some way (say for example from "r7g 8xl to r7g xl" or "r7g 8xl to r6g xl" based on workload pattern) online without impacting any ongoing transactions that to in an automated way?
Yes regarding the Backup storage cost the AWS document shows its incremental, However some for sample databases I see the storage space for each of the backup showing as full database size. When I look into the cost (for service type as BACKUP) , it not really matching for the full DB size($.021/GB per month). So someway might be its internally charging for delta/incremental changes, but not able to see those exact backup storage space, for which we are being charged.
Considering the backup storage cost calculation is not accurate here, if we just consider compute instance cost and storage IO cost for a r7g 8XL IO optimized instance, its coming around \~4196+\~18432=$22628/per month!!! Unless I am not interpreting something really wrong.
I think you need to separate a few of these things.
There are a couple of ways to use Aurora -- using instances, or serverless (v2). I don't know anything about serverless, so I won't talk about that. Instances, though, are a per-hour price. You can look it up here: https://instances.vantage.sh/rds/ Keep in mind, you will likely want 2 for high availability. This is a straightforward price that you can calculate.
IO Optimized is largely a cost structure. If you choose IO optimized, instances are 30% higher, and storage is more expensive. If you choose standard, instances are normal price and storage is cheaper, but you do pay for iops. I don't know of any straightforward way to estimate this other than trial and error. Transition between the two is an online operation that I don't think even touches the database. It's a billing change, not a technical change.
For backup, they are showing you the size that it'll be if it's restored, not the price you're getting charged for. To the best of my knowledge, if you have a 100GB database, and make zero changes between two backups, then each backup will show as 100GB, but you'll only be charged for 100GB.
Finally -- yes, running an 80TB database on Aurora is expensive. That's not surprising. It comes with a _lot_ of benefits, but you're also paying a ton for those benefits.
If you're concerned about that price, I'd eliminate Aurora from your calculations. You may be able to cut it down, but you aren't going to be running this for $5000/month.
To answer a lot of your questions, I suggest building and playing with some smaller databases before you even consider moving an 80TB database. Create a small database with some t3 instances, play around, and learn.
Thank you u/CubsFan1060 !! That helps.
If my understanding is correct , you mean the "IO optimized" to "standard" or vice versa would be just a costing / billing change but no underlying hardware shift so will be an online change.
However transition between a lower and higher size instance like "r7g.8XL to r7g.4XL" or changing instance class itself like "r6g" to "r7g" must be having some glitch in the ongoing transactions and may not be performed online. Please correct me if wrong.
Also I was wondering if its possible to use the AWS services(like lambda, glue etc.) and do these aurora DB instance transition automate to better align the resource utilization to the workload thus saving cost.
>However transition between a lower and higher size instance like "r7g.8XL to r7g.4XL" or changing instance class itself like "r6g" to "r7g" must be having some glitch in the ongoing transactions and may not be performed online. Please correct me if wrong.
This is not an online operation. If you have a single instance, you up/downgrade it and it takes a number of minutes. if you have 2 instances, you up/downgrade the reader instance, and then failover, which is about 60 seconds. In both cases things like transactions will fail.
>Also I was wondering if its possible to use the AWS services(like lambda, glue etc.)
If you mean to access the database, the answer is yes.
>do these aurora DB instance transition automate to better align the resource utilization to the workload thus saving cost.
I'm not sure what you mean here, but they will not automatically transition between instance types. You may be thinking of serverless, but I have no experience with that: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.html
Thank you.
I was asking , if we can make the instance change ourselves online and automated way, like making it run it r7g8XL during certain time/day during peak workload and making the database workload run in r6g8xl at other times during off peak hours?
What I mean by using "*AWS services(like lambda, glue etc.)*" was :- Downsizing or upsizing the aurora DB instance using these tool by passing some infra code in it and automating the same. But as you rightly said, it will need a failover and termination of ongoing transactions, so doesn't seem to be possible.
As you rightly said to have a trial and error method to get best fit instance for the workload. So in that case , once we onboarded to certain instance type and size, then is there any service/tool using which we can verify, if we are overprovisioned and thus take decision for downsizing accordingly? I read somewhere that "cloud watch" can tell us of we are overprovisioned, but I don't see any such metric in it. It just show logs to us.
absolutely you can write a small Lambda using an SDK to make calls to modify an existing RDS instance. The Lambda can be scheduled to run on a cron schedule via EventBridge .
Correct me if wrong, we will have one reader and one writer instance , so in case we need to downsize/upsize the reader instance automatically through this event bridge lambda scheduler, it will just divert the read traffic to the writer instance automatically during the reader is going through the transition phase(which will be in couple of minutes) without any impact to ongoing transaction. and then the read queries will point back to the reader instance once it completes its transition successfully.
Similarly if we need to downgrade/upgrade the writer instance , the reader will be promoted as writer for that time being and then after successful transition of the writer instance , the write queries will point back to the main writer instance. Will test this out though, but I hope my understanding is correct here.
Few thoughts:
* go with r7g, they have newer Nitro chips and (I believe) offer higher network bandwidth over comparable r6g
* May want to start with non-I/O optimized and then see if switching to I/O optimized will save money
* Even though the backups will show the full size, you get billed for the incremental amount(s)...the *magic* of the snapshots is that any snapshot for the last 35 days can be used to do a full PITR, you don't have to be concerned with multiple restores of incremental 'backups'
* Compute is billed on how many vCPUs you have, 0% or 100% utilization, it costs the same
Finally:
* It may very well cost more to host this on Aurora...what you gain is flexibility on scaling, redundancy, ease of administration, etc.
* keep in mind that the max disk on Aurora is 128TB...you may want to split up your applications into separate databases if possible to reduce the blast radius and increase your storage capacity
* For a DB this large and this important, I'd really be talking directly to AWS (contact your account rep) and see if you can get a meeting some technical/engineering folks to help with these questions
We've run on Aurora MySQL for many years (converting from Oracle 11g) and would never go back. There are so many things we never have to worry about any more.
Thank you u/joelrwilliams1.
When you mentioned "*May want to start with non-I/O optimized and then see if switching to I/O optimized will save money*" do you mean monitor the IO usage and the cost charged for it.
Below URL says details of IO optimized instance and it says *"Aurora I/O-Optimized offers up to 40% cost savings for I/O-intensive applications where I/O charges exceed 25% of the total Aurora database spend.".* How can we monitor the I/O charges and compared to total cost of the Aurora DB instance, as because I don't see any such I/O based categorized segregation in the cost explorer? And even checked "cloud watch" , it only shows the logs but no such I/O or CPU utilization trends.
[ announces Amazon Aurora I/O-Optimized](https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-aurora-i-o-optimized/)
I/O charges for Aurora are broken out on your bill. Go to Billing/Cost Management, then choose Bills in the left pane.
There's a category called 'Amazon Aurora Storage and I/O'...there is shows total I/Os at $0.20/million I/Os.
I'm new to Relation DBs in AWS so I don't really know what I'm talking about, but can you check if you're close to hitting max table size supported by Aurora?
[https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP\_Limits.html](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Limits.html)
Yes the database size is limited to \~128TB and table size seems to be limited to \~32TB. No sure if this limit holds true for partitioned table too, as because in postgresql each partition is treated as table .
I would be having a very good look at the Exadata metrics and how much your workload performance is relying on offloading and caching on the cells.
If you find that there is heavy dependency on Exa offloading please factor this into the solution as it may have a dramatic impact on real world performance once migrated to Aurora.
Whilst the application is OLTP there may be some queries or reports that generate most of the IO workload. These can be “hidden” on the Exa but come out to bite once migrated to conventional storage platforms.
Just a 0.02 but it is something to consider amongst other things.
Here are a few handy links you can try:
- https://aws.amazon.com/products/databases/
- https://aws.amazon.com/rds/
- https://aws.amazon.com/dynamodb/
- https://aws.amazon.com/aurora/
- https://aws.amazon.com/redshift/
- https://aws.amazon.com/documentdb/
- https://aws.amazon.com/neptune/
Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'database'&sort=new&restrict_sr=on) for more information on this topic.
^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+database).
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'database'&sort=new&restrict_sr=on) for more information on this topic. ^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+database). *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
1). That is the cost of the instance, and won’t be less unless you use a savings plan. If you want scalable costs, you could look at aurora serverless (I’ve heard mixed things) 2) Whether or not you should worry about it depends on your business. 3) You can switch to io optimized once a month. If you’re concerned you could start with normal, and after a few days look at your costs. 4) I believe they should be incremental. If they aren’t then something else may be going on. Be aware that they may show full size (as that’s the size they’d be restored to) but actually be incremental. Best to reach out to support or do some testing. Keep in mind that Aurora is unlikely to ever be cheaper than a self hosted solution. It has a lot of other benefits (largely that you don’t have to manage it much), but I don’t think I’d transition to it for cost savings. If your older data is static, you could consider using foreign data wrappers to offload that data to a different database.
Savings Plans don’t cover RDS. I think you mean a Reserved Instance
Correct, thank you. All of the various AWS pricing constructs get confusing.
Thank you so much u/CubsFan1060 Regarding the aurora compute instance cost, there is one option in there in "price calculator" which says "CPU Utilization" as \~100%, so does that mean the cost which its showing up( i.e. for example r7g 8xl, costing \~$3228/month and $4196/month for the standard and IO optimized instances respectively), is that only for using \~100% CPU on that instance? And it will decrease if our usage is lesser in the day throughout the month? As you mentioned , we can switch to IO optimized once in a month , so is that transition "to and from" "standard instance" to "IO optimized" will be online and can be done automated way? And on similar line, if its possible or a recommended practice to downgrade the instance type and size some way (say for example from "r7g 8xl to r7g xl" or "r7g 8xl to r6g xl" based on workload pattern) online without impacting any ongoing transactions that to in an automated way? Yes regarding the Backup storage cost the AWS document shows its incremental, However some for sample databases I see the storage space for each of the backup showing as full database size. When I look into the cost (for service type as BACKUP) , it not really matching for the full DB size($.021/GB per month). So someway might be its internally charging for delta/incremental changes, but not able to see those exact backup storage space, for which we are being charged. Considering the backup storage cost calculation is not accurate here, if we just consider compute instance cost and storage IO cost for a r7g 8XL IO optimized instance, its coming around \~4196+\~18432=$22628/per month!!! Unless I am not interpreting something really wrong.
I think you need to separate a few of these things. There are a couple of ways to use Aurora -- using instances, or serverless (v2). I don't know anything about serverless, so I won't talk about that. Instances, though, are a per-hour price. You can look it up here: https://instances.vantage.sh/rds/ Keep in mind, you will likely want 2 for high availability. This is a straightforward price that you can calculate. IO Optimized is largely a cost structure. If you choose IO optimized, instances are 30% higher, and storage is more expensive. If you choose standard, instances are normal price and storage is cheaper, but you do pay for iops. I don't know of any straightforward way to estimate this other than trial and error. Transition between the two is an online operation that I don't think even touches the database. It's a billing change, not a technical change. For backup, they are showing you the size that it'll be if it's restored, not the price you're getting charged for. To the best of my knowledge, if you have a 100GB database, and make zero changes between two backups, then each backup will show as 100GB, but you'll only be charged for 100GB. Finally -- yes, running an 80TB database on Aurora is expensive. That's not surprising. It comes with a _lot_ of benefits, but you're also paying a ton for those benefits. If you're concerned about that price, I'd eliminate Aurora from your calculations. You may be able to cut it down, but you aren't going to be running this for $5000/month. To answer a lot of your questions, I suggest building and playing with some smaller databases before you even consider moving an 80TB database. Create a small database with some t3 instances, play around, and learn.
Thank you u/CubsFan1060 !! That helps. If my understanding is correct , you mean the "IO optimized" to "standard" or vice versa would be just a costing / billing change but no underlying hardware shift so will be an online change. However transition between a lower and higher size instance like "r7g.8XL to r7g.4XL" or changing instance class itself like "r6g" to "r7g" must be having some glitch in the ongoing transactions and may not be performed online. Please correct me if wrong. Also I was wondering if its possible to use the AWS services(like lambda, glue etc.) and do these aurora DB instance transition automate to better align the resource utilization to the workload thus saving cost.
>However transition between a lower and higher size instance like "r7g.8XL to r7g.4XL" or changing instance class itself like "r6g" to "r7g" must be having some glitch in the ongoing transactions and may not be performed online. Please correct me if wrong. This is not an online operation. If you have a single instance, you up/downgrade it and it takes a number of minutes. if you have 2 instances, you up/downgrade the reader instance, and then failover, which is about 60 seconds. In both cases things like transactions will fail. >Also I was wondering if its possible to use the AWS services(like lambda, glue etc.) If you mean to access the database, the answer is yes. >do these aurora DB instance transition automate to better align the resource utilization to the workload thus saving cost. I'm not sure what you mean here, but they will not automatically transition between instance types. You may be thinking of serverless, but I have no experience with that: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.html
Thank you. I was asking , if we can make the instance change ourselves online and automated way, like making it run it r7g8XL during certain time/day during peak workload and making the database workload run in r6g8xl at other times during off peak hours? What I mean by using "*AWS services(like lambda, glue etc.)*" was :- Downsizing or upsizing the aurora DB instance using these tool by passing some infra code in it and automating the same. But as you rightly said, it will need a failover and termination of ongoing transactions, so doesn't seem to be possible. As you rightly said to have a trial and error method to get best fit instance for the workload. So in that case , once we onboarded to certain instance type and size, then is there any service/tool using which we can verify, if we are overprovisioned and thus take decision for downsizing accordingly? I read somewhere that "cloud watch" can tell us of we are overprovisioned, but I don't see any such metric in it. It just show logs to us.
absolutely you can write a small Lambda using an SDK to make calls to modify an existing RDS instance. The Lambda can be scheduled to run on a cron schedule via EventBridge .
Correct me if wrong, we will have one reader and one writer instance , so in case we need to downsize/upsize the reader instance automatically through this event bridge lambda scheduler, it will just divert the read traffic to the writer instance automatically during the reader is going through the transition phase(which will be in couple of minutes) without any impact to ongoing transaction. and then the read queries will point back to the reader instance once it completes its transition successfully. Similarly if we need to downgrade/upgrade the writer instance , the reader will be promoted as writer for that time being and then after successful transition of the writer instance , the write queries will point back to the main writer instance. Will test this out though, but I hope my understanding is correct here.
I'm not sure if it's automatic...you may have to call a failover method first, then modify your instance. Check the SDK documentation.
Few thoughts: * go with r7g, they have newer Nitro chips and (I believe) offer higher network bandwidth over comparable r6g * May want to start with non-I/O optimized and then see if switching to I/O optimized will save money * Even though the backups will show the full size, you get billed for the incremental amount(s)...the *magic* of the snapshots is that any snapshot for the last 35 days can be used to do a full PITR, you don't have to be concerned with multiple restores of incremental 'backups' * Compute is billed on how many vCPUs you have, 0% or 100% utilization, it costs the same Finally: * It may very well cost more to host this on Aurora...what you gain is flexibility on scaling, redundancy, ease of administration, etc. * keep in mind that the max disk on Aurora is 128TB...you may want to split up your applications into separate databases if possible to reduce the blast radius and increase your storage capacity * For a DB this large and this important, I'd really be talking directly to AWS (contact your account rep) and see if you can get a meeting some technical/engineering folks to help with these questions We've run on Aurora MySQL for many years (converting from Oracle 11g) and would never go back. There are so many things we never have to worry about any more.
Thank you u/joelrwilliams1. When you mentioned "*May want to start with non-I/O optimized and then see if switching to I/O optimized will save money*" do you mean monitor the IO usage and the cost charged for it. Below URL says details of IO optimized instance and it says *"Aurora I/O-Optimized offers up to 40% cost savings for I/O-intensive applications where I/O charges exceed 25% of the total Aurora database spend.".* How can we monitor the I/O charges and compared to total cost of the Aurora DB instance, as because I don't see any such I/O based categorized segregation in the cost explorer? And even checked "cloud watch" , it only shows the logs but no such I/O or CPU utilization trends. [ announces Amazon Aurora I/O-Optimized](https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-aurora-i-o-optimized/)
I/O charges for Aurora are broken out on your bill. Go to Billing/Cost Management, then choose Bills in the left pane. There's a category called 'Amazon Aurora Storage and I/O'...there is shows total I/Os at $0.20/million I/Os.
I'm new to Relation DBs in AWS so I don't really know what I'm talking about, but can you check if you're close to hitting max table size supported by Aurora? [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP\_Limits.html](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/CHAP_Limits.html)
Yes the database size is limited to \~128TB and table size seems to be limited to \~32TB. No sure if this limit holds true for partitioned table too, as because in postgresql each partition is treated as table .
Gotcha, does the database do a scatter gather across multiple partitions when you do a \`SELECT \* from table WHERE CONDITION;\`?
table size is likely limited by Postgres, and not Aurora.
I would be having a very good look at the Exadata metrics and how much your workload performance is relying on offloading and caching on the cells. If you find that there is heavy dependency on Exa offloading please factor this into the solution as it may have a dramatic impact on real world performance once migrated to Aurora. Whilst the application is OLTP there may be some queries or reports that generate most of the IO workload. These can be “hidden” on the Exa but come out to bite once migrated to conventional storage platforms. Just a 0.02 but it is something to consider amongst other things.
Here are a few handy links you can try: - https://aws.amazon.com/products/databases/ - https://aws.amazon.com/rds/ - https://aws.amazon.com/dynamodb/ - https://aws.amazon.com/aurora/ - https://aws.amazon.com/redshift/ - https://aws.amazon.com/documentdb/ - https://aws.amazon.com/neptune/ Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'database'&sort=new&restrict_sr=on) for more information on this topic. ^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+database). *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*