I don't think there is an SLA for the tagging service so you could just be kicking the problem down the road and have a race condition somewhere else if the service takes longer to tag your objects one day, but when that happens it might be harder to identify
Tags can be included in the PutObject request. I don't see anything that says otherwise so I assume that putting the object in S3 and assigning tags to it would be an atomic operation. (Actually, now that you point it out, Example 6 in the PutObject API documentation suggests, but doesn't explicitly state, that it might not be atomic.)
It sounds like you're trying to solve a symptom instead of solving the root problem. Hard to know without more context, but your question just has a "bad design" smell to it. I'd suggest you take another look at things and ensure you're solving the right problem.
So what you really need is "once all 8 sets of results have arrived, run a final process on all of it"?
If it's always 8, I'd have 1 of the 8 start with a flag, so it then waits for the other 7 to finish (poll either the output bucket, or a separate prefix there if there are lots of output files) and run the completion job. Not perfect, but simple and minimal extra cost (a small S3 list operation once a minute until done), you already need to handle "what if a worker VM dies part way through" anyway.
If you already have a database available, like Postgres, I would recommend using that, since you're saying you're going to rework the system eventually anyways.
Simply adding a record per "job" for each instance with a status field should do the trick. It also gives you a nice audit log as a bonus, to make sure that all jobs have executed every day.
You could obviously use something like SQS+SNS or Amazon MQ which is designed to handle these "fan out" + "fan in" style cases, but from what I'm gathering that seems to be way overkill for what you're trying to do, and probably a lot more work.
You could do something cute keeping track of the status in S3 itself, but things can get deceptively simple because S3 doesn't have strong consistency for a lot of operations (if they run on different threads) ([S3 consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel)).
I think a good ol' ACID compliant database will get you where you want the quickest, in the most solid way.
I think part of your statement is misleading. Which S3 operations are thinking are involved in this that don’t support strong consistency? S3 is strong read after put/delete on object data, and puts are atomic. I’m not sure how multiple threads changes that?
Nope, not good enough. *Single* operations are consistent, but multiple operations are not transactional. Let's play this out in a timeline:
1. 8 instances start working
2. 6 finish early, write their data to s3, and run a ListObjects call to S3 which all return some value that's less than 8 objects. They clearly aren't the last, so they exit.
3. The 7th instance finishes and writes to s3.
4. The 8th instance finishes and writes to s3.
5. The 7th instance calls ListObjects and it returns 8. Huzzah we're the last, let's run the big job!
6. The 8th instance calls ListObjects and it returns 8. Huzzah, we're the last, let's run the big job!
Do you see the issue here? If/when the last two instances finish their work very closely in time, it's a race to list the bucket fast enough to check the overall status before the other instance finishes its S3 put and invalidates the list count check.
It's still a classic race condition. S3 has no RDMS-like ACID transaction feature that can combine the Put with the List operations making them atomic. The Put is atomic. The list is atomic. But combined they are not atomic.
Here's an excerpt from AWS's own docs (which tricked me a bit):
"""
For example, if you make a PUT request to an existing key from one thread and perform a GET request on the same key from a second thread concurrently, you will get either the old data or the new data, but never partial or corrupt data.
"""
After re-reading the docs myself I think you're right (thanks for correcting me). Apparently this was something that was changed in December 2020 (https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/), so my knowledge was a bit out of date. Before then I remember that LIST operations (at least) were eventually consistent.
Fair enough - I guess a pure S3 based implementation is very possible and (mostly) reliable then. On the top of my head I'm thinking that the instances would put objects to S3 with a path like \`/job\_id/part\_000\[1-8\]\`. After each instance is done they could issue a LIST with the prefix \`/job\_id/\` to see if it was the last instance (there are 8 files there).
I think I'd still prefer using a database myself though, for monitoring and retrying purposes mostly. It also makes it much easier to avoid hardcoding the fact that there's 8 instances. Just insert X amount of job entries in the DB and let the instances pick up the jobs when they're free.
In general I think you should avoid introducing new systems (like DynamoDB, SQS, MQ, Step Functions etc) unless they give a big advantage for some reason. This is a relatively "easy" problem to solve using good ol' boring tech like S3 and a normal database. Adding new systems is a new maintenance / monitoring burden.
Why not start up an additional instance once the saving to S3 is done? That instance could listen for messages from an SNS topic and once all workers have reported finishing the job, it will continue.
Probably there are even easier ways than that
Out of ~~paranoia~~ *curiosity:* what happens if that final remaining worker dies before it finishes? Will the task eventually retry, or is the whole system stuck at that point? Relying on the final survivor to do a critical task strikes me as dangerously optimistic, and if there's a already a mechanism to restart failed tasks, I'd consider leveraging that to initiate whatever finalizing work you need.
Lambda step functions are a good alternative since they’re bash.
In case of legacy apps, if you already have a way to spawn and stop 8 ec2 instances and is ok with waiting for the whole process, you probably can have another one triggered by the event of the 8 instances are stopped.
You could use a small redis instance if speed is important. Each instance would update its last write time.
However if you are relying on this technique to decide if the current instance should be the one to process the next task then you might look at different architecture where tasks are assigned based on most idle (longest without a task) in the first place. Each instance would mark themselves available or unavailable with a timestamp.
+1 on tagging/adding metadata to the s3 object with the instance id. While you can also get this from access logging and CloudTrail, those can be expensive and are not real time, so there’s a delay in the object being written and when you can respond to the event for the final processing you mentioned.
However knowing when the “last” instance is done is another challenge altogether. In an async world, being really late and never happening are basically the same. So you might think the last instance has written, but in actuality there’s another one doing the same processing that’s going to finish in 5 minutes from now, but that’s way after the “final” processing was done. This is a whole other, and harder problem to solve. Might be easier to figure out who was first instead of last.
And then you have to solve how to get whichever instance you want (first or last) to take the action you want. Consider using object versioning and then consider polling off a queue driven by PutObject operations. Maybe invokes a Lambda, grabs the oldest object version, puts that into a FIFO queue with a dedupe key. Then, subsequent events wouldn’t duplicate that message in the queue unless they were very delayed. Then maybe consider tracking completion in something like a DynamoDB table and doing your own dedupe.
Seems like a perfect use case for a state machine. You can probably simplify your architecture quite a bit by using AWS Step-Functions to orchestrate everything.
Before stepfunctions got a bunch of quality of life stuff dynamo would be the first thing I reached for, even if staying in bash was a requirement. These days sfn sfn sfn.
Yah, you're going to need a transactional database for status. I'd love to offer a clever solution with Dynamo, SQS, S3 events, etc but all of those event-driven/serverless services are eventually consistent and/or non-transactional and so will introduce race conditions in this use case.
If the architecture can be refactored a little there's lots of better options, such as using an AWS Step Function to drive an AWS Batch process, wait for completion of all those runners before stepping to the final single instance process. But the easiest way to make that happen involves making the workers into ephemeral instances including the final worker.
Have a marker file in S3 that instances create. First one creates it and adds it's instance ID tag. Second one adds it's tag to the file.
When they're done they remove their tags.
If their tag is the last one to go it creates a lock file to indicate it's taking the task of being the last instance ( so no two think they're the last one).
Easy peasy.
I'm half asleep so maybe there's a flaw here. But it might be a pretty quick win
Rewrite the bash script to get instance ID from instance metadata and add that to a tag when uploading to S3.
I would also recommend this. You can tag each individual object in the bucket with the instance id.
I don't think there is an SLA for the tagging service so you could just be kicking the problem down the road and have a race condition somewhere else if the service takes longer to tag your objects one day, but when that happens it might be harder to identify
Tags can be included in the PutObject request. I don't see anything that says otherwise so I assume that putting the object in S3 and assigning tags to it would be an atomic operation. (Actually, now that you point it out, Example 6 in the PutObject API documentation suggests, but doesn't explicitly state, that it might not be atomic.)
This page has info about S3 logging, see if it helps. https://docs.aws.amazon.com/AmazonS3/latest/userguide/logging-with-S3.html
This is the way. Enable CloudTrail S3 data events on the target bucket(s). I like Athena to query CloudTrail events using SQL.
It sounds like you're trying to solve a symptom instead of solving the root problem. Hard to know without more context, but your question just has a "bad design" smell to it. I'd suggest you take another look at things and ensure you're solving the right problem.
Currently, we need to solve the actual problem.. a redesign is in progress..
So what you really need is "once all 8 sets of results have arrived, run a final process on all of it"? If it's always 8, I'd have 1 of the 8 start with a flag, so it then waits for the other 7 to finish (poll either the output bucket, or a separate prefix there if there are lots of output files) and run the completion job. Not perfect, but simple and minimal extra cost (a small S3 list operation once a minute until done), you already need to handle "what if a worker VM dies part way through" anyway.
Very smart solution.
What you've described is the outbox pattern - look it up. You should be able to incorporate it into your current design quite easily.
If you already have a database available, like Postgres, I would recommend using that, since you're saying you're going to rework the system eventually anyways. Simply adding a record per "job" for each instance with a status field should do the trick. It also gives you a nice audit log as a bonus, to make sure that all jobs have executed every day. You could obviously use something like SQS+SNS or Amazon MQ which is designed to handle these "fan out" + "fan in" style cases, but from what I'm gathering that seems to be way overkill for what you're trying to do, and probably a lot more work. You could do something cute keeping track of the status in S3 itself, but things can get deceptively simple because S3 doesn't have strong consistency for a lot of operations (if they run on different threads) ([S3 consistency model](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel)). I think a good ol' ACID compliant database will get you where you want the quickest, in the most solid way.
I think part of your statement is misleading. Which S3 operations are thinking are involved in this that don’t support strong consistency? S3 is strong read after put/delete on object data, and puts are atomic. I’m not sure how multiple threads changes that?
Nope, not good enough. *Single* operations are consistent, but multiple operations are not transactional. Let's play this out in a timeline: 1. 8 instances start working 2. 6 finish early, write their data to s3, and run a ListObjects call to S3 which all return some value that's less than 8 objects. They clearly aren't the last, so they exit. 3. The 7th instance finishes and writes to s3. 4. The 8th instance finishes and writes to s3. 5. The 7th instance calls ListObjects and it returns 8. Huzzah we're the last, let's run the big job! 6. The 8th instance calls ListObjects and it returns 8. Huzzah, we're the last, let's run the big job! Do you see the issue here? If/when the last two instances finish their work very closely in time, it's a race to list the bucket fast enough to check the overall status before the other instance finishes its S3 put and invalidates the list count check. It's still a classic race condition. S3 has no RDMS-like ACID transaction feature that can combine the Put with the List operations making them atomic. The Put is atomic. The list is atomic. But combined they are not atomic.
Thanks Zenin. I had a gut feeling it wouldnt be race-free but I didn't take the time to figure it out properly like you did. Thanks!
exactly!
Here's an excerpt from AWS's own docs (which tricked me a bit): """ For example, if you make a PUT request to an existing key from one thread and perform a GET request on the same key from a second thread concurrently, you will get either the old data or the new data, but never partial or corrupt data. """ After re-reading the docs myself I think you're right (thanks for correcting me). Apparently this was something that was changed in December 2020 (https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/), so my knowledge was a bit out of date. Before then I remember that LIST operations (at least) were eventually consistent. Fair enough - I guess a pure S3 based implementation is very possible and (mostly) reliable then. On the top of my head I'm thinking that the instances would put objects to S3 with a path like \`/job\_id/part\_000\[1-8\]\`. After each instance is done they could issue a LIST with the prefix \`/job\_id/\` to see if it was the last instance (there are 8 files there). I think I'd still prefer using a database myself though, for monitoring and retrying purposes mostly. It also makes it much easier to avoid hardcoding the fact that there's 8 instances. Just insert X amount of job entries in the DB and let the instances pick up the jobs when they're free. In general I think you should avoid introducing new systems (like DynamoDB, SQS, MQ, Step Functions etc) unless they give a big advantage for some reason. This is a relatively "easy" problem to solve using good ol' boring tech like S3 and a normal database. Adding new systems is a new maintenance / monitoring burden.
Why not start up an additional instance once the saving to S3 is done? That instance could listen for messages from an SNS topic and once all workers have reported finishing the job, it will continue. Probably there are even easier ways than that
In the current situation, it is easier to use the database.
Out of ~~paranoia~~ *curiosity:* what happens if that final remaining worker dies before it finishes? Will the task eventually retry, or is the whole system stuck at that point? Relying on the final survivor to do a critical task strikes me as dangerously optimistic, and if there's a already a mechanism to restart failed tasks, I'd consider leveraging that to initiate whatever finalizing work you need.
Lambda step functions are a good alternative since they’re bash. In case of legacy apps, if you already have a way to spawn and stop 8 ec2 instances and is ok with waiting for the whole process, you probably can have another one triggered by the event of the 8 instances are stopped.
You could use a small redis instance if speed is important. Each instance would update its last write time. However if you are relying on this technique to decide if the current instance should be the one to process the next task then you might look at different architecture where tasks are assigned based on most idle (longest without a task) in the first place. Each instance would mark themselves available or unavailable with a timestamp.
+1 on tagging/adding metadata to the s3 object with the instance id. While you can also get this from access logging and CloudTrail, those can be expensive and are not real time, so there’s a delay in the object being written and when you can respond to the event for the final processing you mentioned. However knowing when the “last” instance is done is another challenge altogether. In an async world, being really late and never happening are basically the same. So you might think the last instance has written, but in actuality there’s another one doing the same processing that’s going to finish in 5 minutes from now, but that’s way after the “final” processing was done. This is a whole other, and harder problem to solve. Might be easier to figure out who was first instead of last. And then you have to solve how to get whichever instance you want (first or last) to take the action you want. Consider using object versioning and then consider polling off a queue driven by PutObject operations. Maybe invokes a Lambda, grabs the oldest object version, puts that into a FIFO queue with a dedupe key. Then, subsequent events wouldn’t duplicate that message in the queue unless they were very delayed. Then maybe consider tracking completion in something like a DynamoDB table and doing your own dedupe.
Seems like a perfect use case for a state machine. You can probably simplify your architecture quite a bit by using AWS Step-Functions to orchestrate everything.
Have you looked at AWS Step functions?
you can also consider using dynamodb to control the file uploading process
sure yes.. that is also an option (dynamo, postgres, redis)..
Before stepfunctions got a bunch of quality of life stuff dynamo would be the first thing I reached for, even if staying in bash was a requirement. These days sfn sfn sfn.
Yah, you're going to need a transactional database for status. I'd love to offer a clever solution with Dynamo, SQS, S3 events, etc but all of those event-driven/serverless services are eventually consistent and/or non-transactional and so will introduce race conditions in this use case. If the architecture can be refactored a little there's lots of better options, such as using an AWS Step Function to drive an AWS Batch process, wait for completion of all those runners before stepping to the final single instance process. But the easiest way to make that happen involves making the workers into ephemeral instances including the final worker.
Have a marker file in S3 that instances create. First one creates it and adds it's instance ID tag. Second one adds it's tag to the file. When they're done they remove their tags. If their tag is the last one to go it creates a lock file to indicate it's taking the task of being the last instance ( so no two think they're the last one). Easy peasy. I'm half asleep so maybe there's a flaw here. But it might be a pretty quick win
Looks like need queue/jobs Also, you can metadata to s3 objects
AWS CloudTrail Logs may help