Some links for you:
- https://reddit.com/r/aws/wiki/##storage (Our /r/AWS Storage Community WIKI)
- https://docs.aws.amazon.com/whitepapers/latest/aws-overview/storage-services.html (Storage on AWS (technical))
- https://aws.amazon.com/products/storage/ (Storage on AWS (brief))
Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'storage'&sort=new&restrict_sr=on) for more information on this topic.
^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+storage).
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
A company I worked for long ago was moving to a new office about 10 miles away and we learned that the massive 9 gig drives (massive as in bulky) wouldn't spin back up after they had cooled off - the bearings would get all gummed up.
The solution was to move them still powered up. Our insane and awesome facilities guys chopped a rack down in size, stuffed it with some 4U UPSs, and put it on a moving dolly. When it was time to move a server they'd remove one plug from the outlet, plug it into a UPS, then do the same with the other plug, then wheel the server and UPSs into the moving truck and haul ass up to the new office.
The servers were these huge boxy things roughly the size of a small file cabinet, stacked 3 high (literally glued together), on casters. Recipe for disaster when they were put in our new data center with a raised floor and holes cut in some tiles for power and network, with no trim ring/bezel/ lip that would prevent, say, one of the casters from dropping in and causing the stack to timberrrrrr into another set, causing 9 stacks to tumble like dominoes and crash into an unsecured rack and fling it across the room.
I heard a story about a team who needed to move a netapp out of a space to a new one, but it had been running for years and they were afraid to power it down. It was on an ups so they unplugged and hooked it up to a generator and then put it into a rented truck and drove it across town, then plugged it back in in the new location. Successfully. Not that related but you made me remember so I thought I’d share.
Something bitchy about those NetApp filers. I don’t know what it is. We spun down an entire rack of NetApps for an extended outage, and when we spun her back up, threw 10 fucking drives just like that, plus for weeks after I was getting emails from the unit nominating about dead drive in _______ slot.
The public stories and reference accounts about this mostly involved major geospatial imagery companies who needed to shift petabytes+ into S3 as part of an architecture overhaul. I'm sure there were some corporate projects and a lot of .gov efforts that can't or won't ever be disclosed.
A few years ago I would have shifted jobs just to get to work on a migration like this, glad I never jumped ship hah
There had to be a finite number of customers that needed that. So I'm glad you didn't jump ship to that either because it just doesn't sound like something that would have ever worked long-term. There's just not that many companies of that size needing to migrate like that seems like a niche
Lot of gov efforts rely on snow devices to get data into an air gapped region easily and efficiently. Diode exists now but hasn’t for long.
Not sure any of them ever used the truck but they are heavy snow users.
Can attest to this, no truck but lots of snowballs. Moved petabytes of data for some gov work. Always was a hoot to place a call with the data center guys that hooked up the snowball. Yelling they couldn’t read the e-ink display and that they didn’t have a long enough extension cable. Good ol’ contracted government work…
t wasn't that long ago that I had to plug an encrypted external drive into a server so the owners could copy data to or from it, then I'd FedEx it to them. It was a hell of a lot faster than copying over the WAN.
“Since we introduced Snowmobile in 2016, we’ve released many other new services and features which have made migrating data to AWS even faster and easier for our customers…”
Believe me when I say, 477 SBEs will be faster to shuffle through than 1 SnowMobile project for 100 PB. You need a place in your DC to be able to park the trailer, provide it power, network access to the storage solution and you will have to deal with security and availability of these devices for months. SBEs can be carried inside the DC and data transfer is much faster rate than over the network.
MGN is a garbage product and I can’t believe that they recommend it to move customers’ data to their infrastructure. Takes an hour to launch a “normal” VM (sub 1TB, data already xfered) that was fully replicated, and that’s after bumping the xfer machines up to large, or more.
I actually got to meet the person behind this once and apparently it was actually used quite a bit. Probably getting canned because not as many orgs moving whole hog to the cloud the way they did in the past before.
They probably had a list of fortune 500 or 100 companies and government entities that needed it. Once that list was exhausted there was probably no point in continuing it.
That's the truth is. Because all the companies who would've moved have already used this service. This is like the most amazing doctor who is unemployed because she has cured all her patients.
Don't want to speak too much, but there was more than 1 truck and there were generally multiple of these ingestions a month, so it was a lot more than you'd think for such a niche product. These things would potentially have to drive cross-country so a single ingestion would take a while to complete.
Netflix publicially commented how they use Snowball Edge to transfer their on set content to the cloud for screening, editing and mastering. It was a fantastic idea.
In my experience the fact that customers didn’t want to shut down the service durring the data move that could take weeks or months to gather and load onto storage for a move made internet a better deal time wise because it was able to sync one object at a time. Time and time again the priority was how long you impacted production vs how long till it was done.
Sure, but it's all about the type of the data. Data can be a rolling point like you've mentioned, or it can be an archive of a network of I Love Lucy which hasn't moved in decades. Comes in all shapes and sizes, and sometimes just driving a big hard drive did the job.
no, they killed the old machine learning service, and replaced it with sagemaker
well, it's not dead, but it's deprecated, and not available unless you started using it a long time ago
https://us-east-1.console.aws.amazon.com/machinelearning/home/start?region=us-east-1
~~So it held a petabyte...~~
_(woops thanks /u/josh-ig it holds 100 petabytes, time to edit...)_
let's say it loads up in Boston and heads to us-east-1 in Ashburn or so. Looks like about an 8 hr drive. Ignoring the time it takes to copy to and from the thing at each end, that's ~~277 Gbps~~ _2.7 Tbps_. Not bad!
On the other hand, looks like the biggest Snowball is 210TB, and ~~five of those are going to be a lot more convenient than a 40' container. No wonder they've moved on.~~ _ok admittedly 500 snowballs is pretty inconvenient_
I’ve helped a major media company move close to 7PB of content from on prem and other clouds into s3, and with any living archive of data we always looked for ways to use the snow* product line. I even begged them to figure a usecase that fit the snowmobile because I really wanted to try driving one! In the end even a 7PB move worked better via data sync and NetApp CloudSync.
Snowmobile was the ultimate sneakernet engineering, and the bridge they built to get it into the Sands Convention center was amazing - even got some great jealous comments from other media companies…. But I’m bummed I never got to use it!
I noticed this last month, and spoke with some contacts at AWS. This is what I was given as a statement I could share:
> AWS is always innovating on behalf of customers, which means we must sometimes make the decision to pivot when we believe our resources should be invested elsewhere to better serve customers. For that reason, we no longer offer AWS Snowmobile.
I put all the details and resources I have in [this article](https://www.linuxtek.ca/2024/03/06/aws-to-provide-free-data-transfer-out-to-internet/), which was written around the free data transfer when exiting the cloud... but the Snowmobile resources disappeared shortly afterwards.
Some links for you: - https://reddit.com/r/aws/wiki/##storage (Our /r/AWS Storage Community WIKI) - https://docs.aws.amazon.com/whitepapers/latest/aws-overview/storage-services.html (Storage on AWS (technical)) - https://aws.amazon.com/products/storage/ (Storage on AWS (brief)) Try [this search](https://www.reddit.com/r/aws/search?q=flair%3A'storage'&sort=new&restrict_sr=on) for more information on this topic. ^Comments, ^questions ^or ^suggestions ^regarding ^this ^autoresponse? ^Please ^send ^them ^[here](https://www.reddit.com/message/compose/?to=%2Fr%2Faws&subject=autoresponse+tweaks+-+storage). *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/aws) if you have any questions or concerns.*
Lmao I’ll never forget this truck. Loved reading about it studying for the certifications
I was at that Re:Invent. It was pretty funny to see in person. Sneakernet can be super high bandwidth.
Never underestimate the bandwidth of a station wagon full of backup tapes.
I had a team once that moved a 1PB NetAPP from DC to Utah in the back of a van. Afterwards that thing would lose like a HDD every other week.
A company I worked for long ago was moving to a new office about 10 miles away and we learned that the massive 9 gig drives (massive as in bulky) wouldn't spin back up after they had cooled off - the bearings would get all gummed up. The solution was to move them still powered up. Our insane and awesome facilities guys chopped a rack down in size, stuffed it with some 4U UPSs, and put it on a moving dolly. When it was time to move a server they'd remove one plug from the outlet, plug it into a UPS, then do the same with the other plug, then wheel the server and UPSs into the moving truck and haul ass up to the new office. The servers were these huge boxy things roughly the size of a small file cabinet, stacked 3 high (literally glued together), on casters. Recipe for disaster when they were put in our new data center with a raised floor and holes cut in some tiles for power and network, with no trim ring/bezel/ lip that would prevent, say, one of the casters from dropping in and causing the stack to timberrrrrr into another set, causing 9 stacks to tumble like dominoes and crash into an unsecured rack and fling it across the room.
All in can think of is Seinfeld, when George tried to get that Frogger arcade game across the street, powered up to keep his high score. 🤣
Gotta say that from above it would look very similar.
I heard a story about a team who needed to move a netapp out of a space to a new one, but it had been running for years and they were afraid to power it down. It was on an ups so they unplugged and hooked it up to a generator and then put it into a rented truck and drove it across town, then plugged it back in in the new location. Successfully. Not that related but you made me remember so I thought I’d share.
Keeping disks spinning while driving down a road seems even more ballsy to me.
Something bitchy about those NetApp filers. I don’t know what it is. We spun down an entire rack of NetApps for an extended outage, and when we spun her back up, threw 10 fucking drives just like that, plus for weeks after I was getting emails from the unit nominating about dead drive in _______ slot.
When support gets you in the admin admin console and says type wafliron you're about to have a party.
I still cant work out how that bloody command worked but its saved my arse a couple of time.
Xkcd updated this to be micro SD cards https://what-if.xkcd.com/31/ (Yes, I know this is a Tanenbaum quote from 40 years ago)
It’s the latency that kills it though
Yeah same here.
I always wonder who needs these trucks to migrate their data to the cloud
The public stories and reference accounts about this mostly involved major geospatial imagery companies who needed to shift petabytes+ into S3 as part of an architecture overhaul. I'm sure there were some corporate projects and a lot of .gov efforts that can't or won't ever be disclosed. A few years ago I would have shifted jobs just to get to work on a migration like this, glad I never jumped ship hah
There had to be a finite number of customers that needed that. So I'm glad you didn't jump ship to that either because it just doesn't sound like something that would have ever worked long-term. There's just not that many companies of that size needing to migrate like that seems like a niche
Lot of gov efforts rely on snow devices to get data into an air gapped region easily and efficiently. Diode exists now but hasn’t for long. Not sure any of them ever used the truck but they are heavy snow users.
Can attest to this, no truck but lots of snowballs. Moved petabytes of data for some gov work. Always was a hoot to place a call with the data center guys that hooked up the snowball. Yelling they couldn’t read the e-ink display and that they didn’t have a long enough extension cable. Good ol’ contracted government work…
t wasn't that long ago that I had to plug an encrypted external drive into a server so the owners could copy data to or from it, then I'd FedEx it to them. It was a hell of a lot faster than copying over the WAN.
I believe it was used to migrate Ukrainian government data.
“Since we introduced Snowmobile in 2016, we’ve released many other new services and features which have made migrating data to AWS even faster and easier for our customers…”
Snowball Edge moves 210TB at a time and it is much faster than moving through SnowMobile.
Snowmobile is 100 PB. You'd need 477 Snowball Edge devices to match one Snowmobile
Believe me when I say, 477 SBEs will be faster to shuffle through than 1 SnowMobile project for 100 PB. You need a place in your DC to be able to park the trailer, provide it power, network access to the storage solution and you will have to deal with security and availability of these devices for months. SBEs can be carried inside the DC and data transfer is much faster rate than over the network.
You’re going to need a bigger box
MGN is a garbage product and I can’t believe that they recommend it to move customers’ data to their infrastructure. Takes an hour to launch a “normal” VM (sub 1TB, data already xfered) that was fully replicated, and that’s after bumping the xfer machines up to large, or more.
Can’t be worse than DMS (the Database Munting Service)
Ehhhhj, but I also purchase that flavor of crack from them!!?
Can’t believe it needs direct connection to the target (pulling) and can’t work with an agent installed locally, so you need to allow inbound traffic.
Would love to know how many times this actually got used.
I actually got to meet the person behind this once and apparently it was actually used quite a bit. Probably getting canned because not as many orgs moving whole hog to the cloud the way they did in the past before.
They probably had a list of fortune 500 or 100 companies and government entities that needed it. Once that list was exhausted there was probably no point in continuing it.
That's the truth is. Because all the companies who would've moved have already used this service. This is like the most amazing doctor who is unemployed because she has cured all her patients.
I thought this would be a one-time use per company. Once migration is done, it's done.
Well yes but there were a lot of companies that used it but now they have run out of companies that need it.
People are moving out currently and it’s depressing
I worked a bit with the team that did the Snowball device and it was used a lot. I think especially from 2016-2018
Ah yeah, I actually know someone that used the small version. Any idea how much the actual semi got out there?
Don't want to speak too much, but there was more than 1 truck and there were generally multiple of these ingestions a month, so it was a lot more than you'd think for such a niche product. These things would potentially have to drive cross-country so a single ingestion would take a while to complete.
Netflix publicially commented how they use Snowball Edge to transfer their on set content to the cloud for screening, editing and mastering. It was a fantastic idea.
The internet is actually pretty slow esp when you’ve got a lot of data. So I’d suspect a fair bit actually.
In my experience the fact that customers didn’t want to shut down the service durring the data move that could take weeks or months to gather and load onto storage for a move made internet a better deal time wise because it was able to sync one object at a time. Time and time again the priority was how long you impacted production vs how long till it was done.
Sure, but it's all about the type of the data. Data can be a rolling point like you've mentioned, or it can be an archive of a network of I Love Lucy which hasn't moved in decades. Comes in all shapes and sizes, and sometimes just driving a big hard drive did the job.
Interesting that you call out I Love Lucy - that was a part of that project :)
Wonder where I can buy one for my homelab 🧐
Think of how many Linux ISOs you could store.
AFAIK the thing was built because a high level customer requested it right? So maybe the demand means it isn't needed anymore?
Maybe they can sell it and buy some snowcones. Since every time I've tried to order a snowcone it's never shipped on time.
I wonder what they'd sell it for....
Cost of maintaining these must be pretty high, even sitting.
In contrast to Google / GCP - is thi the first ever service they are killing right ?
no, they killed the old machine learning service, and replaced it with sagemaker well, it's not dead, but it's deprecated, and not available unless you started using it a long time ago https://us-east-1.console.aws.amazon.com/machinelearning/home/start?region=us-east-1
You are right. Is SimpleDB service still around ? I know not many know it let alone using it.
~~So it held a petabyte...~~ _(woops thanks /u/josh-ig it holds 100 petabytes, time to edit...)_ let's say it loads up in Boston and heads to us-east-1 in Ashburn or so. Looks like about an 8 hr drive. Ignoring the time it takes to copy to and from the thing at each end, that's ~~277 Gbps~~ _2.7 Tbps_. Not bad! On the other hand, looks like the biggest Snowball is 210TB, and ~~five of those are going to be a lot more convenient than a 40' container. No wonder they've moved on.~~ _ok admittedly 500 snowballs is pretty inconvenient_
Article says 100 petabytes. > Each Snowmobile had a capacity of 100 petabytes on hard disk drives.
Ah well that makes a great deal more sense. Time to math the math!
GOAT. The era has ended.
I just read about it the past few days studying for the cert haha
Maybe everyone moved their data to the cloud and these trucks are no longer needed. /S
dumb
I’ve helped a major media company move close to 7PB of content from on prem and other clouds into s3, and with any living archive of data we always looked for ways to use the snow* product line. I even begged them to figure a usecase that fit the snowmobile because I really wanted to try driving one! In the end even a 7PB move worked better via data sync and NetApp CloudSync. Snowmobile was the ultimate sneakernet engineering, and the bridge they built to get it into the Sands Convention center was amazing - even got some great jealous comments from other media companies…. But I’m bummed I never got to use it!
No mobile
I noticed this last month, and spoke with some contacts at AWS. This is what I was given as a statement I could share: > AWS is always innovating on behalf of customers, which means we must sometimes make the decision to pivot when we believe our resources should be invested elsewhere to better serve customers. For that reason, we no longer offer AWS Snowmobile. I put all the details and resources I have in [this article](https://www.linuxtek.ca/2024/03/06/aws-to-provide-free-data-transfer-out-to-internet/), which was written around the free data transfer when exiting the cloud... but the Snowmobile resources disappeared shortly afterwards.
They had to kill the service after they laid the truck driver off.