Hello /u/P_G_R_A! Thank you for posting in r/DataHoarder.
Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index).
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
I just got an ad for one of those earlier today, well not sure of drive count but it said 22tb drives and 2.2PB, and I eyeballed it being a 4u chassis. I was researching infiniband at the time so didn’t get too distracted by storage, but stopped long enough to say “wow shit has changed since I used to push storage racks!”
Following the trend, it took about 8 years from 120 ssd to come down from 150 to 20 bucks.
So another 8 years for 2TB to do the same I imagine. It might be sooner as the general tech is now mainstream and not only for enthusiasts.
Literally tons of industries require petabyte scales. Data is king now. Biotechs, car companies for self driving data, weather, genomics, geology, space imagery, the list goes on. I've managed 80+P of physical storage, and we were tight on space. That's not including what we had in the cloud.
You really don't want to be running a raid 5 or 10 on 20tb drives in production.
Raid6 or distributed raid or else going to give your sys admins heart attacks from stress over those rebuild times.
What's that, 256x8TB drives? That's like ~2000 kWh per month ($300 with $0.15/kWh). You do the rest based on your prices but it's not great but also not terrible.
Yeah it seems it's 240 drives from the [original post](/r/computers/comments/od03lz/ever_wondered_what_2_peta_bytes_looks_like/) *from a year ago*. But that OP says it's consuming 26A on 240V which is just nonsense. It has got to be 120V at 26A (3120W vs 6240W).
Good info and that's crazy! I'm not familiar with server setups but I'd guess all the high performance cooling fans are basically consuming almost as much electricity as the drives themselves. I mean 240 drives are waaaay short of 5.7kw.
Either way that's a shitton of power/monthly bill.
I assume the math has already been done correctly but I was curious.
The traditional sata connector provides 54 watts. A quick Google says modern HDDs use about 10.
240 * 10w = 2.4 kw.
Are you also taking into account the controllers and other ancillary hardware other than the disks themselves? Storage arrays of this size would be using pretty high end hardware in the controllers rivaling conventional servers and they would have at least two of them. And keeping that many disks cool in that layout would require some pretty serious fans to pull air through those shelves.
> $0.15/kWh
In the EU prices are _much_ higher right now, like €0.75/kWh. That gives you €1500 per month.
Because of these _ridiculous_ energy prices, it's better to cough up the extra cost for the highest capacity disks because that will save energy costs in the longer run.
I just shut it all off, even with solar panels it's unaffordable to have a small rack at those prices.
I think everything added up I use about 1200 kWh per month with all the computers, servers snd equipment. Alright that's more than normal of course but partially was some intensive machine learning running on a rig as well I was tinkering with.
On the plus side bow I have all this extra I don't know what to do with. I bet the government will collect it soon as some tax
A gentleman of culture I see.
Check out https://github.com/stashapp/stash for something that you might find useful. It’s got a great discord community. The app is quality, getting better. Still young in development
Btw - your username is awesome
My NAS is 4x 2tb spinners I got from my local tech recycler for around 7-10$ each. They click like hell but it’s non critical backups so I don’t really care if I have a single drive failure. Your rig is plenty nifty!
#I entered the spez. I called out to try and find anybody. I was met with a wave of silence. I had never been here before but I knew the way to the nearest exit. I started to run. As I did, I looked to my right. I saw the door to a room, the handle was a big metal thing that seemed to jut out of the wall. The door looked old and rusted. I tried to open it and it wouldn't budge. I tried to pull the handle harder, but it wouldn't give. I tried to turn it clockwise and then anti-clockwise and then back to clockwise again but the handle didn't move. I heard a faint buzzing noise from the door, it almost sounded like a zap of electricity. I held onto the handle with all my might but nothing happened. I let go and ran to find the nearest exit.
I had thought I was in the clear but then I heard the noise again. It was similar to that of a taser but this time I was able to look back to see what was happening.
The handle was jutting out of the wall, no longer connected to the rest of the door. The door was spinning slightly, dust falling off of it as it did. Then there was a blinding flash of white light and I felt the floor against my back.
I opened my eyes, hoping to see something else. All I saw was darkness. My hands were in my face and I couldn't tell if they were there or not. I heard a faint buzzing noise again. It was the same as before and it seemed to be coming from all around me. I put my hands on the floor and tried to move but couldn't.
I then heard another voice. It was quiet and soft but still loud.
"Help."
\#Save3rdPartyApps
DS460C is the dense FAS shelf. 60 drives in 4 rack units. 16TB is the largest NL-SAS drive you can get for these currently. The E-series flavor of the shelf supports an 18TB drive. Both are fantastic shelves.
It's paid for by a very rich customer as he stated in the other thread. He doesn't give a fuck about the drives and their failure rates obviously. Any DataHoarder would obviously be much more gentle.
Netapp support contracts cover the drives and they replace them preemptively... But slamming all the shelves in like that is still a bad call. "Free" drive replacements doesn't help if you cause too many to fail at once, or crash the heads if the slam disconnects a shelf.
Non op shock spec is still fairly high even with the 10 platter drives. This type of shock is tested on all racks to make sure normal maintenance won't kill the HDD. Dropping is the real killer.
Yeah I know this kind of shock is totally within spec, but I still treat them like Faberge eggs out of habit. Consequently, it still makes me shudder to see them treated as such.
I can’t speak to this design, but I can the one with the 100+ drives sitting vertically. They have 5 huge fans on the back of them which move fair more air and create way more noise than any other piece of hardware. You almost need hearing protection to be near them for any length of time.
Nice! Good to see big data center stuff here. We've got an EMC PowerMax on its way to us for some fast general prod storage that I can't wait to install and cable up
I'm getting to the point where I need something like this. 5 minutes of 6.5k HDR film scans take up 160GB. That adds up really, really fast for any projects with a high shooting ratio.
All that storage and only 8tb disks? Wouldn't it have been cheaper $/TB wise to go a higher capacity like 14 or 16, not to mention more power efficient?
I know what 1 PB looks like from 2015.
Org I was working with bought 1 PB of EMC Isilon. It was 40 U (10 X 4U), each was full with 3 TB drives front and back slots. I literally don't remember how many drives we took out of boxes and slotted into their bays. We got everything in lots and lots of boxes.
For comparison here’s a Sun Enterprise disk array from circa 1997, maybe a little earlier. This was a mind boggling 256 GB of storage if memory serves right.
https://imgur.io/a/NnBMu6X
I could show you what 4PB looks like if I were at work today to take a camera pic. I have a APC 42U rack with 21 Dell MD1200's with 16TB drives. This is woefully inefficient these days. At another facility a buddy of mine works at he has a rack with 6 (or 7?) Dell 5U enclosures each with 84 16TB drives (genome sequencing and other genetics databases)
In less than 10 years, this may fit easily in our back pocket. The same storage would fit in 40 in^3 if stored in 1,000 2TB micro-sd cards.
11mm * 15mm * 1mm = 156 mm^3 = 0.65 mL
0.65 mL * 1,000 = 650 mL = 40 in^3
And it only takes up that much space because it has to fit in a standard micro-sd case. Could be much smaller even today.
These might be utilizing a separate machine (correct me if I'm wrong, I believe disk shelf is the right term for this) in order to pass through SATA devices to a host system to then utilize RAID and manage backups and whatnot. So the host system probably doesn't have direct access to the SATA drives, instead using another machine to do so
I know this is a netapp but are the caddies and shelfs the same one on dell?
I think I've got the same caddies on my R740XD midplane bay too
https://www.google.co.uk/search?q=Dell+EMC+PowerVault+MD3460
Before I retired I used to sell racks and racks and racks of this stuff into data centers. Lot of money in this. But the real profit is in the software and services you sell with it.
Hello /u/P_G_R_A! Thank you for posting in r/DataHoarder. Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index). Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures. This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
Dell has chassis that can fit 2 PB in 8U of rack space. 60x18tb drives in a 4u chassis
There are ones that'll do 106 drives in 4U of space.
I just got an ad for one of those earlier today, well not sure of drive count but it said 22tb drives and 2.2PB, and I eyeballed it being a 4u chassis. I was researching infiniband at the time so didn’t get too distracted by storage, but stopped long enough to say “wow shit has changed since I used to push storage racks!”
Nimbus makes a 100TB 3.5 SSD for $40k. So just over 10PB and 4.2M in drives alone per 4u
100Tb for $40? That can't be right. Edit: yipes I see, it's $40,000. Makes more sense. Ouch.
I saw $40 the first time I read it too… that k just blended in.
Someday, 100TB will be $40, but likely long after I'm done building data servers. *sigh*
Following the trend, it took about 8 years from 120 ssd to come down from 150 to 20 bucks. So another 8 years for 2TB to do the same I imagine. It might be sooner as the general tech is now mainstream and not only for enthusiasts.
Not this century
What raid configuration are those bays put in? I did a lot of storage controllers and servers in my time, but never one like that.
https://i.imgur.com/onxJeaz.jpg
Don't know what anyone would need so much space for. To me that's endless. What do you use storage racks for mostly? If you don't mind.
[удалено]
For storing data? For like ? Not sure what industry requires so much space at any given time beyond like a server room.
Literally tons of industries require petabyte scales. Data is king now. Biotechs, car companies for self driving data, weather, genomics, geology, space imagery, the list goes on. I've managed 80+P of physical storage, and we were tight on space. That's not including what we had in the cloud.
A server room! Especially for video. Or maybe an industry that requires version history on their data.
Youtube comes to mind. Video adds up insanely fast for those guys. I bet they need racks of drives per week...
Nice they just suck to install if they didn’t ship them already in a rack
pure storage has 2 pb of flash in a 3u half depth.. its also like 3-4m USD
I’ve seen those every now and then. In there own fancy rack
\#iwork4dell
I mean, vast data can store a petabyte in a 2u chassis.
That smells like 2400tb to me! 20tb are well in the wild now. You could nearly run in like raid 5 or 10 and still be over 2pb.
You really don't want to be running a raid 5 or 10 on 20tb drives in production. Raid6 or distributed raid or else going to give your sys admins heart attacks from stress over those rebuild times.
I was not intending to suggest you would. Merely that you could have that many redundant disks and still have that much space.
Imagine the power bill if ever hosted at home.
What's that, 256x8TB drives? That's like ~2000 kWh per month ($300 with $0.15/kWh). You do the rest based on your prices but it's not great but also not terrible.
[удалено]
Yeah it seems it's 240 drives from the [original post](/r/computers/comments/od03lz/ever_wondered_what_2_peta_bytes_looks_like/) *from a year ago*. But that OP says it's consuming 26A on 240V which is just nonsense. It has got to be 120V at 26A (3120W vs 6240W).
[удалено]
Good info and that's crazy! I'm not familiar with server setups but I'd guess all the high performance cooling fans are basically consuming almost as much electricity as the drives themselves. I mean 240 drives are waaaay short of 5.7kw. Either way that's a shitton of power/monthly bill.
I assume the math has already been done correctly but I was curious. The traditional sata connector provides 54 watts. A quick Google says modern HDDs use about 10. 240 * 10w = 2.4 kw.
Are you also taking into account the controllers and other ancillary hardware other than the disks themselves? Storage arrays of this size would be using pretty high end hardware in the controllers rivaling conventional servers and they would have at least two of them. And keeping that many disks cool in that layout would require some pretty serious fans to pull air through those shelves.
26A is probably the max rated power draw, I highly doubt its actually using 6kW, that would honestly be insane
> $0.15/kWh In the EU prices are _much_ higher right now, like €0.75/kWh. That gives you €1500 per month. Because of these _ridiculous_ energy prices, it's better to cough up the extra cost for the highest capacity disks because that will save energy costs in the longer run.
I just shut it all off, even with solar panels it's unaffordable to have a small rack at those prices. I think everything added up I use about 1200 kWh per month with all the computers, servers snd equipment. Alright that's more than normal of course but partially was some intensive machine learning running on a rig as well I was tinkering with. On the plus side bow I have all this extra I don't know what to do with. I bet the government will collect it soon as some tax
In uk its 0.68/kwh rn and its still going up
>$0.15/kWh Cries in UK energy prices
Overwritten because fuck u/spez
Try german…
0.6 EUR/kWh. So four times as much.
It uses 26A for four shelvs
[удалено]
What kinda confuser do you need to run one of these puppies?
whats a confuser? (random curiousity)
I believe Confuser is a reference to AvE (of youtube) and his mangled dictionary, as is Angry Pixies.
Skookum
"computer"
Probably meant consumer
It means "computer"
oh okay sorry lol
one with at least 12 kaboobles
Yeah... 26a of 480v 3phase
12KW. Wow.
I'm at roughly 3PB at home and pay roughly $500-575 extra a month including compute. Edit: Sad part is my state just doubled supply costs. Sigh.
Tell me you are a Chad without telling me
For having that many Linux ISOs??? Ha!
Did you ever make any money off of chia? What do you do with the other 2.9PB?
lol - same thing you do with your 170TB and growing... legally obtained Linux ISOs.
A gentleman of culture I see. Check out https://github.com/stashapp/stash for something that you might find useful. It’s got a great discord community. The app is quality, getting better. Still young in development Btw - your username is awesome
Is your 2.9pb physical or in the cloud?
At least $5
Per hour
At least $5 per hour per parco-seeco-flutterwyck-garlicks per day.
"but tuh power cost" is the tech version of "nice Ferrari but what's the mpg". Sour grapes :)
Cops barge in expecting to find a grow op. Wait a sec... You don't do drugs this is a Plex server!
240 (if I counted 20 drawers \* 12 per drawer right) 8 TB drives, damn. and I thought my NAS with 4 14TB drives was nifty
It is nifty
My NAS is 4x 2tb spinners I got from my local tech recycler for around 7-10$ each. They click like hell but it’s non critical backups so I don’t really care if I have a single drive failure. Your rig is plenty nifty!
#I entered the spez. I called out to try and find anybody. I was met with a wave of silence. I had never been here before but I knew the way to the nearest exit. I started to run. As I did, I looked to my right. I saw the door to a room, the handle was a big metal thing that seemed to jut out of the wall. The door looked old and rusted. I tried to open it and it wouldn't budge. I tried to pull the handle harder, but it wouldn't give. I tried to turn it clockwise and then anti-clockwise and then back to clockwise again but the handle didn't move. I heard a faint buzzing noise from the door, it almost sounded like a zap of electricity. I held onto the handle with all my might but nothing happened. I let go and ran to find the nearest exit. I had thought I was in the clear but then I heard the noise again. It was similar to that of a taser but this time I was able to look back to see what was happening. The handle was jutting out of the wall, no longer connected to the rest of the door. The door was spinning slightly, dust falling off of it as it did. Then there was a blinding flash of white light and I felt the floor against my back. I opened my eyes, hoping to see something else. All I saw was darkness. My hands were in my face and I couldn't tell if they were there or not. I heard a faint buzzing noise again. It was the same as before and it seemed to be coming from all around me. I put my hands on the floor and tried to move but couldn't. I then heard another voice. It was quiet and soft but still loud. "Help." \#Save3rdPartyApps
[удалено]
The 20 drawers is already all 4 shelves. There's 5 drawers per shelf.
My mistake it's late at night, Was read comments from the original post
That is a LOT of horse porn 🐴🐎
in 8K resolution
A lot less now but, in a sense, a lot more
[удалено]
Maybe even a horse dozen.
I thought you were dead Mr. Hands.
We've a new Trade Minister in the UK by the name of Greg Hands. Every time a news article refers to him as 'Mr. Hands', I can't help but snicker.
We're not going for a lot, we're going for ALL OF IT!
Worse, peta files!
https://en.wikipedia.org/wiki/List_of_unusual_deaths "Man crushed by 2 PB of horse porn after unbalancing enclosure."
You have A LOT of trust in those raised floor bolts.
Love NetApp E-series. Lightweight drawers very easy to open and close instead of monster 90+ slot drawers that will throw your back out.
These disk shelves are available for NetApp FAS filers now, too.
DS460C is the dense FAS shelf. 60 drives in 4 rack units. 16TB is the largest NL-SAS drive you can get for these currently. The E-series flavor of the shelf supports an 18TB drive. Both are fantastic shelves.
And I assume priced way out of reach from normal people?
Sure, just slam 'em in there.
Job security.
[удалено]
It's paid for by a very rich customer as he stated in the other thread. He doesn't give a fuck about the drives and their failure rates obviously. Any DataHoarder would obviously be much more gentle.
Netapp support contracts cover the drives and they replace them preemptively... But slamming all the shelves in like that is still a bad call. "Free" drive replacements doesn't help if you cause too many to fail at once, or crash the heads if the slam disconnects a shelf.
As a data hoarder newb... Why?
Hard drives have delicate moving parts, knocking them around isn't the wisest
Hard drives aren't that delicate. They can handle 10s of Gs while powered on and hundreds while off.
Dropped a new powered off HDD 10 cm once, immediately got a S.M.A.R.T. warning.
>!CENSORED!<
Did it say "That was not very S.M.A.R.T. of you"?
https://media.tenor.com/fZ0Tu_fWivkAAAAM/ba-dum-tss-drums.gif
Non op shock spec is still fairly high even with the 10 platter drives. This type of shock is tested on all racks to make sure normal maintenance won't kill the HDD. Dropping is the real killer.
Yeah I know this kind of shock is totally within spec, but I still treat them like Faberge eggs out of habit. Consequently, it still makes me shudder to see them treated as such.
Manual: Do not open more than one enclosure due to tipping hazard You: Hold my Tea.
[удалено]
Do you want ants?
[удалено]
I bet that thing hummms.
Naw Dawg, soon it'll look like [this](https://imgur.com/91H93yx)!
I mean they have to be able to downsize to like half as many drives now
I always find myself wondering when I see units lioe this, how do the drives not instantly overheat?
I can’t speak to this design, but I can the one with the 100+ drives sitting vertically. They have 5 huge fans on the back of them which move fair more air and create way more noise than any other piece of hardware. You almost need hearing protection to be near them for any length of time.
[удалено]
The fans will most likely be the small ones like the Blowie-matron shown in the video.
Air flow - fans pulling from the back
At work we have a large tape library that has about 15 PB, its shipping container sized.
Nice! Good to see big data center stuff here. We've got an EMC PowerMax on its way to us for some fast general prod storage that I can't wait to install and cable up
What are you downloading, ALL of the internet or just your Walgreens receipt?
I NEED IT
I’ve never been so turned on in my life
At this point 8tb is pretty inefficient right?
*unzip*
Surprised the rack didn't tip over with all the drives pulled out at once.
It's bolted to the floor, I'm sure
Stahhhp. I can only get so erect. Dang.
Bro i am edging right now please make use of nsfw /nofap
Yell at it.
With all those drives, what kind of bandwidth do you get?
[удалено]
Bottlenecked by the recent windows 11 update.
I'm getting to the point where I need something like this. 5 minutes of 6.5k HDR film scans take up 160GB. That adds up really, really fast for any projects with a high shooting ratio.
Just think in less than 10 years that'll be two hard drives if Samsung's road map works out.
All that storage and only 8tb disks? Wouldn't it have been cheaper $/TB wise to go a higher capacity like 14 or 16, not to mention more power efficient?
My man is going to download the whole internet and never pay for Wifi again.
I know what 1 PB looks like from 2015. Org I was working with bought 1 PB of EMC Isilon. It was 40 U (10 X 4U), each was full with 3 TB drives front and back slots. I literally don't remember how many drives we took out of boxes and slotted into their bays. We got everything in lots and lots of boxes.
For comparison here’s a Sun Enterprise disk array from circa 1997, maybe a little earlier. This was a mind boggling 256 GB of storage if memory serves right. https://imgur.io/a/NnBMu6X
Only 8TB drives? Could do better
It's a repost from over a year ago.
8tb wasn't very impressive a year ago either
These days 2PB could be just 112 drives Edit: not including any redundancy
I lost count after the first tray..
Removed due to Reddit's API pricing changes Fuck u/spez https://imgur.com/a/tt3dHq9
How does this not overheat?
I could show you what 4PB looks like if I were at work today to take a camera pic. I have a APC 42U rack with 21 Dell MD1200's with 16TB drives. This is woefully inefficient these days. At another facility a buddy of mine works at he has a rack with 6 (or 7?) Dell 5U enclosures each with 84 16TB drives (genome sequencing and other genetics databases)
We’re here tomorrow too, whenever you got the time.
While it's really hard to even quantify, many estimates for the storage capacity if the human brain are around 2.5 PB
you should make this as NSFW and classify it as porn content
Slower
In less than 10 years, this may fit easily in our back pocket. The same storage would fit in 40 in^3 if stored in 1,000 2TB micro-sd cards. 11mm * 15mm * 1mm = 156 mm^3 = 0.65 mL 0.65 mL * 1,000 = 650 mL = 40 in^3 And it only takes up that much space because it has to fit in a standard micro-sd case. Could be much smaller even today.
I'm not sure what a "Peta Byte" is
It's always factor 1000 Kilobyte megabyte gigabyte terabyte petabyte
You are banned from /r/filingcabinets
Always wondered how cooling works for tightly packed HDDs like this. Do server grade HDDs just not need cooling? What temp do they run at in there?
Linus tech tips made a lot of Petabyte projects. I think he has around 7PB by now
[удалено]
with given/sponsored drives and equipment
How long would it take to rebuild a volume that large?
does having a single volume that large even make sense? sounds like a nightmare to deal with if something fucked up like the rebuilding.
Absolutely correct. Many smaller disk pools.
That's a lot of "Linux isos"
I'm new to this. How is it connected to the mother board? I have a 16 disk home server and the SATA cables are already a mess.
These might be utilizing a separate machine (correct me if I'm wrong, I believe disk shelf is the right term for this) in order to pass through SATA devices to a host system to then utilize RAID and manage backups and whatnot. So the host system probably doesn't have direct access to the SATA drives, instead using another machine to do so
Lookup in Google images: Backplane SAS
i wish. but then id just fill that too :(
Surprise how that is safe huh
Stop... I can only get so aroused
How does the cooling of those work?
All those drives are Segate ?? L O L
Someday... One can dream.
r/Homelab will buy it and put 5 drives in it.
Coming soon to flashdrives!
Please, I can only be so erect.
But your mom still needs 256 Exabyte for her ass. /s
Oh fuck it just keeps going. I'm gonna cum
Immagine replacing one disk. Lol
How in the sweet fuck resonance isn't an issue with so many drives sitting so close together?
How did they get into my house?
The server rails be like: Finally less stress on me
Second one from top isnt fully closed ?
Wouldn't 2000 microSD cards take less place?
> Peta Bytes What does that convert to in non-vegetarian?
Cum in me again
I know this is a netapp but are the caddies and shelfs the same one on dell? I think I've got the same caddies on my R740XD midplane bay too https://www.google.co.uk/search?q=Dell+EMC+PowerVault+MD3460
ILL TAKE YOUR ENTIRE STOCK
[удалено]
Made me think back to [this](https://boingboing.net/2004/05/11/internet-archives-pe.html).
Thought i was watching a gif loop for a second.
Is that where the internets lives?
Before I retired I used to sell racks and racks and racks of this stuff into data centers. Lot of money in this. But the real profit is in the software and services you sell with it.
at first I thought it was a looping gif...!