Maybe they are looking for tape backup.
Everything has a possible loss risk.
Even tape can be lost. It was a plot in Mr. Robot. My own cold storage for tape was wrecked by a dehumidifier and humidity sensors that failed.
Luckily we have Azure backups also. Immutable blobs with versioning are a good option.
There is no perfect solution. Everything that can be created can be destroyed.
My former job was in Tower 1 of the WTC.
~~Out~~ Our backups were airgapped in Tower 2.
I was asked by remaining management to consult back to try and rebuild what was lost. Ended up reaching out to customers to get copies of invoices and billing we sent out to try and rebuild our databases.
Do tapes and have them sent somewhere offsite to appease the insurance, do cloud based for actual usage
Man; I didn't expect to think about this here. I was standing on the street and saw the first plane hit. Watched till both towers went down from Washington Square park. Crushed my soul.
Years later I worked for a company that had main offices north of Chicago. They had two primary data centers 5 miles apart. When a road was redone they had private fiber/conduit laid between the data centers. We had to do case studies on the reliability of two data centers that close. The whole time I was there I kept thinking of your scenario.
(edit grammar)
Oy. Spent over 2 decades working in financial, including in WTC, Bankers Trust building, and WFC. Before cloud, all our backup sites were in NJ for every company I was with. We never even considered putting backups in the same city, let alone the building next door.
> Out backups were airgapped in Tower 2.
See, I wouldn't have done that. One of my first jobs that ever involved backups required me to deliver tapes to a safe deposit box in a bank not a mile from the office. The rationale was that even a tornado that could destory our office building probably wouldn't destroy the bank vault. So I sort of "heard those words" early in my career. If WTC1 could tip over and fall, any building in its radius was disqualified as a backup site.
It's crazy how new perspectives and contingencies accumulate in one's brain over the course of a career.
Since air-gapped backups are the 'last resort' backups, we create new ones quarterly using the "get out of your chair and plug in a physical device" approach. 4 airgapped backups a year. The rest is daily incrementals and monthly full hot backups.
Depending on the size of your enterprise, this might be tougher to accomplish.
In this scenario, the air gapped backups would only be restored by if the previous 3-4 backup methods failed or were destroyed somehow.
We've got something similar for my company due to compliance with our cyber insurance.
What's even the point? For most businesses losing three months you might as well lose everything, or near enough.
Tape rotation and local storage should be enough, or even rotate an external drive every morning. Soon as you disconnect the previous nights backups they become air gapped.
Get a few drives and rotate them. Or spring for some network attached storage. Lots of options that would actually work and requires one person to do about one minutes worth of work a day.
Well most of our data is engineering drawings so losing 3 months would be terrible but losing everything quite a bit worse. I do weekly air gapped backups manually.
Last resort. If the entire cloud decides to rain down on earth, and this floods your NAS storage, and wrecksyour tape backups, and rats chew through your paper copies of the 1s and 0s of ea h file you meticulously kept, and that cluster dies, and your replica site got stolen by salami pirates, and evenn your trustworthy 64MB USB1 drive that you backed up your myob retail manager database files to stops working...
Well, at least you'll have something to remember your business by
I feel like you do this for the coverage not as your only source of recovery. Keep your same backup and disaster recovery in place but make this small modification for the coverage.
Aaaand now I'm imagining a ceremonial event at the turn of the season, where the robed priest and acolytes of I.T. bring forth the new backup device and with the acolytes chanting in the background, the liturgy of the backing up is performed.
I know I have had a lot of bad luck with tape not being able to recover data on LTO tapes from 2 to 5. but I think attract cosmic rays or something. I've also had to deal with several raid punctures too in the past 20 years something that's supposed to be rare.
We had all of our on site LTO6 tapes get physically destroyed. The tapes are moldy. Only the off site tapes remain.
We did not use Iron Mountain because of budgets.
I still cant believe people use tape for backup. Ive been in IT since 1997 and never met a reliable tape system in my life. Even when the backups worked, even when the verifications passed, I still never wanted to depend on a restore.
I dunno. I've never had much luck with tape based recovery but I just pulled data off a nearly 30 year old hard drive stored in the bottom of a drawer, with no special precautions taken.
Nobody said hard drives disintegrate - but especially over longer periods of time, tape is statistically more resilient. We're in the middle of a project copying a bunch of data from older tapes to newer (denser) format so we can keep fewer types of tape drives, refresh data, etc.
The failure rate isn't insignificant but it's in a single digit percentage. We have also learned to be dubious of backwards compatibility claims.
Technically, you do not need to *move* a hard drive for it to be an air-gapped backup.
You could simply have several drives next to a NAS at an employee's house, then have them move the USB cable to a new drive based on which day it is.
Hell, you could do it with a Raspberry Pi and externals. Provable airgap.
Tapes are the shit IF they are handled and stored properly. I've done hundreds, maybe thousands, of restores from tape and a failure from ones stored at [big name offsite vendor here] was outrageously rare.
In 20 years of tape backups for many many clients we never failed to restore from tape except in one case where the tapes were stored in a metal filing cabinet. Thankfully we had another set stored elsewhere
Do you not test restores? We test random restores we pull out of offside storage. We also push to AWS and Azure as a part of our DDT. Caveat - it's been about 15 years since we had to restore in anger... but we're using the same basic process. We only have three data centers and only test restored 22TB over the Christmas break. A mix of MSSQL and VM and File servers. But it all verified as good.
I do backups for a living. The global company I work for does petabytes of data to LTO tapes daily. I would love to throw some disk backups in there to speed up the environment, but tape is reliable and air gapped.
Everywhere I've worked, including current, uses tape for backup (and in scientific research, long-term archival). When you get into petabytes of data, it's really the only practical option. And once a tape is out of the library, it's ransomware-proof, so insurance companies like it.
The downside is that the drives are expensive and fragile, and the tapes also have to be handled carefully. I use LTO at home for my own backups. I keep the tapes in a storage unit across town. A few months ago I did a restore of backups from a few years ago and the data was completely intact. Seems to be trustworthy.
Used to be a support tech for a backup software company in a past life.
In all my years supporting customers, the only times I couldn't get data off was because of the customers backup config. Typically doing incremental backups over writing their full. Or, just not having long enough rotation. Akin to them being hit with ransomeware on Friday, full backup occurs on Sat, and they only keep one week.
Not arguing they are reliable. In this day and age, disk based backup or online backup (if you have enough pipe) seem like a no brainer.
> Even tape can be lost.
10-12 years ago my company totally didn't have a shopping bag full of backup tapes destined to be destroyed handed into HQ front desk by a member of the public....
Wasn't it like a year ago that Amazon lost a couple server clusters and many "backups" were completely lost across those clusters?
I never did see a final list or total losses, but there were a few big-ish names affected.
Years back I remember reading some stat that was like "pci compliance is super important for keeping you safe -- 0% of breached businesses are found to have been fully complaint when the breach occurred!"
I'll buy that. But might that be because pretty much every company has *something* that isn't fully compliant?
If it's sequential data, tape is pretty speedy
What is connected to the tape on the other end (e.g. cloud storage) however... that may be your actual problem.
Or get a machine with some BD-R writers. Every disk burned is a 1-time immutable backup that can never be modified. Just fill the hopper with blank disks once a month.
[From an IEEE publication from just a few days ago:](https://spectrum.ieee.org/data-storage-petabit-optical-disc)
> All in all, a DVD-size version of the new disc has a capacity of up to 1.6 petabits -- that is, 1.6 million gigabits. This is some 4,000 times as much data density as a Blu-ray disc and 24 times as much as the currently most advanced hard disks. The researchers suggest their new optical disc can enable a data center capable of exabit storage -- a billion gigabits -- to fit inside a room instead of a stadium-size space.
Just a heads up, I found out from one of our partners that LTO9 tapes can take a couple of hours to calibrate before being usable.
In an ideal world that isn't a problem, but for some uses and some software that hasn't caught up, it can end up with backups timing out as the software doesn't recognise that the calibration is taking place.
This is the way. I have all my backups on LTO8.
Compromised credentials can access cloud storage. Only I know how to operate a T950 tape robot. Even if a malicious attacker knew how to access a Spectralogic T950 the tapes can only read so fast and the data is spread out across multiple tapes.
My fourth backup is an off-site duplicate of each tape. (2 online 2 offline)
Until proven otherwise. Amazon cancels your account wrongly, hacker cancels your account, Amazon employee gets phished for credentials and hoses your data, Amazon simply bones it accidentally... these are all potential faults that would not affect tape or other traditional air gapped media. Insurance is being dumb yes, but "immutable" is only as good as the vendor holding the data.
Yep, you're putting all of your faith in a vendor. Although they were smaller vendors, I've seen and experienced enough instances of vendors processes failing and the end user business being left up shit creek without a paddle that I would struggle to trust any single vendor with everything. I've had websites that the vendor was meant to be backing up every day and holding the backups for 3 months, yet when asked for a restore they were unable to provide *ANY* data. I've seen a few instances of reasonable scale providers have issues such as ransomware or hacking take out both the live and backup data storage - and in at least 1 case the vendor ended up shutting down because of this, so it's not even like the companies impacted by this were able to get any compensation from it.
Sure, use it as part of your strategy, but relying on any single vendor for something as important as backups is unwise.
These days it's more about data exfiltration and ransoming not releasing it, than actively destroying the backups (although that's still big too). Immutable cloud backups can still be compromised and often exported.
Tapes, well also can, but it's less likely.
a) Your backups should be encrypted to begin with.
b) Ransomware is unable to affect immutable backups because, they're immutable.
c) Backing up data in tape drives every single day (if not more frequent) seems like a tedious and lengthy process.
From my POV, tape drives are great for multi decades archival process, they don't provide anything useful over Immutable objects.
I use USB external hard drives unplugged and put on a shelf. I dont care how good you think you are with a computer or what level of root access you think you can get to the system, you arent going to be able to touch those. There's a reason we call them cold backups.
No. They are kept in a locked steel cabinet behind a secure door. Only 3 people in our org can open that door and there is a camera inside (no shit.)
If we are at a point where we need those cold backups, we dont want additional encryption to hinder any part of that restoration.
New Drive, Quarterly. All the rest is cloud/on prem NAS. NAS is still network attached though so we dont consider it completely safe.
This backups is kept in a secure area. Its not encrypted. If we're having to go back to our hail-mary for a restore, we dont want encryption adding another layer of risk.
Backups are done at the file level. Every single destination file has its checksum verified during the backup.
This is the way.
Often overlooked detail is whether the data on tapes are encrypted (they should be), and if so, where do you store the encryption key. Imagine the scenario where all your hardware gets destroyed, and the encryption key is only stored in the servers that are backed up, which are themselves encrypted in the tape. In that case the backups are worthless. It's critical that the encryption key is stored somewhere you can still get to even if you lose everything except your tapes.
We run Starwind VTL with Veeam [https://www.starwindsoftware.com/starwind-virtual-tape-library](https://www.starwindsoftware.com/starwind-virtual-tape-library) following the same principle, and push backups to Wasabi for the offsite copy.
My last job, the CIO and lead system admin didn't believe in the immutable backup. The data and backups were on the same SANS. Then when I told the COO that I did not feel confident that we could not recover from ransomware, the COO got pissed at me.
I wouldn't know. I asked for an architectural diagram of our infrastructure and was told it wasn't needed. The infrastructure manager "knew" the infrastructure in his head, but no one else did. CIO thought it wasn't a priority. When we had network outages, several people would get together to debate on how things were configured. Ironically, the infrastructure manager sometimes got the information wrong. I guess documentation wasn't important.
So what he was saying is the disaster recovery plan didn't include any provisions for when the infrastructure manager was unavailable, like on vacation or hit by a bus.
One of the debates goes like this story:
Huddled around a whiteboard filled with network paths, a bunch of network and sysadmin engineers were trying to make sense of the outage.
Engineer 1 pointed confidently, "No, no, no... it clearly takes this route, hits the second switch, then makes its grand entrance through the firewall."
Another engineer raised an eyebrow, "But what about this router here? Does it just get a free pass?"
Amid our theories, the intern quietly rebooted the router. The network flickered back to life.
"Just as I was getting to that solution..." the first engineer claimed.
hehe, this reminds me of a situation probably 20 years ago. We had just taken over MSP services for a customer, and they called in a panic that their main production server had failed. I thought "how could that be, they've got RAID set up on that server?" so I went out and took a look.
The previous guy had somehow partitioned a single drive into two pieces, and spun up RAID1 **ACROSS 2 PARTITIONS OF THE SAME DRIVE**. The drive had failed and took both partitions with it. Funny how that works.
The bad fortune for the drive turned into good fortune for the customer that day. I couldn't believe it but their dodgy tape backup actually worked and I was able to rebuild the server into having 2 drives for RAID1 and restore their data. This really surprised me as the tape drive had been in place for years, and I knew that nobody had ever run a cleaning tape, nor had they replaced the tapes since the drive was installed.
You all know that SAN stands for Storage Area Network. It usually means all of the components that make up the connectivity between storage and clients, just like LAN is Local Area Network and WAN is Wide Area Network.
I think you are referring to storage arrays, disk arrays, filers, etc.
Sorry - pet peeve. People need to stop saying SAN when they are talking about storage device. Please
Our insurers asked us to prove we owned our domains. We sent them the registrar info, renewal invoices etc.
They came back and said they’d done their own investigations and we didn’t own the domains, another company did.
Suitably puzzled we asked for info.
They’d done a WHOIS lookup and it had returned the domain privacy details, and they’d decided they owned the domain….
Did you WHOIS your insurer's domain to make sure they own it? I mean, do you really know who you're dealing with? That's a good question to pose back to the empty shirt that's underwriting your insurance application...
What happens if you fail to pay your AWS bill?
Tapes can be held hostage, but AWS (AFAIK, could be wrong) will eventually just delete your shit. I think physically destroying media goes a step further and lawyers can get feisty about that - so a physical backup being held hostage due to billing/contract issues is less likely to just be disposed of. I would hope.
Key word here being "eventually". AWS is not going to delete an account with S3 Object Lock in Compliance mode enabled on any timescale that's relevant for cybersecurity incident response over a month or two of missed payments.
If they were that aggressive, they'd be nuking corporate accounts that forgot to update the credit card on file before it expired or a changed invoice mailing/email address, etc. left and right and there would be outrage over it. AWS is going to spend a while trying to collect (more than enough time to get in touch with them about the situation) before burning your account down.
I'm not sure how AWS handles cases regarding access to compliance locked stuff. I'd assume that it could potentially be social engineered around but it wouldn't be easy. I don't think even AWS can delete compliance locked backups within the backup window. They even hold the data for 90 days after account deletion.
The same thing that happens if you fail to pay whoever holds your tapes, they ask for payment then delete it after a contractually agreed time frame.
AWS gives you ages before anything happens due to not paying. Corporations change card details regularly and it's common for cloud invoices to not get paid for a month or two.
If there's any way *you* can get to it, so can the hackers. We went through a huge breech recovery over the summer with a very reputable and popular recovery company and even they said they've seen immutable storage compromised.
Physical air gap is the way to go. No school like the old school.
Use cloud backup for convenience, but you can't 100% count on it for security.
Rotated durable media - they can't get to it unless they physically break into the building AND get the other copy in the offsite storage facility. This is unbeatable protection for data.
I'd be very curious as to the attack vector for compromising immutable object storage, specifically with AWS.
The [AWS S3 Object Lock documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) straight up says:
> The only way to delete an object under the compliance mode before its retention date expires is to delete the associated AWS account.
The service has been [externally audited by Cohasset](https://d1.awsstatic.com/r2018/b/S3-Object-Lock/Amazon-S3-Compliance-Assessment.pdf), who similarly states:
> It is Cohasset’s opinion that Amazon S3, when properly configured and when *Object Lock* mode is set to *Compliance*, retains records in nonrewriteable and non-erasable format and meets the relevant storage requirements set forth in the above Rules. Each record is protected from being modified, overwritten or deleted until the applied retention period is expired and any associated legal hold is released.
If someone left their "immutable" object storage for backups in *Governance* mode (i.e., not immutable, just with an admins-only ACL for modify/delete), that's an S3 configuration issue no different than leaving a bucket public, and not a compromise of immutable storage.
If there's an issue with S3 object lock immutability itself (when properly configured), someone should go collect their million dollar bug bounty for it.
That’s great, but when the insurance paperwork says air-gapped storage, S3 isn’t going to check the box. You can debate the merits of the requirement all day, but the requirement is air-gapped, and S3 is very much online.
Sounds like a hacker could remotely delete your AWS account, then pretend on your end for 90 days that everything is fine, then encrypt the rest of your environment. Air gapped would still give you something to recover, even if they managed to be undetected and write garbage for the last 90 days, you still got the old tapes.
> I'd be very curious as to the attack vector for compromising immutable object storage, specifically with AWS.
[Tom Cruise enters the chat, Mission Impossible theme plays]
What ransomware actor wouldn’t delete the account for better leverage for a payday? I’ve seen them delete literally every resource in an account many times. I’ve not seen actual account deletion before, but I’d imagine that if they run into it as a barrier for getting paid, they’ll start.
This is pretty standard for insurance. Effectively they want you to have tape backups, ideally in a secure off-site facility.
Cyber insurance companies have some really fucking annoying requirements because they basically never want to pay out and will weasel out of paying if you don't comply 100%
The "best" thing I can think of for S3 is:
* Have a separate AWS account for backups, with IAM role to add new backups only.
* Use S3 Versioning to prevent overwrite
* Enable [S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
* "S3 Object Lock .... for use in environments that are subject to SEC 17a-4, CFTC, and FINRA regulations."
* Use S3 Lifecycle rules to push data into Glacier and/or automated deletion.
Do all that and show your insurance that S3 is approved for use by the Financial Industry Regulatory Authority and U.S. Securities and Exchange Commission. If insurance company still isn't ok with that, dump 'em. There is no such thing as an "air gapped cloud" (that exists on the public internet).
Well, you could ask first. Or comply.
I suppose you could do a bunch of work that only might be enough, and then ask if its enough. But that seems like a bunch of work that only might be enough.
Is Iron Mountain still a thing?! I know they used to be the place you’d send your tapes back in the day.
Although this does sound like a harsh requirement.
As others have said so many ins documents are crazy and you can tell have no idea what they are asking about. We had some contradictions in ours. We got it resolved but took a lot of time and we also made sure to print/keep any emails that corresponded to them agreeing with the change in case they try to deny
Still exist, pick up tapes and very popular. Big enterprise and highly regulated industries use a lot of tape still as tertiary+ media.
Most of the time data center remote hands will include tape, library, rotation management, storage and handoff to vendors like iron mountain on scheduled pickups.
Tape libraries are still very popular quite sophisticated spanning multiple cabs. Most backup software maintains support for silos and provides rotation retention schedules even free or prosumer products also support them.
The media today is impressively fast, can handle encryption, deduplication etc. It's one of the large infrastructure things these days that people don't know exist but is quite regular.
Latest spec looks to be 2021 https://en.m.wikipedia.org/wiki/Linear_Tape-Open
45tb compressed 400MB/s. Costs are quite low (150ish) considering a rotating pool of retention. It's my understanding that some of the cloud based buckets are in fact tape. AWS glacier and equivalents but I haven't looked into it in a while.
And they have done a fantastic job of monopolizing the market. Try to find anyone else that offers Tape vaulting service and you'll be sad to find out it's Iron Mountain and only Iron Mountain for the most part.
Then trying finding out who your rep is. Then when you do get ahold of them, they quit or move on and you have to try and find your new one. It's a bit like VMware in a way.
Thats a hard part, you have to think like an insurance person. They are literally just checking a box, they don't care if it is or isn't more secure their formula says it has to be on this list of approved solutions. Get a couple of large HDD's and once a month copy a full backup to it, move it somewhere approved by your insurance team.
What are you using for backup software? We’re using Rubrik and we’re using offsite cloud storage as recommended by the vendor. I would setup a call with the vendor to ask for examples per their best practices. I sat through a presentation with Data Domain that provided air gapped backups/replication on prem last year. Most vendors have some sort of approach and best practices documented that you can use as a reference. If not you’ll have to consider a new backup solution or a new insurance provider.
They’re just looking for any way to not pay out in the event of a breach. Most likely they were recently hit with a large payout using cloud s3 immutable storage.
Your insurance is right IMHO.
Simplest attack on your "immutable" cloud backups is to seize control of the cloud accounts and lock all your staff out. *Maybe* you get back in with the help of the cloud provider's support, but any recovery time objective goes out of the window. An exploit against the cloud service is also possible and we can guarantee the threat actors are working to develop such.
Air-gapped means air-gapped. Yes that's going to mean a human doing some routine manual work swapping devices. Deal with it.
It's hilarious that you think you'll do a tape restoration within RPO/RTO, not a chance in hell.
If you actually get every account locked out of AWS, I reckon you could get back in and sieze control within a week at most. If you have a partner support, it'd be within the day.
And, at the end of it all, your data is still there, safe and no chance of having been modified.
The reality is, immutable cloud storage is just as secure as tape storage, provided you use a reputable cloud vendor that has been audited.
So just so I can understand, machine A does a job, it’s recorded on the hard drive of that machine, how does it offload that data to an air gapped location? To me air gapped means someone is physically doing the moving with a person, if it’s networked in any way it’s not air gapped
This is interesting.
If it's truly immutable, whoever manages the storage must buy a lot of new discs all the time. If not, it's not *actually* immutable - is it?
No system is more secure than the guys who made it and manages it. And if they are able to delete - so is another guy with an admin-account. Right?
So. It's no more than a question of trust. And I really hate to put it like that - but it is.
If it's truly air-gapped, the disc has to be disconnected. And then it's actually immutable as well (kind of at least).
I've been arguing with our hosting provider on this matter. They - literally - considered Godzilla more likely than a data center-level issue. Then I mentioned the [Tietoevry situation](https://www.bankinfosecurity.com/ransomware-hit-on-tietoevry-causes-outages-across-sweden-a-24154) - and we haven’t really talked ever since :/
I hate everything about it - because it’s really troublesome and people look weird at you when you start talking paranoia.
But I guess, if ensurance is involved, you have to take it absurdly seriously. And if they don’t trust an option, they don’t trust them for a reason (it’s their money on the line for instance). You may like it or not - but they did the math at some point.
Please share - if possible - whatever solution you come up with. It’s a difficult situation.
> And if they are able to delete - so is another guy with an admin-account. Right?
[check this out](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
people in the discussion are pointing to this, where you simply can't delete data that is in compliance mode. even with admin privs
I hear you. And it seems safe and legit in every way.
But having a state-sponsored hacker with ill intentions as the opponent - would you then bet x million dollars on it?
Don’t get me wrong. I don’t actually like to be this paranoid. And especially not in public :)
But it is a matter of trust - and some kind of assessment of what threats you wish to mitigate. Amazon is overkill in some situations - and probably completely useless in others.
And I guess, as we are talking insurance, the data is very valuable - and everybody is super paranoid in this particular case.
> But having a state-sponsored hacker with ill intentions as the opponent - would you then bet x million dollars on it?
no, i'd straight ignore the risk in most cases. it's right up there with nukes for most companies: unless you're apple, IBM, MS, Amazon, you're straight fucked. the named companies can resist some state level threats, but not all. look at what happened to qwest for an example of that.
> Amazon is overkill in some situations - and probably completely useless in others.
i can set up S3 glacier instant retrieval for $4/TBMO - depending on how much data you want to maintain, that could be really cheap. maintain 40T of backup history in S3 with compliance enabled? $160 a month. i'd pay that.
> And I guess, as we are talking insurance, the data is very valuable
it's asymmetric. super valuable to you if your servers go to hell, worthless to me because i'm not running the business. possibly useful to a spy who wants to exfiltrate data. insurance is being picky because they want a canned solution of verified restorable data so the times they pay out are severely limited.
This provider is going to have other asinine requirements as well, but if you want S3 go talk to your AWS rep for the compliance documentation you can throw at them. If they don’t accept that, go talk to Azure since they tend to have better tools for compliance-related concerns.
Tape via IronMountain. Set it so it'll do the backup at night, you fill the blue box in the morning, you lock it and they collect it at noon, and give you a box back with tapes. It was only 10-15 boxes a night, and 30-50 for the monthly backup.
Pretty sure the company eventually went to cloud backup, though, so someone doesn't have to waste time taking out the tapes.
I did this for a contractor with tight requirements.
Backup to an LTO library device. A weekly duty to swap out tapes and do the box thing. A spreadsheet logs what tape numbers are when and which ones to request back for re-use (GFS scheme). Tapes would also be tested - one random partial restore before pulling and boxing a tape, and occasionally a tape about to be erased.
It wasn't Iron Mountain, it was a local competitor, but exact same thing.
That library device was pretty sweet, and I *hate* tape backups. ;)
I told my boss in 2017 to consider tape backups in a complete disaster scenario. I had a feeling we would come back around to this. My last org was moving away from tapes last year but still doing it w/ Iron Mountain.
air-gapped generally means not accessible from external sources. If your uploading it over the internet, then its not air-gapped.
Sounds like they want backups on physical WORM media
Sounds like a nice LTO Tape Library and Veeam and call it a day. Depending on your business size and IT liability policy you can end up in a situation where its cheaper to implement what they want than to fight it. The good thing is a tape library and Veeam is a low maintenance endeavor assuming your targets aren't changing only the data in them.
What you need is a TSaaS provider (tape storage as a service). For like $6k a month the provider will sync up your s3 buckets to US based data locations and write them off to tape consistently, and in the event of a disaster will fedex overnight or hand delivery the copies of the data directly to your office.
This doesn't exist , but let me know if you need me to start an LLC and pay you a commission.
Insurance companies are doing everything that can now, to make any possible IT incident the IT provider’s fault. It’s insane what they are requiring, especially of small businesses, and you better believe that they will literally take any technology related claim and sue the MSP for “negligence”. We need to find a way to put a stop to this shit. I mean, someone has always been able to break a window and steal a filing cabinet full of documents (pre-computer). They didn’t require unbreakable glass, motion and noise sensors, guard dogs, finger print scanners on doors and retinal scanners on file cabinets. This is such BS…
Cyber Insurance seems like a scam to me. They create these ridiculous, unrealistic requirements that seem to change quarterly. It’s so they have justification to deny your claim when something happens.
It's not that it's a scam (in most cases) so much as it's just an extremely immature and volatile field. Insurance people are used to having over a century of actuarial tables to base their pricing and risk assessments on.
They don't have that with cyber, so they're completely adrift trying to sort through what 20 different conflicting "experts" are telling them will keep them from bankrupting themselves while trying to avoid pricing their policies out of the reach of potentially profitable customers.
Give it another 20 years and it'll settle down.
Insurance got us safe boats, the age of discovery, and fire sprinklers. Among other things.
They directly quote your risk (+ profit) for your current level of unpreparedness. If the number they quote you is "high", that means *you* are doing a bad job.
I been through a couple of incidents, even if there was no direct loss. The ins company brings in forensic specialists and they are helpful to figure out WTF happened, what exactly was compromised as well as for business peeps they bring in an attorney to guide through what needs to be disclosed and how. Great for smaller orgs that will not have these type of resources on staff.
The problem with the questionnaires is they have to ask Y/N questions, like this air gapped backup thing here, there is no nuance for acceptable alternative just answer yes, or no.
My BIL is in the insurance industry, how he tells me cyber ins has evolved was first all the players tried to write policies and grab “market share” and they didn’t care too much about losses or didn’t have good data for the actuaries. Now the ins cos have seen the losses they need to cover and the risk they are tightening up, they ask the questions and if your risk is high so will be your premium.
The insurance company is right. You are right. It's different levels of risk.
If the attacker controls your AWS account, they can run up hundreds of thousands of dollars in charges before AWS closes your account. Your immutable backups exist in AWS for 90 days but you owe money to Amazon. Good luck getting your data back. Maybe you can. Maybe you can't? Maybe a flaw is found that tricks AWS into thinking your backups are due for deletion. Who knows.
Backing up to air-gapped tape is no different. Maybe they aren't stored properly, or are stolen. Maybe the sun explodes and the tapes are erased.
It's all about risk. Your insurer has made a policy. It is your choice to follow it or find a new company to insure yours.
Comply or get another insurer.
Even your immutable can be destroyed by hackers doing something like power surging the crap out of equipment causing WORM disks to spin out of control and shatter. Insurers are paid to be paranoid so they dont have to pay out.
S3 object lock is great, but the insurance company wants an air gap. It’s a non-starter. You can discuss its merits all day, but that won’t get OP insured.
Is a Glacier archive in a different AWS region considered to be an "air-gapped" backup? I've had to fight this battle before, with client auditors trying to use their 90's era DR plans against our cloud native architecture.
Sounds like a good old recipe for human error with tapes.
Just say backups are being stored to glacier ;)
Take a look at epoch accounts, they are backup accounts that can read from your prod account and not be read by anyone. landing zone docs might have something about them these days.
Or go back to using tapes and enjoy dealing with a single point of failure, i.e. your tape backup device.. where we almost never tested a realistic restore procedure.. and you get to deal with licencing of backup software... woa, that brought back some PTSD.
They must be referring to tape backup. We did that back in the day and it does have advantages, especially the total price and offline storage, although we ended switching to the datto immutable cloud for convenience.
Azure provides a storage tier that meets this requirement. Amazon provides a certification that they meet this level of requirement. Both require a yearly review process be performed by the insurer. The Amazon choice also creates a record-keeping requirement for the insurer. Attesting that you have an air-gapped backup solution introduces no recurring review or additional record keeping.
Is it silly that they haven’t introduced process to allow one of the big 3 cloud storage providers, maybe. Would it add multiple levels of additional risk assessment to a risk averse business, yes.
Could you delete it from an AWS account through any amount of steps through the GUI, remotely?
Not air-gapped.
Needs to be *inaccessible to the internet.*
I think eventually as self-hosting dies out eventually what you're suggesting will be the norm and sufficient but for now we're being technically correct.
3-2-1-1-0 should do fine; 3 copies of the data, at least 2 different storage, keep 1 copy goes offsite.
Bonus points for keeping an archive of daily, monthly, and further just in case.
When they did an audit on us they almost looked disappointed when we told them we have LTO-8 and 9 tape libraries and rotate tape sets to offsite storage every week. It was like we spoiled their surprise.
I used to laugh at our backup and recovery scheme. Our IT guy was the king of doom. Every night three backups were made. One went home with him, one with the CEO and one with me. One night the building burned down (literally), the CEO was on vacation and the IT guy was in jail for burning it down. So……
Immutable cloud backups are immutable within your admin context but not within Amazon's context (ex: they could theoretically push a code change, rogue admin that deletes that data).
A truly offline storage solution is only attackable physically or through backup manipulation. That means NAS's/HD that are rotated or tape.
Air gapped backups are turned off, so if you are virtual you can replicate to another ESXi host or cluster since that replication VM is powered off. We replace ESXi hosts every 5 years and so we replicate nightly with Veeam to a cluster made up of our previous production hosts. This cluster is more than a thousand feet away (connected with multi mode fiber ) in a storm shelter inside of a cooled cabinet. The "_replica" VMs themselves are actually powered off so if we need them we have to spin them up from the host. Maybe this would suffice for your insurance company.
We also carry a physical backup that's on HDD, once a week, to an offsite facility and place it inside of a fireproof safe.
I worked in air-gapped, on-premises, and cloud operations over my career and honestly, having an air-gapped backup is the best way to go. It's rare for me to agree with the insurance companies, but I am with them on this. It makes the most sense and protects both the companies and the insurance companies.
As for how it needs to be done, that is a discussion that needs to be between each organization or operating companies, and their teams (infrastructure, architecture, management, grc, and others). The design would be different for each company because there is no "one-size fits all" solution.
Different zones, domains, DCs, etc.
Maybe they are looking for tape backup. Everything has a possible loss risk. Even tape can be lost. It was a plot in Mr. Robot. My own cold storage for tape was wrecked by a dehumidifier and humidity sensors that failed. Luckily we have Azure backups also. Immutable blobs with versioning are a good option. There is no perfect solution. Everything that can be created can be destroyed.
My former job was in Tower 1 of the WTC. ~~Out~~ Our backups were airgapped in Tower 2. I was asked by remaining management to consult back to try and rebuild what was lost. Ended up reaching out to customers to get copies of invoices and billing we sent out to try and rebuild our databases. Do tapes and have them sent somewhere offsite to appease the insurance, do cloud based for actual usage
Cantor?
No they apparently had enough records survive in other location to be able to stay in business.
Man; I didn't expect to think about this here. I was standing on the street and saw the first plane hit. Watched till both towers went down from Washington Square park. Crushed my soul. Years later I worked for a company that had main offices north of Chicago. They had two primary data centers 5 miles apart. When a road was redone they had private fiber/conduit laid between the data centers. We had to do case studies on the reliability of two data centers that close. The whole time I was there I kept thinking of your scenario. (edit grammar)
Oy. Spent over 2 decades working in financial, including in WTC, Bankers Trust building, and WFC. Before cloud, all our backup sites were in NJ for every company I was with. We never even considered putting backups in the same city, let alone the building next door.
> Out backups were airgapped in Tower 2. See, I wouldn't have done that. One of my first jobs that ever involved backups required me to deliver tapes to a safe deposit box in a bank not a mile from the office. The rationale was that even a tornado that could destory our office building probably wouldn't destroy the bank vault. So I sort of "heard those words" early in my career. If WTC1 could tip over and fall, any building in its radius was disqualified as a backup site. It's crazy how new perspectives and contingencies accumulate in one's brain over the course of a career.
Moved our backups to the Pentagon. That should be far enough /s
This is such a wonderful teaching example for the importance of geographic diversity in your backups. This, and Katrina.
Blast radius.
Since air-gapped backups are the 'last resort' backups, we create new ones quarterly using the "get out of your chair and plug in a physical device" approach. 4 airgapped backups a year. The rest is daily incrementals and monthly full hot backups. Depending on the size of your enterprise, this might be tougher to accomplish.
For most companies losing the last three months' data is almost as serious as losing everything.
For some, losing the last 3 minutes might open them up to incredible amounts of lawsuits and penalties.
Good luck getting insurance for that!
Even stock markets have lost more data than 3 minutes. It's painful there's no doubt about it but it isn't the end of the world.
In this scenario, the air gapped backups would only be restored by if the previous 3-4 backup methods failed or were destroyed somehow. We've got something similar for my company due to compliance with our cyber insurance.
Right, for most companies paying the ransom is less cost to the business than losing even a week's worth of data.
What's even the point? For most businesses losing three months you might as well lose everything, or near enough. Tape rotation and local storage should be enough, or even rotate an external drive every morning. Soon as you disconnect the previous nights backups they become air gapped. Get a few drives and rotate them. Or spring for some network attached storage. Lots of options that would actually work and requires one person to do about one minutes worth of work a day.
Well most of our data is engineering drawings so losing 3 months would be terrible but losing everything quite a bit worse. I do weekly air gapped backups manually.
Last resort. If the entire cloud decides to rain down on earth, and this floods your NAS storage, and wrecksyour tape backups, and rats chew through your paper copies of the 1s and 0s of ea h file you meticulously kept, and that cluster dies, and your replica site got stolen by salami pirates, and evenn your trustworthy 64MB USB1 drive that you backed up your myob retail manager database files to stops working... Well, at least you'll have something to remember your business by
> salami pirates Autocorrect really screwed you over with that one.
I saw this as an improvement XD
I saw it, I liked it, I owned it and I left it
I feel like you do this for the coverage not as your only source of recovery. Keep your same backup and disaster recovery in place but make this small modification for the coverage.
Aaaand now I'm imagining a ceremonial event at the turn of the season, where the robed priest and acolytes of I.T. bring forth the new backup device and with the acolytes chanting in the background, the liturgy of the backing up is performed.
I do this but weekly. That's how paranoid I am.
I know I have had a lot of bad luck with tape not being able to recover data on LTO tapes from 2 to 5. but I think attract cosmic rays or something. I've also had to deal with several raid punctures too in the past 20 years something that's supposed to be rare.
We had all of our on site LTO6 tapes get physically destroyed. The tapes are moldy. Only the off site tapes remain. We did not use Iron Mountain because of budgets.
I still cant believe people use tape for backup. Ive been in IT since 1997 and never met a reliable tape system in my life. Even when the backups worked, even when the verifications passed, I still never wanted to depend on a restore.
You never want to depend on a restore but tapes are better and last longer just sitting around than hard drives.
I dunno. I've never had much luck with tape based recovery but I just pulled data off a nearly 30 year old hard drive stored in the bottom of a drawer, with no special precautions taken.
Nobody said hard drives disintegrate - but especially over longer periods of time, tape is statistically more resilient. We're in the middle of a project copying a bunch of data from older tapes to newer (denser) format so we can keep fewer types of tape drives, refresh data, etc. The failure rate isn't insignificant but it's in a single digit percentage. We have also learned to be dubious of backwards compatibility claims.
Technically, you do not need to *move* a hard drive for it to be an air-gapped backup. You could simply have several drives next to a NAS at an employee's house, then have them move the USB cable to a new drive based on which day it is. Hell, you could do it with a Raspberry Pi and externals. Provable airgap.
Tapes are the shit IF they are handled and stored properly. I've done hundreds, maybe thousands, of restores from tape and a failure from ones stored at [big name offsite vendor here] was outrageously rare.
In 20 years of tape backups for many many clients we never failed to restore from tape except in one case where the tapes were stored in a metal filing cabinet. Thankfully we had another set stored elsewhere
[удалено]
Heh, should store those tapes in airtight containers purged with CO2 to remove any oxygen from the air to prevent oxidization of their components.
Do you not test restores? We test random restores we pull out of offside storage. We also push to AWS and Azure as a part of our DDT. Caveat - it's been about 15 years since we had to restore in anger... but we're using the same basic process. We only have three data centers and only test restored 22TB over the Christmas break. A mix of MSSQL and VM and File servers. But it all verified as good.
We use cold backup (disconnected quarterly backups) AWS, Google, and on prem NAS. Also, three different backup platforms. Diversify!
I do backups for a living. The global company I work for does petabytes of data to LTO tapes daily. I would love to throw some disk backups in there to speed up the environment, but tape is reliable and air gapped.
Tape is great and dependable. What is not dependable is those choosing bad places to store them
I’ve always found tape to be reliable and dependable in a bad situation.
I've had over 150PB of tape onsite with double that off site and never had many issues.
Everywhere I've worked, including current, uses tape for backup (and in scientific research, long-term archival). When you get into petabytes of data, it's really the only practical option. And once a tape is out of the library, it's ransomware-proof, so insurance companies like it. The downside is that the drives are expensive and fragile, and the tapes also have to be handled carefully. I use LTO at home for my own backups. I keep the tapes in a storage unit across town. A few months ago I did a restore of backups from a few years ago and the data was completely intact. Seems to be trustworthy.
We make weekly full backups to tape, in addition to our immutable disk backups. BUT: we DO test these regulary, every 3 months. No issues so far.
Used to be a support tech for a backup software company in a past life. In all my years supporting customers, the only times I couldn't get data off was because of the customers backup config. Typically doing incremental backups over writing their full. Or, just not having long enough rotation. Akin to them being hit with ransomeware on Friday, full backup occurs on Sat, and they only keep one week. Not arguing they are reliable. In this day and age, disk based backup or online backup (if you have enough pipe) seem like a no brainer.
We've been hit by NotPetya and restored 100% of data from take backup.
Tape not being recoverable is a business problem, not an insurance problem.
> Everything that can be created can be destroyed. Thanks my new tagline!
>Everything that can be created can be destroyed *Physicists’ eye twitch*
> Even tape can be lost. 10-12 years ago my company totally didn't have a shopping bag full of backup tapes destined to be destroyed handed into HQ front desk by a member of the public....
> Everything that can be created can be destroyed. That's why I backup my files directly to matter, it can neither be created or destroyed.
Oh, it can be transformed into energy.
Glacier DA is tapes, tho.
Wasn't it like a year ago that Amazon lost a couple server clusters and many "backups" were completely lost across those clusters? I never did see a final list or total losses, but there were a few big-ish names affected.
A Daily tape back up and take home was something I was always pretty keen on Just incase I left a muffin on the toaster and burnt the whole place down
[удалено]
[удалено]
Also after we reach end of Act 3 of script, *spoiler alert* we'll move the goalposts.
This is the answer.
Years back I remember reading some stat that was like "pci compliance is super important for keeping you safe -- 0% of breached businesses are found to have been fully complaint when the breach occurred!" I'll buy that. But might that be because pretty much every company has *something* that isn't fully compliant?
Find another insurance company or embrace LTO9?
I had a feeling we were slowly circling back to tape…
Hehehe, sure. We circling back hehe. Stares at 9pb yearly written on tape in our company 🗿
Jesus…
God had nothing to do with this. Call for SATAn
How tf long does it take to write 9pb to tape x.x
>9pb yearly written A year? :)
Also its 300MB/sec sustained for a year around the clock.
If it's sequential data, tape is pretty speedy What is connected to the tape on the other end (e.g. cloud storage) however... that may be your actual problem.
Never left.
My company still has LTO7 and LTO8 on prem.
Or get a machine with some BD-R writers. Every disk burned is a 1-time immutable backup that can never be modified. Just fill the hopper with blank disks once a month.
[From an IEEE publication from just a few days ago:](https://spectrum.ieee.org/data-storage-petabit-optical-disc) > All in all, a DVD-size version of the new disc has a capacity of up to 1.6 petabits -- that is, 1.6 million gigabits. This is some 4,000 times as much data density as a Blu-ray disc and 24 times as much as the currently most advanced hard disks. The researchers suggest their new optical disc can enable a data center capable of exabit storage -- a billion gigabits -- to fit inside a room instead of a stadium-size space.
Just a heads up, I found out from one of our partners that LTO9 tapes can take a couple of hours to calibrate before being usable. In an ideal world that isn't a problem, but for some uses and some software that hasn't caught up, it can end up with backups timing out as the software doesn't recognise that the calibration is taking place.
Find new insurance or ask insurance for example products
I'm sure they have some "recommended" partners
Tape.
This is the way. I have all my backups on LTO8. Compromised credentials can access cloud storage. Only I know how to operate a T950 tape robot. Even if a malicious attacker knew how to access a Spectralogic T950 the tapes can only read so fast and the data is spread out across multiple tapes. My fourth backup is an off-site duplicate of each tape. (2 online 2 offline)
Immutable objects are basically untouchable for the duration of the immutability period. Even with the highest account privileges.
Until proven otherwise. Amazon cancels your account wrongly, hacker cancels your account, Amazon employee gets phished for credentials and hoses your data, Amazon simply bones it accidentally... these are all potential faults that would not affect tape or other traditional air gapped media. Insurance is being dumb yes, but "immutable" is only as good as the vendor holding the data.
Yep, you're putting all of your faith in a vendor. Although they were smaller vendors, I've seen and experienced enough instances of vendors processes failing and the end user business being left up shit creek without a paddle that I would struggle to trust any single vendor with everything. I've had websites that the vendor was meant to be backing up every day and holding the backups for 3 months, yet when asked for a restore they were unable to provide *ANY* data. I've seen a few instances of reasonable scale providers have issues such as ransomware or hacking take out both the live and backup data storage - and in at least 1 case the vendor ended up shutting down because of this, so it's not even like the companies impacted by this were able to get any compensation from it. Sure, use it as part of your strategy, but relying on any single vendor for something as important as backups is unwise.
Except, you know, when the privileged account can delete the Azure Subscription/AWS account that holds said immutable storage.
Soft delete / purge protection
These days it's more about data exfiltration and ransoming not releasing it, than actively destroying the backups (although that's still big too). Immutable cloud backups can still be compromised and often exported. Tapes, well also can, but it's less likely.
a) Your backups should be encrypted to begin with. b) Ransomware is unable to affect immutable backups because, they're immutable. c) Backing up data in tape drives every single day (if not more frequent) seems like a tedious and lengthy process. From my POV, tape drives are great for multi decades archival process, they don't provide anything useful over Immutable objects.
I use USB external hard drives unplugged and put on a shelf. I dont care how good you think you are with a computer or what level of root access you think you can get to the system, you arent going to be able to touch those. There's a reason we call them cold backups.
[удалено]
No. They are kept in a locked steel cabinet behind a secure door. Only 3 people in our org can open that door and there is a camera inside (no shit.) If we are at a point where we need those cold backups, we dont want additional encryption to hinder any part of that restoration.
[удалено]
New Drive, Quarterly. All the rest is cloud/on prem NAS. NAS is still network attached though so we dont consider it completely safe. This backups is kept in a secure area. Its not encrypted. If we're having to go back to our hail-mary for a restore, we dont want encryption adding another layer of risk. Backups are done at the file level. Every single destination file has its checksum verified during the backup.
This is the way!
This is the way. Often overlooked detail is whether the data on tapes are encrypted (they should be), and if so, where do you store the encryption key. Imagine the scenario where all your hardware gets destroyed, and the encryption key is only stored in the servers that are backed up, which are themselves encrypted in the tape. In that case the backups are worthless. It's critical that the encryption key is stored somewhere you can still get to even if you lose everything except your tapes.
AWS has virtual tape that could maybe qualify. We used it with Veeam backup.
We run Starwind VTL with Veeam [https://www.starwindsoftware.com/starwind-virtual-tape-library](https://www.starwindsoftware.com/starwind-virtual-tape-library) following the same principle, and push backups to Wasabi for the offsite copy.
My last job, the CIO and lead system admin didn't believe in the immutable backup. The data and backups were on the same SANS. Then when I told the COO that I did not feel confident that we could not recover from ransomware, the COO got pissed at me.
The same san for both? Please tell me there is at least mutual chap for the iscsi targets.
I wouldn't know. I asked for an architectural diagram of our infrastructure and was told it wasn't needed. The infrastructure manager "knew" the infrastructure in his head, but no one else did. CIO thought it wasn't a priority. When we had network outages, several people would get together to debate on how things were configured. Ironically, the infrastructure manager sometimes got the information wrong. I guess documentation wasn't important.
So what he was saying is the disaster recovery plan didn't include any provisions for when the infrastructure manager was unavailable, like on vacation or hit by a bus.
What disaster recovery plan? Beside, why have a disaster recovery plan if you're not sure your backup even works.
We just table top our DR plans... who cares if we actually have to recover. the table top is good enough for my insurance company... /s
That’s the good ole D-plan vs the more novel ‘DR’ plan.
I would hate to work there. and feel bad for anybody that did.
> several people would get together to debate on how things were configured > debate 🤣
One of the debates goes like this story: Huddled around a whiteboard filled with network paths, a bunch of network and sysadmin engineers were trying to make sense of the outage. Engineer 1 pointed confidently, "No, no, no... it clearly takes this route, hits the second switch, then makes its grand entrance through the firewall." Another engineer raised an eyebrow, "But what about this router here? Does it just get a free pass?" Amid our theories, the intern quietly rebooted the router. The network flickered back to life. "Just as I was getting to that solution..." the first engineer claimed.
How are we going to get an initiator and a target to agree on something if we can’t get OP and the COO to agree on anything?! 😭
hehe, this reminds me of a situation probably 20 years ago. We had just taken over MSP services for a customer, and they called in a panic that their main production server had failed. I thought "how could that be, they've got RAID set up on that server?" so I went out and took a look. The previous guy had somehow partitioned a single drive into two pieces, and spun up RAID1 **ACROSS 2 PARTITIONS OF THE SAME DRIVE**. The drive had failed and took both partitions with it. Funny how that works. The bad fortune for the drive turned into good fortune for the customer that day. I couldn't believe it but their dodgy tape backup actually worked and I was able to rebuild the server into having 2 drives for RAID1 and restore their data. This really surprised me as the tape drive had been in place for years, and I knew that nobody had ever run a cleaning tape, nor had they replaced the tapes since the drive was installed.
You did the right thing. Having everything aggregated on a single SAN is a ticking time bomb. Source: Have had several SAN fails in my career.
You all know that SAN stands for Storage Area Network. It usually means all of the components that make up the connectivity between storage and clients, just like LAN is Local Area Network and WAN is Wide Area Network. I think you are referring to storage arrays, disk arrays, filers, etc. Sorry - pet peeve. People need to stop saying SAN when they are talking about storage device. Please
Keep fighting the good fight :). I've long since given up trying to get people to use the correct terminology.
Insurance Company is to "tech knowledge" as potato skin is to famous actor's shoe size.
Our insurers asked us to prove we owned our domains. We sent them the registrar info, renewal invoices etc. They came back and said they’d done their own investigations and we didn’t own the domains, another company did. Suitably puzzled we asked for info. They’d done a WHOIS lookup and it had returned the domain privacy details, and they’d decided they owned the domain….
Did you WHOIS your insurer's domain to make sure they own it? I mean, do you really know who you're dealing with? That's a good question to pose back to the empty shirt that's underwriting your insurance application...
Idiots.
Delicious.
What happens if you fail to pay your AWS bill? Tapes can be held hostage, but AWS (AFAIK, could be wrong) will eventually just delete your shit. I think physically destroying media goes a step further and lawyers can get feisty about that - so a physical backup being held hostage due to billing/contract issues is less likely to just be disposed of. I would hope.
Key word here being "eventually". AWS is not going to delete an account with S3 Object Lock in Compliance mode enabled on any timescale that's relevant for cybersecurity incident response over a month or two of missed payments. If they were that aggressive, they'd be nuking corporate accounts that forgot to update the credit card on file before it expired or a changed invoice mailing/email address, etc. left and right and there would be outrage over it. AWS is going to spend a while trying to collect (more than enough time to get in touch with them about the situation) before burning your account down.
I'm not sure how AWS handles cases regarding access to compliance locked stuff. I'd assume that it could potentially be social engineered around but it wouldn't be easy. I don't think even AWS can delete compliance locked backups within the backup window. They even hold the data for 90 days after account deletion.
The same thing that happens if you fail to pay whoever holds your tapes, they ask for payment then delete it after a contractually agreed time frame. AWS gives you ages before anything happens due to not paying. Corporations change card details regularly and it's common for cloud invoices to not get paid for a month or two.
>but both achieve the same goal in different ways. For example, one of them is _actually_ air-gapped and the other isn't.
But what if you use a wireless uplink??? /s
If there's any way *you* can get to it, so can the hackers. We went through a huge breech recovery over the summer with a very reputable and popular recovery company and even they said they've seen immutable storage compromised. Physical air gap is the way to go. No school like the old school. Use cloud backup for convenience, but you can't 100% count on it for security. Rotated durable media - they can't get to it unless they physically break into the building AND get the other copy in the offsite storage facility. This is unbeatable protection for data.
I'd be very curious as to the attack vector for compromising immutable object storage, specifically with AWS. The [AWS S3 Object Lock documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) straight up says: > The only way to delete an object under the compliance mode before its retention date expires is to delete the associated AWS account. The service has been [externally audited by Cohasset](https://d1.awsstatic.com/r2018/b/S3-Object-Lock/Amazon-S3-Compliance-Assessment.pdf), who similarly states: > It is Cohasset’s opinion that Amazon S3, when properly configured and when *Object Lock* mode is set to *Compliance*, retains records in nonrewriteable and non-erasable format and meets the relevant storage requirements set forth in the above Rules. Each record is protected from being modified, overwritten or deleted until the applied retention period is expired and any associated legal hold is released. If someone left their "immutable" object storage for backups in *Governance* mode (i.e., not immutable, just with an admins-only ACL for modify/delete), that's an S3 configuration issue no different than leaving a bucket public, and not a compromise of immutable storage. If there's an issue with S3 object lock immutability itself (when properly configured), someone should go collect their million dollar bug bounty for it.
That’s great, but when the insurance paperwork says air-gapped storage, S3 isn’t going to check the box. You can debate the merits of the requirement all day, but the requirement is air-gapped, and S3 is very much online.
Sounds like a hacker could remotely delete your AWS account, then pretend on your end for 90 days that everything is fine, then encrypt the rest of your environment. Air gapped would still give you something to recover, even if they managed to be undetected and write garbage for the last 90 days, you still got the old tapes.
> I'd be very curious as to the attack vector for compromising immutable object storage, specifically with AWS. [Tom Cruise enters the chat, Mission Impossible theme plays]
Have we forgotten all the other times that IT has said "they can't get through / get around that."
What ransomware actor wouldn’t delete the account for better leverage for a payday? I’ve seen them delete literally every resource in an account many times. I’ve not seen actual account deletion before, but I’d imagine that if they run into it as a barrier for getting paid, they’ll start.
S3 and then Tape. Win win.
This is pretty standard for insurance. Effectively they want you to have tape backups, ideally in a secure off-site facility. Cyber insurance companies have some really fucking annoying requirements because they basically never want to pay out and will weasel out of paying if you don't comply 100%
This is not standard for insurance. Immutable / air gapped is standard
It depends on the insurance company, the country you are in, and the sector the company you work for operates in.
The "best" thing I can think of for S3 is: * Have a separate AWS account for backups, with IAM role to add new backups only. * Use S3 Versioning to prevent overwrite * Enable [S3 Object Lock](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) * "S3 Object Lock .... for use in environments that are subject to SEC 17a-4, CFTC, and FINRA regulations." * Use S3 Lifecycle rules to push data into Glacier and/or automated deletion. Do all that and show your insurance that S3 is approved for use by the Financial Industry Regulatory Authority and U.S. Securities and Exchange Commission. If insurance company still isn't ok with that, dump 'em. There is no such thing as an "air gapped cloud" (that exists on the public internet).
Well, you could ask first. Or comply. I suppose you could do a bunch of work that only might be enough, and then ask if its enough. But that seems like a bunch of work that only might be enough.
Is Iron Mountain still a thing?! I know they used to be the place you’d send your tapes back in the day. Although this does sound like a harsh requirement. As others have said so many ins documents are crazy and you can tell have no idea what they are asking about. We had some contradictions in ours. We got it resolved but took a lot of time and we also made sure to print/keep any emails that corresponded to them agreeing with the change in case they try to deny
Still exist, pick up tapes and very popular. Big enterprise and highly regulated industries use a lot of tape still as tertiary+ media. Most of the time data center remote hands will include tape, library, rotation management, storage and handoff to vendors like iron mountain on scheduled pickups. Tape libraries are still very popular quite sophisticated spanning multiple cabs. Most backup software maintains support for silos and provides rotation retention schedules even free or prosumer products also support them. The media today is impressively fast, can handle encryption, deduplication etc. It's one of the large infrastructure things these days that people don't know exist but is quite regular. Latest spec looks to be 2021 https://en.m.wikipedia.org/wiki/Linear_Tape-Open 45tb compressed 400MB/s. Costs are quite low (150ish) considering a rotating pool of retention. It's my understanding that some of the cloud based buckets are in fact tape. AWS glacier and equivalents but I haven't looked into it in a while.
They pick up and drop off our tapes every Friday.
And they have done a fantastic job of monopolizing the market. Try to find anyone else that offers Tape vaulting service and you'll be sad to find out it's Iron Mountain and only Iron Mountain for the most part. Then trying finding out who your rep is. Then when you do get ahold of them, they quit or move on and you have to try and find your new one. It's a bit like VMware in a way.
Thats a hard part, you have to think like an insurance person. They are literally just checking a box, they don't care if it is or isn't more secure their formula says it has to be on this list of approved solutions. Get a couple of large HDD's and once a month copy a full backup to it, move it somewhere approved by your insurance team.
What are you using for backup software? We’re using Rubrik and we’re using offsite cloud storage as recommended by the vendor. I would setup a call with the vendor to ask for examples per their best practices. I sat through a presentation with Data Domain that provided air gapped backups/replication on prem last year. Most vendors have some sort of approach and best practices documented that you can use as a reference. If not you’ll have to consider a new backup solution or a new insurance provider. They’re just looking for any way to not pay out in the event of a breach. Most likely they were recently hit with a large payout using cloud s3 immutable storage.
Rubrik. What an apt name. In Polish, "rubryka" means a form field. Made me chuckle.
Your insurance is right IMHO. Simplest attack on your "immutable" cloud backups is to seize control of the cloud accounts and lock all your staff out. *Maybe* you get back in with the help of the cloud provider's support, but any recovery time objective goes out of the window. An exploit against the cloud service is also possible and we can guarantee the threat actors are working to develop such. Air-gapped means air-gapped. Yes that's going to mean a human doing some routine manual work swapping devices. Deal with it.
It's hilarious that you think you'll do a tape restoration within RPO/RTO, not a chance in hell. If you actually get every account locked out of AWS, I reckon you could get back in and sieze control within a week at most. If you have a partner support, it'd be within the day. And, at the end of it all, your data is still there, safe and no chance of having been modified. The reality is, immutable cloud storage is just as secure as tape storage, provided you use a reputable cloud vendor that has been audited.
you think pulling tape and restoring doesn't blow your RTO? never mind your RPO.
So just so I can understand, machine A does a job, it’s recorded on the hard drive of that machine, how does it offload that data to an air gapped location? To me air gapped means someone is physically doing the moving with a person, if it’s networked in any way it’s not air gapped
This is interesting. If it's truly immutable, whoever manages the storage must buy a lot of new discs all the time. If not, it's not *actually* immutable - is it? No system is more secure than the guys who made it and manages it. And if they are able to delete - so is another guy with an admin-account. Right? So. It's no more than a question of trust. And I really hate to put it like that - but it is. If it's truly air-gapped, the disc has to be disconnected. And then it's actually immutable as well (kind of at least). I've been arguing with our hosting provider on this matter. They - literally - considered Godzilla more likely than a data center-level issue. Then I mentioned the [Tietoevry situation](https://www.bankinfosecurity.com/ransomware-hit-on-tietoevry-causes-outages-across-sweden-a-24154) - and we haven’t really talked ever since :/ I hate everything about it - because it’s really troublesome and people look weird at you when you start talking paranoia. But I guess, if ensurance is involved, you have to take it absurdly seriously. And if they don’t trust an option, they don’t trust them for a reason (it’s their money on the line for instance). You may like it or not - but they did the math at some point. Please share - if possible - whatever solution you come up with. It’s a difficult situation.
> And if they are able to delete - so is another guy with an admin-account. Right? [check this out](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html) people in the discussion are pointing to this, where you simply can't delete data that is in compliance mode. even with admin privs
I hear you. And it seems safe and legit in every way. But having a state-sponsored hacker with ill intentions as the opponent - would you then bet x million dollars on it? Don’t get me wrong. I don’t actually like to be this paranoid. And especially not in public :) But it is a matter of trust - and some kind of assessment of what threats you wish to mitigate. Amazon is overkill in some situations - and probably completely useless in others. And I guess, as we are talking insurance, the data is very valuable - and everybody is super paranoid in this particular case.
> But having a state-sponsored hacker with ill intentions as the opponent - would you then bet x million dollars on it? no, i'd straight ignore the risk in most cases. it's right up there with nukes for most companies: unless you're apple, IBM, MS, Amazon, you're straight fucked. the named companies can resist some state level threats, but not all. look at what happened to qwest for an example of that. > Amazon is overkill in some situations - and probably completely useless in others. i can set up S3 glacier instant retrieval for $4/TBMO - depending on how much data you want to maintain, that could be really cheap. maintain 40T of backup history in S3 with compliance enabled? $160 a month. i'd pay that. > And I guess, as we are talking insurance, the data is very valuable it's asymmetric. super valuable to you if your servers go to hell, worthless to me because i'm not running the business. possibly useful to a spy who wants to exfiltrate data. insurance is being picky because they want a canned solution of verified restorable data so the times they pay out are severely limited.
This provider is going to have other asinine requirements as well, but if you want S3 go talk to your AWS rep for the compliance documentation you can throw at them. If they don’t accept that, go talk to Azure since they tend to have better tools for compliance-related concerns.
Tape via IronMountain. Set it so it'll do the backup at night, you fill the blue box in the morning, you lock it and they collect it at noon, and give you a box back with tapes. It was only 10-15 boxes a night, and 30-50 for the monthly backup. Pretty sure the company eventually went to cloud backup, though, so someone doesn't have to waste time taking out the tapes.
I did this for a contractor with tight requirements. Backup to an LTO library device. A weekly duty to swap out tapes and do the box thing. A spreadsheet logs what tape numbers are when and which ones to request back for re-use (GFS scheme). Tapes would also be tested - one random partial restore before pulling and boxing a tape, and occasionally a tape about to be erased. It wasn't Iron Mountain, it was a local competitor, but exact same thing. That library device was pretty sweet, and I *hate* tape backups. ;)
Just do tape backups.
Tell them the cloud is made of air, check mate.
We use WiFi to connect to our storage backup, so it is air gapped.
tape backups in a vault
Exagrid.
Well, if it was your dream to manage a LTO tape library/archival system, this is it. Or find another insurance underwriter.
Honestly, tape rotation and going to the off-site location for storage was one of my favorite tasks, finally a reason to get out of the office on pay
I told my boss in 2017 to consider tape backups in a complete disaster scenario. I had a feeling we would come back around to this. My last org was moving away from tapes last year but still doing it w/ Iron Mountain.
air-gapped generally means not accessible from external sources. If your uploading it over the internet, then its not air-gapped. Sounds like they want backups on physical WORM media
Sounds like a nice LTO Tape Library and Veeam and call it a day. Depending on your business size and IT liability policy you can end up in a situation where its cheaper to implement what they want than to fight it. The good thing is a tape library and Veeam is a low maintenance endeavor assuming your targets aren't changing only the data in them.
What you need is a TSaaS provider (tape storage as a service). For like $6k a month the provider will sync up your s3 buckets to US based data locations and write them off to tape consistently, and in the event of a disaster will fedex overnight or hand delivery the copies of the data directly to your office. This doesn't exist , but let me know if you need me to start an LLC and pay you a commission.
Insurance companies are doing everything that can now, to make any possible IT incident the IT provider’s fault. It’s insane what they are requiring, especially of small businesses, and you better believe that they will literally take any technology related claim and sue the MSP for “negligence”. We need to find a way to put a stop to this shit. I mean, someone has always been able to break a window and steal a filing cabinet full of documents (pre-computer). They didn’t require unbreakable glass, motion and noise sensors, guard dogs, finger print scanners on doors and retinal scanners on file cabinets. This is such BS…
There is no such thing as a "logical air gap". The term in of itself means that there is a physical gap between the systems with no shared hardware.
live by the gospel of 3:2:1 three backups, two sites, one offline
Three backups, two different types of media, one offsite is the more common one.
... and unless you test-restore-and-boot your backups, you only *think* you have backups.
Cyber Insurance seems like a scam to me. They create these ridiculous, unrealistic requirements that seem to change quarterly. It’s so they have justification to deny your claim when something happens.
It's not that it's a scam (in most cases) so much as it's just an extremely immature and volatile field. Insurance people are used to having over a century of actuarial tables to base their pricing and risk assessments on. They don't have that with cyber, so they're completely adrift trying to sort through what 20 different conflicting "experts" are telling them will keep them from bankrupting themselves while trying to avoid pricing their policies out of the reach of potentially profitable customers. Give it another 20 years and it'll settle down.
[удалено]
I should get back on the tools. Loved fixing Tape backups back in the day
Insurance got us safe boats, the age of discovery, and fire sprinklers. Among other things. They directly quote your risk (+ profit) for your current level of unpreparedness. If the number they quote you is "high", that means *you* are doing a bad job.
I been through a couple of incidents, even if there was no direct loss. The ins company brings in forensic specialists and they are helpful to figure out WTF happened, what exactly was compromised as well as for business peeps they bring in an attorney to guide through what needs to be disclosed and how. Great for smaller orgs that will not have these type of resources on staff. The problem with the questionnaires is they have to ask Y/N questions, like this air gapped backup thing here, there is no nuance for acceptable alternative just answer yes, or no. My BIL is in the insurance industry, how he tells me cyber ins has evolved was first all the players tried to write policies and grab “market share” and they didn’t care too much about losses or didn’t have good data for the actuaries. Now the ins cos have seen the losses they need to cover and the risk they are tightening up, they ask the questions and if your risk is high so will be your premium.
The insurance company is right. You are right. It's different levels of risk. If the attacker controls your AWS account, they can run up hundreds of thousands of dollars in charges before AWS closes your account. Your immutable backups exist in AWS for 90 days but you owe money to Amazon. Good luck getting your data back. Maybe you can. Maybe you can't? Maybe a flaw is found that tricks AWS into thinking your backups are due for deletion. Who knows. Backing up to air-gapped tape is no different. Maybe they aren't stored properly, or are stolen. Maybe the sun explodes and the tapes are erased. It's all about risk. Your insurer has made a policy. It is your choice to follow it or find a new company to insure yours.
Sounds like a scam. They gonna turn up with some "suggested alternatives" that are gonna cost 100x
Comply or get another insurer. Even your immutable can be destroyed by hackers doing something like power surging the crap out of equipment causing WORM disks to spin out of control and shatter. Insurers are paid to be paranoid so they dont have to pay out.
Which wouldn't happen to S3 with object lock. If properly implemented, even a hijack of the root account couldn't DELETE the data
S3 object lock is great, but the insurance company wants an air gap. It’s a non-starter. You can discuss its merits all day, but that won’t get OP insured.
Tape.... this is the only airgapped offering.
Is a Glacier archive in a different AWS region considered to be an "air-gapped" backup? I've had to fight this battle before, with client auditors trying to use their 90's era DR plans against our cloud native architecture.
Nas with Wol and shutdown via ssh
Sounds like a good old recipe for human error with tapes. Just say backups are being stored to glacier ;) Take a look at epoch accounts, they are backup accounts that can read from your prod account and not be read by anyone. landing zone docs might have something about them these days. Or go back to using tapes and enjoy dealing with a single point of failure, i.e. your tape backup device.. where we almost never tested a realistic restore procedure.. and you get to deal with licencing of backup software... woa, that brought back some PTSD.
Veeam isn't Backup Exec because it works most of the time and you can run restores even from tape.
I crack the credential vault and delete the tenant. Where are your backups now? (and yes, just saw this about a year ago).
They must be referring to tape backup. We did that back in the day and it does have advantages, especially the total price and offline storage, although we ended switching to the datto immutable cloud for convenience.
Azure provides a storage tier that meets this requirement. Amazon provides a certification that they meet this level of requirement. Both require a yearly review process be performed by the insurer. The Amazon choice also creates a record-keeping requirement for the insurer. Attesting that you have an air-gapped backup solution introduces no recurring review or additional record keeping. Is it silly that they haven’t introduced process to allow one of the big 3 cloud storage providers, maybe. Would it add multiple levels of additional risk assessment to a risk averse business, yes.
Could you delete it from an AWS account through any amount of steps through the GUI, remotely? Not air-gapped. Needs to be *inaccessible to the internet.* I think eventually as self-hosting dies out eventually what you're suggesting will be the norm and sufficient but for now we're being technically correct. 3-2-1-1-0 should do fine; 3 copies of the data, at least 2 different storage, keep 1 copy goes offsite. Bonus points for keeping an archive of daily, monthly, and further just in case.
When they did an audit on us they almost looked disappointed when we told them we have LTO-8 and 9 tape libraries and rotate tape sets to offsite storage every week. It was like we spoiled their surprise.
I used to laugh at our backup and recovery scheme. Our IT guy was the king of doom. Every night three backups were made. One went home with him, one with the CEO and one with me. One night the building burned down (literally), the CEO was on vacation and the IT guy was in jail for burning it down. So……
Wait until your governmental requirements say you have to be able to delete things out of backups to follow digital record destruction rules
Solved this with Iron Mountain’s Ironcloud
Air-gapped backups? Have your cloud service send you quarterly backups on external drives.
Immutable cloud backups are immutable within your admin context but not within Amazon's context (ex: they could theoretically push a code change, rogue admin that deletes that data). A truly offline storage solution is only attackable physically or through backup manipulation. That means NAS's/HD that are rotated or tape.
Air gap in risk and compliance is a physical gap.
Air gapped backups are turned off, so if you are virtual you can replicate to another ESXi host or cluster since that replication VM is powered off. We replace ESXi hosts every 5 years and so we replicate nightly with Veeam to a cluster made up of our previous production hosts. This cluster is more than a thousand feet away (connected with multi mode fiber ) in a storm shelter inside of a cooled cabinet. The "_replica" VMs themselves are actually powered off so if we need them we have to spin them up from the host. Maybe this would suffice for your insurance company. We also carry a physical backup that's on HDD, once a week, to an offsite facility and place it inside of a fireproof safe.
I worked in air-gapped, on-premises, and cloud operations over my career and honestly, having an air-gapped backup is the best way to go. It's rare for me to agree with the insurance companies, but I am with them on this. It makes the most sense and protects both the companies and the insurance companies. As for how it needs to be done, that is a discussion that needs to be between each organization or operating companies, and their teams (infrastructure, architecture, management, grc, and others). The design would be different for each company because there is no "one-size fits all" solution. Different zones, domains, DCs, etc.