|
dougdrums posted:That still qualifies as reachable from the internet though. I keep the backups for my business in a deposit box. I remeber one place where they just had a zip file on the dc and were like, "yeah of course we keep backups!" Yes, but that isn’t a backup server. How would you configure things such that the server is both useful and not transitively reachable from the Internet?
|
#
?
Feb 14, 2019 12:43
|
|
|
|
| # ? May 12, 2024 20:58 |
|
|
Have two of them (that is, configure a system for backups, and keep another version offline). Drop off a copy on my way home. Maybe use a third party too.
dougdrums fucked around with this message at 12:49 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 12:44
|
|
|
Drop off a server? (I like the “on my way home” backup rotation strategy for a mail provider’s data, though.)
|
|
#
?
Feb 14, 2019 12:50
|
|
|
Subjunctive posted:Yes, but that isn’t a backup server. How would you configure things such that the server is both useful and not transitively reachable from the Internet? I like tape robots. Backup completes, robot pulls the tape and drops it in a bin that gets sent to iron mountain daily/weekly.
|
|
#
?
Feb 14, 2019 12:51
|
|
|
Subjunctive posted:How do we know that I t was directly reachable from the internet? They could have pivoted from something else they compromised, no? I think it's a fair assumption to make if the attacker was burning down servers as they went (which is what the narrative has implied). Regardless, the backup server obviously had access to the Internet, since whatever exploit that was used was able to create an outbound reverse SSH tunnel. Say they had a DMZ and the backup server was behind that. Whatever host he used to bounce his attack off of, it still shouldn't have been able to make an inbound connection across that line. Like I said, that doesn't even touch the fact the backup server had direct outbound access to the Internet. I honestly can't think of any good reason to let a backup server be able to get to the Internet other than laziness in design or operation.
|
|
#
?
Feb 14, 2019 12:53
|
|
|
RFC2324 posted:I like tape robots. Backup completes, robot pulls the tape and drops it in a bin that gets sent to iron mountain daily/weekly. Sure, there are lots of ways to rotate media to distant storage. That’s not the same as the backup server that generates those media being unreachable from the internet transitively. (It also puts constraints on your restore options.) You could write exactly the same tweet in the presence of a tape robot.
|
|
#
?
Feb 14, 2019 12:55
|
|
|
Subjunctive posted:Sure, there are lots of ways to rotate media to distant storage. That’s not the same as the backup server that generates those media being unreachable from the internet transitively. (It also puts constraints on your restore options.) The question asked was how to make a backup server useful and the data is not at risk from the internet. A properly managed tape library addresses that.
|
|
#
?
Feb 14, 2019 12:57
|
|
|
Subjunctive posted:Drop off a server? (I like the “on my way home” backup rotation strategy for a mail provider’s data, though.) Yeah no poo poo, I was clearly just giving an example for my case. You store a copy on media somewhere safe, regardless of scale. It's not exactly a novel concept.
|
|
#
?
Feb 14, 2019 13:01
|
|
|
Even if it's a mounted NFS volume (through an ACL that only allows NFS traffic) that your backup server writes to, and that volume has snapshot policies and/or is replicated somewhere else and you don't store the AWS keys in someone's home folder out of laziness, that's still an improvement
|
|
#
?
Feb 14, 2019 13:02
|
|
|
Subjunctive posted:Sure, there are lots of ways to rotate media to distant storage. That’s not the same as the backup server that generates those media being unreachable from the internet transitively. (It also puts constraints on your restore options.) You still have the previous version if your backup server gets hosed. It's a redundancy. You have to assume (like in this case) that if it's hooked up, it's at risk. You can keep a backup server live and restrict traffic, there's nothing wrong with that. You shouldn't stake your whole business on it being hosed though. The maersk/petya case is a good example. dougdrums fucked around with this message at 13:14 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 13:03
|
|
|
RFC2324 posted:The question asked was how to make a backup server useful and the data is not at risk from the internet. A properly managed tape library addresses that. What is "proper" in your experience? Any rotating tape library configured & managed by TCP/IP can get attacked
|
|
#
?
Feb 14, 2019 13:04
|
|
|
RFC2324 posted:The question asked was how to make a backup server useful and the data is not at risk from the internet. A properly managed tape library addresses that. That’s not the question I asked, and I don’t see it elsewhere. The assertion I was questioning was that the backup server itself should not be transitively reachable from the internet, which means that it can’t receive data over the network from an internet-connected host, right? I’ve seen red team reports of attackers pivoting from a DMZ’d host to internal networks by using exploits against internal desktops that had accessed services in the DMZ. Transitivity is powerful. Similarly of attackers sabotaging the backup system (by loving with encryption key management IIRC) and waiting through a full rotation before wiping things. If your data scale and restore SLA are compatible, then tape rotation can reduce risk, but it can’t eliminate it. I’ve worked at places where (a part of) the backup infrastructure is literally shipping-crate sized clusters of hard drives paired to each sub-DC. Tape really isn’t an option, especially if you want to have automated restore validation across multiple time scales.
|
|
#
?
Feb 14, 2019 13:12
|
|
|
dougdrums posted:You still have the previous version if your backup server gets hosed. You might indeed, but the quoted tweet is about the backup server itself getting hosed, no? dougdrums posted:You store a copy on media somewhere safe, regardless of scale. Your notion of scale is pretty unimaginative. Where do you think gmail’s cold backup lives that isn’t transitively reachable?
|
|
#
?
Feb 14, 2019 13:17
|
|
|
I agree it's not necessary to take it offline transitively. If transporting physical media is impractical, you should at least back it up with a third party. Most places aren't google.
|
|
#
?
Feb 14, 2019 13:18
|
|
|
dougdrums posted:I agree it's not necessary to take it offline transitively. If transporting physical media is impractical, you should at least back it up with a third party. I’m not sure why it matters that it’s a separate organization that owns the second tier of storage. Why is that better than just having your own infrastructure in some place that doesn’t share much physical risk? It seems worse to me, from the perspective of coordinating policies and monitoring, but I haven’t thought about that aspect a lot. The places I’ve worked that consisted physical transport impractical all just had their own servers “far enough” away to match their risk appetite and budget.
|
|
#
?
Feb 14, 2019 13:21
|
|
|
Fair enough, most of the people I used to work with were likely to gently caress something up, or they didn't have the resources to maintain it otherwise. Either way it reduces the risk of your business being burned to the ground in a few hours time. You can't eliminate it totally. If you've got the resources to maintain it, no reason not to. Hence why I [meant to say] "maybe". Having offline media is still useful even if you don't make copies frequently. dougdrums fucked around with this message at 13:35 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 13:31
|
|
|
Subjunctive posted:The assertion I was questioning was that the backup server itself should not be transitively reachable from the internet, which means that it can’t receive data over the network from an internet-connected host, right? Correct. The backup server should reach out to the DMZ host, but not be accessible in the reverse direction. You should be doing a pull for backup operations.
|
|
#
?
Feb 14, 2019 13:34
|
|
|
Yeah I think I've might of misread you in my morning state, The way I have mine set up personally is to make outbound ssh connections to the stuff I need backed up, and restrict it to those specific hosts. All inbound is dropped. I might gently caress it up, or there's some other hereto unknown vulnerability, so I keep copies offline. I have no idea how many users they had. dougdrums fucked around with this message at 13:53 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 13:49
|
|
|
Proteus Jones posted:Correct. That sounds Internet-connected to me? “Go the wrong way through the supposedly-one-way firewall” is why every toolkit has a billion ways to create a tunnel back to the C2. If anything can create a connection into both isolated networks, an attacker can pivot by exploiting that thing (transitively). This is not an uncommon thing when attacking a network that isn’t vulnerable to last year’s top 5 issues. One of the red team reports I read at a previous job involved pivoting off of a conference room controller tablet, and using it for persistence. There are two kinds of connected: connected if the attacker is good enough or gets lucky, and airgapped. The connection doesn’t even need to be synchronous: we used to deal with attacks over store-and-forward UUCP!
|
|
#
?
Feb 14, 2019 13:58
|
|
|
Subjunctive posted:That sounds Internet-connected to me? “Go the wrong way through the supposedly-one-way firewall” is why every toolkit has a billion ways to create a tunnel back to the C2. If anything can create a connection into both isolated networks, an attacker can pivot by exploiting that thing (transitively). This is not an uncommon thing when attacking a network that isn’t vulnerable to last year’s top 5 issues. I get what you're saying and I did make an assumption on the backup being nakedly available. However, in this specific instance, they still should have not been able to create that reverse ssh tunnel. Allowing your backup server to touch things in the DMZ is not the same as allowing it out to the Internet. And yes, they probably could have created a relay system if they were able to exploit the backup pull session, but that's not what was in the tweet.
|
|
#
?
Feb 14, 2019 14:05
|
|
|
Are we talking about someone manually making a copy and then walking it over to an airgapped machine? Yeah that's dumb I'd agree. I think the confusion was when I said it shouldn't be connected to the internet, I meant there should be backups on physical media somewhere. In vfemail's case (which boasted the extra security as a feature), I don't think they had enough users to make it impractical, or they could do incremental backups or something. Just seems like they were playing with fire.
|
|
#
?
Feb 14, 2019 14:13
|
|
|
Here’s why I worry so much about transitivity, as a concrete example. It’s been long enough and there was some public reporting, so I feel ok talking about it When I managed Facebook mobile, we had a developer’s machine go nuts, and eventually we found that someone had landed a payload on it which was continuously searching the file system for files with certain strings in them and trying to map the network. This machine lived on the corp network, which was obviously “unreachable” from the internet. Unrouted addresses, proxies and firewalls that all pointed outwards, the usual stuff. What we eventually discovered was that someone had: - found a site that got a lot of traffic from facebook.com addresses (in this case a mobile developer documentation/community site) - compromised its web site - built a payload specialized for our environment (developer laptop layout, network naming conventions, etc) - found/bought a zero-day Java exploit - served that payload only to facebook.com visitors They didn’t get anything out of it, mostly because the payload wasn’t actually set up right for our world, but they got to run code where they “couldn’t get to”, through a very competently administered “one way” network connection.
|
|
#
?
Feb 14, 2019 14:16
|
|
|
Proteus Jones posted:but that's not what was in the tweet. Indeed, there’s not much in the tweet. In my opinion there isn’t enough to judge them as incompetent, which is why I commented in the first place. (That and the pervasive sentiment that secure and reliable backup of internet connected systems has a straightforward and universal solution. Working in complex, targeted environments humbles you!) When something doesn’t make sense to me, my first assumption is that there’s a piece of context that I’m missing. That’s why I ask a lot of earnest questions, which unfortunately people sometimes take as passive-aggressive trolling. I don’t know if the attempted connection back even worked—as in my example above, payloads and attackers often try things that don’t play out.
|
|
#
?
Feb 14, 2019 14:27
|
|
|
Oh yeah, like I would consider my backup server scheme above to be connected to the internet, regardless of how it's configured. I mean like physically connected, there's a non-zero chance of it communicating to and from the outside world, Bulgaria, 300 lb guy in new jersey, whatever. Fwiw I did red team stuff and had plenty of people argue that I could not possibly had access to a host without internal help/abusing the scope because it was "not connected to the internet". It didn't matter because I did it from the parking lot anyways. The point I was making with that tweet was that rm just unlinks files. I assumed that they already had total control of their stuff through some unknown exploit beforehand. dougdrums fucked around with this message at 14:39 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 14:30
|
|
|
So knowing all this, what would be an appropriate backup strategy for a mail host that might end up being the target of a state level actor with a handful of zero days? Assume that losing 24 hours of data is allowable, and a 7 day restore period is reasonable. A tape library that's able to do Disk to Disk to Tape would be my go-to, main storage takes snapshot, snapshot gets published to a new NFS share, share is copied to disks attached to the library, snaphot gets written to tape, tape gets shoved in a box, box sits on a shelf. Restores are spot tested on tape deck on desk attached to an air gapped desktop machine.
|
|
#
?
Feb 14, 2019 15:00
|
|
|
Microsegment the firewall rules between all servers. Backup server is allowed to vms on the backup ports only. Then you have a higher tier management endpoint with separate credentials that can reach the backup machine on 3389, 22, whatever minimal access you need, but not from anywhere else. An identity-based firewall helps too, limiting traffic from the backup server to the backup service account. Also limiting access to the backup server to a high-tier backup admin account. While the server is still technically internet accessible, it would take 3-4 hops and numerous credential pivots to get there, assuming the attacker knows the infrastructure enough to get through the inbound maze.
|
|
#
?
Feb 14, 2019 15:05
|
|
|
Most small/medium lovely, resource-limited IT shops probably would get most benefit using third party off-site backup providers with no administrative rights in the provider's environment. Let's face it, most businesses exclusively running their own backup infrastructure all out of the same network and auth spaces as their servers or managing backups in S3 buckets with lazy, reused passwords aren't hardened against total overnight wipeout. Potato Salad fucked around with this message at 16:02 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 15:57
|
|
|
If you don't have backups in a different physical location with distinct access methods, rights, and credentials, they're not backups. This is pretty elementary.
|
|
#
?
Feb 14, 2019 16:34
|
|
|
Subjunctive posted:Your notion of scale is pretty unimaginative. Where do you think gmail’s cold backup lives that isn’t transitively reachable? When protecting yourself from attacks by a state-level actor, it certainly helps to *be* a state-level actor. But anyways, google uses tape backups.
|
|
#
?
Feb 14, 2019 17:27
|
|
|
Klyith posted:When protecting yourself from attacks by a state-level actor, it certainly helps to *be* a state-level actor. Is that still the case for everything? FB made a big shift in media for photo backup in 2015, but I haven’t kept up with anyone really. Even at the time I didn’t know how to interpret "we store another copy of the most important data on digital tape”. Does that mean they can fully restore gmail if everything online is wiped? I wonder who I can ask...
|
|
#
?
Feb 14, 2019 18:38
|
|
|
Subjunctive posted:Is that still the case for everything? FB made a big shift in media for photo backup in 2015, but I haven’t kept up with anyone really. The NSA?
|
|
#
?
Feb 14, 2019 19:53
|
|
|
Volmarias posted:The NSA? My contacts there are stale, sadly. Maybe Russia!
|
|
#
?
Feb 14, 2019 21:19
|
|
|
Subjunctive posted:Is that still the case for everything? FB made a big shift in media for photo backup in 2015, but I haven’t kept up with anyone really. EDIT: Was trying to remeber the space they take up, but failed. 
BlankSystemDaemon fucked around with this message at 22:07 on Feb 14, 2019 |
|
#
?
Feb 14, 2019 21:59
|
|
|
Subjunctive posted:Is that still the case for everything? FB made a big shift in media for photo backup in 2015, but I haven’t kept up with anyone really. Beats the hell out of me, but there are more recent references than that 2013 article so I guess so? Evidently tapes have continued to improve in storage density even though HDs are losing steam. And yes they restored from tape back in 2011 when a bug in their FS wiped a few tens of thousands of gmail accounts. I feel like "everything online" starts getting very fuzzy when you're talking about the massive operations like google and facebook. Like I remember a different article about google's arrays of cheap hardware servers talking about having lots of spares, 'cause their cheap junk is less reliable than real servers. And their spares periodically boot up and synch data, then shut down again. "RAID is not a backup"... Well, maybe you just haven't made it redundant enough. Hold my beer.
|
|
#
?
Feb 14, 2019 22:17
|
|
|
Klyith posted:"RAID is not a backup"... Well, maybe you just haven't made it redundant enough. Hold my beer. Yeah, and a benefit of hot replicas is that you can also use them as capacity. To one person it’s an aggressively-populated CDN, and to another person it’s part of a tight-SLA DR strategy.
|
|
#
?
Feb 14, 2019 22:21
|
|
|
CLAM DOWN posted:If you don't have backups in a different physical location with distinct access methods, rights, and credentials, they're not backups. This is pretty elementary. I've always liked the 3-2-1 rule, as in your data is not fully safe unless you have at least 3 independent (as in not directly linked like RAID or cloud sync) copies, in 2 different physical locations, 1 of which is offline.
|
|
#
?
Feb 14, 2019 23:33
|
|
|
CLAM DOWN posted:If you don't have backups in a different physical location with distinct access methods, rights, and credentials, they're not backups. This is pretty elementary. Nah, it just matches a different availability/threat model. Depends on how hard it is to reacquire the data and how important the data is, too.
|
|
#
?
Feb 14, 2019 23:37
|
|
|
https://twitter.com/matthew_d_green/status/1097332682189619200 Don't undermine your own crypto.
|
|
#
?
Feb 18, 2019 04:14
|
|
|
I guess the laws of mathematics did trump the laws of Australia in the end.
|
|
#
?
Feb 18, 2019 04:45
|
|
|
|
| # ? May 12, 2024 20:58 |
|
|
We’ll see about that in court!
|
|
#
?
Feb 18, 2019 07:02
|
|
