|
dogstile posted:"I can't send anything out! It must be the network" - App guy I'm a network guy and this kind of interaction drives me insane. Nobody else understands networks at even the most rudimentary level but rushes to blame the network for anything and everything. No we didn't try replacing the patch cable ugh why are you network guys always trying to avoid doing any work??
|
#
?
Jul 4, 2018 00:34
|
|
|
|
| # ? Jun 2, 2024 11:58 |
|
|
guppy posted:I'm a network guy and this kind of interaction drives me insane. Nobody else understands networks at even the most rudimentary level but rushes to blame the network for anything and everything. No we didn't try replacing the patch cable ugh why are you network guys always trying to avoid doing any work?? Um I ran a traceroute to my server and there were some stars in the middle of the output. Checkmate, network dorks 
|
|
#
?
Jul 4, 2018 00:52
|
|
|
Kashuno posted:Well it beats shoveling leaves! Ok I walked into that one
|
|
#
?
Jul 4, 2018 01:01
|
|
|
guppy posted:I'm a network guy and this kind of interaction drives me insane. Nobody else understands networks at even the most rudimentary level but rushes to blame the network for anything and everything. No we didn't try replacing the patch cable ugh why are you network guys always trying to avoid doing any work?? I am no longer "a network guy" but I am still pretty adept at the art of proving it's not the network. Just accept you gotta do it every time there is a problem, life becomes less frustrating.
|
|
#
?
Jul 4, 2018 04:45
|
|
|
adorai posted:I am no longer "a network guy" but I am still pretty adept at the art of proving it's not the network. Just accept you gotta do it every time there is a problem, life becomes less frustrating. My job splits networks and network security into two groups, and it's ten-fold worse for us. It's always* (*rarely) network security's fault. Yes, we're running microseg rules, so when you spin up that new AD sever without telling us it will absolutely get blocked. That's a core tenant of the design, to ensure rogue services aren't hijacking your environment. When you don't go through proper change control, poo poo breaks and we will not accept blame. Now whenever there's an outage, no matter the type, you can bet a netsec guy will be one of the first in the room to prove our rules are working as intended.
|
|
#
?
Jul 4, 2018 05:15
|
|
|
Docjowles posted:Um I ran a traceroute to my server and there were some stars in the middle of the output. Checkmate, network dorks Had the CTO of a hosted cloud app tell me this. And not like CTO who should never be anywhere near engineering. He was the guy they brought in to troubleshoot the problem.
|
|
#
?
Jul 4, 2018 07:17
|
|
|
Judge Schnoopy posted:My job splits networks and network security into two groups, and it's ten-fold worse for us. It's always* (*rarely) network security's fault. Two groups here as well, and we're using microseg rules also. It's a procedural nightmare because the security team refuses to acknowledge the absurdity of their request. They want each new server to come with a list of each destination IP and port it will need to connect to, line by line. This stops being reasonable right around the moment you add your 6th server to the datacenter, much less your 6,000th. I can not possibly be expected to enumerate every IP and port that a new web server will connect to. Between database, management and monitoring servers alone, even if you let me use port ranges for things like RPC, we're talking about a thousand lines. And this is before you consider the fact that the information is useless to the network team. You don't create an ACL for every single IP and port, you put them into object groups and assign ACLs there. Then you tell the network team "hey can you give me the same ACLs on web38 that we have on web37?" and everyone moves on to more important matters. The security team, however, believes that you need to enumerate the access line by line. That's dumb enough, but it's also impossible. So, how do we get around it? By doubling down on the stupidity. Included with each network access request that we submit is secure copy of the firewall config. We then tell the security team that "the server with IP 10.1.10.38 will have the same access as IP 10.1.10.37 in the attached documentation". That documentation is thousands of unparseable lines. There is no chance that anyone is reviewing this document and coming to any reasonable understanding of what is being done. But this is the ridiculous game we play in order to satisfy our security team. Who, I must add, I hate.
|
|
#
?
Jul 4, 2018 07:23
|
|
|
nielsm posted:Lock app guy and network guy in a room together. Please do not lock me in a room with a guy who is both wrong and angry. I will kick the door down to get out so I don't murder him. I don't do well with idiots who aren't aware they're idiots (precisely because most of the time, that's me and i'm very, very aware of it). Also i'm pretty sure I have the IT version of PTSD from the last time i got locked in a room with my (old, i've changed jobs) manager (who was trying to throw me under the bus), the head of HR (who was looking for an excuse to fire me) and one of the company bosses (who is a loving idiot when it comes to networks not designed in 1990). dogstile fucked around with this message at 10:26 on Jul 4, 2018 |
|
#
?
Jul 4, 2018 10:22
|
|
|
MC Fruit Stripe posted:Two groups here as well, and we're using microseg rules also. It's a procedural nightmare because the security team refuses to acknowledge the absurdity of their request. They want each new server to come with a list of each destination IP and port it will need to connect to, line by line.  amazing. Microseg without well defined security group, ip sets, services, and service groups is hilarious. Let alone a department that isn't flexible when it comes to changes.The more I hear about how other companies handle network security, the happier I am to be where I'm at.
|
|
#
?
Jul 4, 2018 13:37
|
|
|
Am I the only one who thinks salespeople who use your LinkedIn info to figure out your work email are the worst? Had two people do that to me recently. They found my LinkedIn profile and emailed me at the standard <first initial><last initial>@<company> to try to make contact. It comes off as totally scummy.
|
|
#
?
Jul 4, 2018 14:25
|
|
|
very stable genius posted:Am I the only one who thinks salespeople who use your LinkedIn info to figure out your work email are the worst? Had two people do that to me recently. They found my LinkedIn profile and emailed me at the standard <first initial><last initial>@<company> to try to make contact. It comes off as totally scummy. I get at least 5 calls or emails each day. I added a note to my linkedin profile that says "do not contact with cold sales calls" and it does not deter them.
|
|
#
?
Jul 4, 2018 15:49
|
|
|
adorai posted:I am no longer "a network guy" but I am still pretty adept at the art of proving it's not the network. Just accept you gotta do it every time there is a problem, life becomes less frustrating. We do our best. But it's hard to prove that to people who don't understand the proof.
|
|
#
?
Jul 4, 2018 16:31
|
|
|
guppy posted:I'm a network guy and this kind of interaction drives me insane. Nobody else understands networks at even the most rudimentary level but rushes to blame the network for anything and everything. No we didn't try replacing the patch cable ugh why are you network guys always trying to avoid doing any work?? I’ve had three cable related network problems in the past few months during customer deployments. Bad cables and 3rd party transceiver weirdness. It’s usually not the network but when it is, and when it’s a layer 1 problem, man is it loving obnoxious. Judge Schnoopy posted:
Unless it’s a greenfield deployment I don’t think anyone does MicroSeg that well, at least with NSX. For relatively well defined services it can be pretty straightforward, but most of my customers who buy it have these terrible sprawling applications and servers that exist in a fuzzy state between multiple security zones and it just becomes a nightmare trying to get to a default deny stance. NSX also has some weirdness with directionality and the “applied to” section where you can get into trouble when rules get applied non-intuitively. And merely having a firewall in place can cause issues as I saw with one customer where we were allowing traffic, but the default timeouts for the DFW were incompatible with their client connections to their server application and as a consequence the client tcp connections would be closed out by the firewall before the client naturally timed out and released them, leading to a bunch of stuck open sockets on the server. Eventually the server would run out of available sockets and have to be rebooted. That one was fun to troubleshoot.
|
|
#
?
Jul 4, 2018 18:28
|
|
|
NSX is a dumpster fire that burns ever so less brightly than other microseg engines. The biggest redeeming factor is the API and powerNSX, allowing for a ton of template work and command line tools that eliminate the bullshit ambiguity that the GUI creates. But yes getting to a default deny state is stupid hard for any microseg deployment. We took the approach of locking down one segment of the customer at a time (monitoring, Citrix, back end services, AD). Much easier than doing it all at once and trying to put out a million fires day one.
|
|
#
?
Jul 4, 2018 18:40
|
|
|
I love watching our NSX guy slowly die inside every time something breaks due to the vmware sales rep going "you can totally do this" and the actual answer is "only to a point, then everything breaks in weird and wonderful ways, also here's the bug ids for all the issues you are having"
|
|
#
?
Jul 5, 2018 06:07
|
|
|
The benefits of NSX or other hypervisor-driven security (NSX, k8s, etc.) is that you don't need perimeter firewalls anymore in front of your applications which is a huge operational boon to sysadmins who can now offer a on-prem self-service model with all the features of the Butt, and a huge boon to the network admins because suddenly the firewall needs to be good at inspection and sandboxing in a very small rulebase instead of thousands of lines of PERMIT HOST X.X.X.X EQ 443. I've dealt with people where it takes a straight up month to get a firewall rule put in, which was asinine ten years ago to say nothing of it today when you can stand up an entire service in AWS with all the associated policy in like 5 minutes. it also has many architectural benefits on the network, suddenly your network just has to be fast and you don't need to worry about layer 2 adjacency between anything because it's aaaaall routed baby. unfortunately it's still all early days so right now I wouldn't be touching "microsegmentation" solutions with a ten foot pole tbh
|
|
#
?
Jul 5, 2018 11:18
|
|
|
I thought the point was having packet filtering be managed in a different silo, lest your iptables rules on every host end up at allow any any.
|
|
#
?
Jul 5, 2018 13:59
|
|
|
tortilla_chip posted:I thought the point was having packet filtering be managed in a different silo, lest your iptables rules on every host end up at allow any any. There are three big use cases of NSX. One is free, and that is the guest introspection stuff. I think you might have to license vsphere, but you don't need the full NSX product despite it being called part of the NSX suite. It allows you to run AV on your host, and not on each individual guest. The next is vxlan. You overlay layer 2 on top of layer 3, so you can have just vlan X to each host, then VXLAN encapsulates your 700 other VLANs for you. If you add a host, the management is very minimal and it's pretty hard for someone to gently caress something up (i.e. run switchport trunk vlan allowed yy and forgetting the word 'add'). Finally, you get the microsegmentation stuff. Similar to Cisco ACI. It lets you manage firewalls in a very programmatic way.
|
|
#
?
Jul 5, 2018 14:44
|
|
|
The guest introspection bit can also be done by vShield and we use that with Bitdefender for host-based AV and it works fairly well. Uses a lot less resources and let us more easily keep our Non-persistent VDIs covered without having to worry about updating signatures on the image or worrying about it somehow not getting updated while a user is logged in.
|
|
#
?
Jul 5, 2018 16:18
|
|
|
Internet Explorer posted:The guest introspection bit can also be done by vShield and we use that with Bitdefender for host-based AV and it works fairly well. Uses a lot less resources and let us more easily keep our Non-persistent VDIs covered without having to worry about updating signatures on the image or worrying about it somehow not getting updated while a user is logged in. https://esxsi.com/2016/07/29/nsx-manager-upgrade/
|
|
#
?
Jul 5, 2018 16:25
|
|
|
abigserve posted:The benefits of NSX or other hypervisor-driven security (NSX, k8s, etc.) is that you don't need perimeter firewalls anymore in front of your applications which is a huge operational boon to sysadmins who can now offer a on-prem self-service model with all the features of the Butt, and a huge boon to the network admins because suddenly the firewall needs to be good at inspection and sandboxing in a very small rulebase instead of thousands of lines of PERMIT HOST X.X.X.X EQ 443. I've dealt with people where it takes a straight up month to get a firewall rule put in, which was asinine ten years ago to say nothing of it today when you can stand up an entire service in AWS with all the associated policy in like 5 minutes. The NSX firewall is a single top to bottom ordered rule list, which can get extremely complicated when you’re trying to do absolute least privilege access between all hosts in the domain. It also doesn’t do deep packing inspection or any of the other “next gen firewall” stuff so pretty much everyone will still have perimeter firewalls as well for things like separating DMZ, WAN and Campus from server networks. Meaning you’ve now got firewall rules in two places and need something like Netflow to see the whole end to end flow of packets. It’s great in that it allows you to put up som barriers where you really couldn’t before. For instance environments that grew organically over time and try to apply security posture years later where Dev and Prod and Pre-Prod and Audit systems all live next to each other in layer 2 zones. Now you can put some walls up between them. But in brownfield like that you lol pretty much always end up with more holes in the walls than you really want.
|
|
#
?
Jul 5, 2018 17:47
|
|
|
YOLOsubmarine posted:The NSX firewall is a single top to bottom ordered rule list, which can get extremely complicated when you’re trying to do absolute least privilege access between all hosts in the domain. It also doesn’t do deep packing inspection or any of the other “next gen firewall” stuff so pretty much everyone will still have perimeter firewalls as well for things like separating DMZ, WAN and Campus from server networks. Meaning you’ve now got firewall rules in two places and need something like Netflow to see the whole end to end flow of packets. It removes the need for ACLs on routers and layer 3 switches, which are typically not managed centrally / not desired when rule lists become huge / can't attach to virtual objects. Otherwise you're stuck hairpinning traffic through a firewall to process segment rules. And to your point on netflow, ANY microseg deployment is going to require a robust, intuitive, manageable syslog application on day 0. I guarantee microseg will get scrapped within a week without it.
|
|
#
?
Jul 5, 2018 17:56
|
|
|
Judge Schnoopy posted:It removes the need for ACLs on routers and layer 3 switches, which are typically not managed centrally / not desired when rule lists become huge / can't attach to virtual objects. Otherwise you're stuck hairpinning traffic through a firewall to process segment rules. I’ve yet to run across a customer that actually uses stateless ACLs, but in any case it doesn’t remove the need for physical security even for east/west traffic. You still need to protect and segment physical devices from one another. So you’re still going to have internal firewalls doing at least basic rule enforcement if you’re doing things correctly. There is traffic the DFW is good for and traffic it is less good for and if you try to turn it into THE organization firewall you’re going to end up unhappy and likely insecure. Also, there are certainly physical firewall devices that can “attach” to virtual objects by pulling that info out of VMware and translating it to IP addresses, just as NSX does. One of the niceties or something like Palo Alto with NSX is having your physical and distributed firewall use the same security objects.
|
|
#
?
Jul 5, 2018 19:05
|
|
|
We use ACLs in conjunction with iptables, it scales reasonably well (10s of thousands of hosts)
|
|
#
?
Jul 6, 2018 00:35
|
|
|
tortilla_chip posted:We use ACLs in conjunction with iptables, it scales reasonably well (10s of thousands of hosts) Out of curiosity, what configuration management are you using for that?
|
|
#
?
Jul 6, 2018 01:40
|
|
|
wc -l firewall.sh 2688
|
|
#
?
Jul 6, 2018 02:14
|
|
|
YOLOsubmarine posted:The NSX firewall is a single top to bottom ordered rule list, which can get extremely complicated when you’re trying to do absolute least privilege access between all hosts in the domain. It also doesn’t do deep packing inspection or any of the other “next gen firewall” stuff so pretty much everyone will still have perimeter firewalls as well for things like separating DMZ, WAN and Campus from server networks. Meaning you’ve now got firewall rules in two places and need something like Netflow to see the whole end to end flow of packets. Operationally NSX sounds like a straight up mess but the concept is good. Essentially it doesn't make sense to separate the firewall logic from the hypervisor in the DC anymore, so I think in the future we'll see firewalls used mainly for inspection and basic filtering purposes while the complicated stuff is handled at the other layer. You'll still have them routing through a perimeter firewall but it'll be a permit any into NSX/Openstack/Kubernetes/whatever That's my opinion because with the rise of AWS/Azure/Gcloud clients now have a very easy way of building complete environments as shadow IT and it's usually orders of magnitude faster to spin up than on-prem stuff and the way those services handle security makes way more sense than the traditional model.
|
|
#
?
Jul 6, 2018 03:08
|
|
|
Where’s the tl;dr for NSX?
|
|
#
?
Jul 6, 2018 03:16
|
|
|
NSX is easy to deploy, but gets messy quick. You better have a good syslog tool that's ingesting both passed and dropped packets by NSX to prove its not causing app problems. The API is very good, especially with powerNSX, but expect to build custom tools in powershell to fit your exact use case. It makes the mess easier to manage, especially when you build template tools to push in bulk. NSX takes a huge strain off the rest of your network equipment if microseg is your goal. If you're not doing microseg and just want inspection, there are better tools, especially in the newer Palo Alto offerings. Make sure NSX will actually solve a business need before messing with it. But that's true of any IT tool.
|
|
#
?
Jul 6, 2018 03:31
|
|
|
Tab8715 posted:Where’s the tl;dr for NSX? It's a car that goes fast.
|
|
#
?
Jul 6, 2018 03:41
|
|
|
Roargasm posted:wc -l firewall.sh It's not long once you remove comments. eg, egrep -vi "kill me|gently caress|defenestrate|murder|drunk" firewall.sh | wc -l 4
|
|
#
?
Jul 6, 2018 03:44
|
|
|
Capirca and illumio
|
|
#
?
Jul 6, 2018 04:30
|
|
|
Despite my shitpost directly above, we demoed Illumio at my last job (2015) and it seemed very cool. Unfortunately our company was circling the drain at that point, and their pricing wasn't "free lol" so it didn't work out. It was neat tech, though. Even then they had the live maps of traffic being allowed or denied which was  I'm guessing that may be a bit harder to read when you're talking 10s of thousands of hosts, though?The environment was exactly what YOLO described, "It’s great in that it allows you to put up some barriers where you really couldn’t before. For instance environments that grew organically over time and try to apply security posture years later where Dev and Prod and Pre-Prod and Audit systems all live next to each other in layer 2 zones." A bunch of servers all in the same VLAN because they were 'production', GLHF writing manual firewall policies for what can talk to what. Our network engineer actually started down some unholy path of running tcpdump, parsing the output, and outputting iptables rules to create our own microsegmentation solution. Thankfully we all got laid off and got better jobs before that saw the light of day. If you feel like posting more about how you're using Illumio and how it's working out, I'd be interested 
|
|
#
?
Jul 6, 2018 05:24
|
|
|
Docjowles posted:Our network engineer actually started down some unholy path of running tcpdump, parsing the output, and outputting iptables rules to create our own microsegmentation solution. Thankfully we all got laid off and got better jobs before that saw the light of day. 
|
|
#
?
Jul 6, 2018 13:31
|
|
|
Our primary use case for illumio is controlling east-west traffic within a deployment group. The secondary case is where we need to manage state for north-south flows. Our deployment model uses a nested hierarchy for workloads, so you can explicitly talk to any construct above you in the tree, but not any other network nodes at the same depth. We don't use any kind of overlay, so as the environments grew we started to hit TCAM limits when trying to implement all the policy via the network layer. This hybrid approach bought us some time, although I'm a bit concerned about IPv6 policy throwing a wrench in the works. Ideally we'll move the bulk of policy to host level, with some broad zone based filtering in the network fabric. Overlays will probably help as well, unfortunately they just aren't a construct that is baked into our current automation model.
|
|
#
?
Jul 6, 2018 15:09
|
|
|
tortilla_chip posted:Our primary use case for illumio is controlling east-west traffic within a deployment group. The secondary case is where we need to manage state for north-south flows. Network traffic doesn't follow compass directions fyi
|
|
#
?
Jul 6, 2018 15:16
|
|
|
If only there was a way to communicate complicated concepts with some sort of metaphor.
|
|
#
?
Jul 6, 2018 15:21
|
|
|
CLAM DOWN posted:Network traffic doesn't follow compass directions fyi Thanks Dr. Geniusbrain PhD
|
|
#
?
Jul 6, 2018 15:39
|
|
|
tortilla_chip posted:If only there was a way to communicate complicated concepts with some sort of metaphor. My point is it's a stupid metaphor and people need to stop using buzzwords and simply explain technical concepts accurately.
|
|
#
?
Jul 6, 2018 15:44
|
|
|
|
| # ? Jun 2, 2024 11:58 |
|
|
tortilla_chip posted:Our primary use case for illumio is controlling east-west traffic within a deployment group. The secondary case is where we need to manage state for north-south flows. What the gently caress does this even mean?
|
|
#
?
Jul 6, 2018 15:47
|
|
