Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Pile Of Garbage
May 28, 2007



Thanks Ants posted:

Are any of the ~*microsegmentation*~ things that vendors offer worth a drat? If someone buys a load of Aruba gear and plans to run ClearPass, how much pain are they going to have?

Hasn't microsegmentation been superseded by zero-trust? That's the latest hotness as I understand it, running NGFWs and devices with endpoint clients that communicate info to the firewall which determines policy. My employer has started doing it internally with Fortinet (FortiGate firewalls and FortiClient EMS), one of the first policies they setup was to block access to the internal ERP system from any client running Hyper-V (No I don't know why lol).

Adbot
ADBOT LOVES YOU

Nuclearmonkee
Jun 10, 2009


Pile Of Garbage posted:

Hasn't microsegmentation been superseded by zero-trust? That's the latest hotness as I understand it, running NGFWs and devices with endpoint clients that communicate info to the firewall which determines policy. My employer has started doing it internally with Fortinet (FortiGate firewalls and FortiClient EMS), one of the first policies they setup was to block access to the internal ERP system from any client running Hyper-V (No I don't know why lol).

You need both zero trust access and micro-segmentation. Your users are granted access to resources specifically by policy, and the infrastructure they're accessing needs to segmented so that each thing can only talk to the things it needs to talk to.

Pile Of Garbage
May 28, 2007



Ah yeah good point!

Tetramin
Apr 1, 2006

I'ma buck you up.

Pile Of Garbage posted:

That poo poo sounds insane. Also a good use-case for FortiGates with VDOMs if you're doing multi-tenant stuff like that. Also seconding using Ansible for network management, we've had some big wins with it specifically for automating mass management cut-over and firmware upgrades.

On the opposite end of the spectrum over a decade ago I came into a multi-tenant "private cloud" environment where all the customer servers were on the same loving /24. One of the first things I did there was setup separate VLANs lol

We do have an ACI environment that we use to host like, 20 or so of our customers, and that works pretty well, but I don’t know exactly why we only use it for some customers.

It’s kinda hard to explain, but my division used to be two separate companies that made nearly identical products, both of them were acquired and form our division. Both product suites are still sold and maintained, and have totally separate environments and data centers.

Anyways, the environment for the other product does this segmentation using Fortinet VDOMs and a multi context ASA for the hosted environment, and it works fantastically with no risk of maxing out lol.

Managing the ACLs isn’t actually that bad. We do use Ansible for some things, but mostly for things like setting up new VLANs for a new customer, upgrades and poo poo like that.

The guy on our team who was building it out left the company before I joined, so I’m gonna take the opportunity to get some experience with automation and beef up what we can use it for.

Tetramin fucked around with this message at 20:24 on Sep 1, 2023

Partycat
Oct 25, 2004

ClearPass as RADIUS for auth templates is fine and works well at the edge. Supposedly it will connect with FortiManager to push rules or tags for role based policy into their firewalls to control access dynamically between groups.

I’m not positive what the intended architecture is though. Is that like, PVLAN or group based VLAN access so still somewhat horizontal between same role users? Or is it like VXLAN with tags , user role policy based rules that are address agnostic?

Aruba also offers dynamic tunneling for users from their edge, akin to how they do so through access points with wired ports. You can backhaul the user to a controller/gateway appliance and drop them on the appropriate network there , if you want that sort of thing.

funk_mata
Nov 1, 2005

I'm hot for you and you're hot for me--ooka dooka dicka dee.
Clapping Larry
I'm doing some network studying for the hell of it and I'm curious... Has there been widespread adoption of TRILL or SPB in enterprises? The last I heard of those two technologies was that vendors were charging an arm and a leg for the feature. I figured if that were the case then admins would throw up their hands and say RSTP is good enough, or maybe the "advent" of leaf-spine and L3 to the TOR made the technology obsolete.

Partycat
Oct 25, 2004

As far as I know TRILL may underpin some vendor solution but it did not see much adoption and is out of favor , VXLAN EVPN being the new hotness .

SamDabbers
May 26, 2003



It's much cleaner to do a layer 2 overlay on top of a layer 3 routed network than it is to try to route layer 2. Also you can do it from the edges and don't have to upgrade all the switches.

Nuclearmonkee
Jun 10, 2009


Partycat posted:

As far as I know TRILL may underpin some vendor solution but it did not see much adoption and is out of favor , VXLAN EVPN being the new hotness .

SamDabbers posted:

It's much cleaner to do a layer 2 overlay on top of a layer 3 routed network than it is to try to route layer 2. Also you can do it from the edges and don't have to upgrade all the switches.

Yeah VXLAN is what you want. TRILL was a very short lived thing that never caught on. VXLAN is great because you can get pretty much all of the benefits of splitting out horrible L2 failure domains with a good config when you replace your core and distribution switches. It's better the closer you get to the edge, but it doesn't have to get all the way there. I have a ton of industrial networking that still lives on old IOS flavored industrial switching that goes across the VXLAN infrastructure. Just trunk it to a leaf, slap some anycast gateways on there and you're set.

Pile Of Garbage
May 28, 2007



Figured I'd post this here as a heads-up for anyone with Cisco Emergency Responder: https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-cer-priv-esc-B9t3hqk9

Partycat
Oct 25, 2004

Going to go EOL announced soon anyway so time to upgrade

Partycat
Oct 25, 2004

Patch came out for that over the weekend but it has a really good note:

quote:

COP file provides fix to address CSCwh34565 in CER 12.5.1SU4. This COP file is also applicable to CUCM 14.0.1.14890-65 (DoD specific release)

oops

guppy
Sep 21, 2004

sting like a byob
Does the LACP rate (for PDUs) have to match on both ends? I know that when you configure LACP on port-channel members, the rate you specify (fast or slow) dictates how fast the device tells its partner it wants to receive PDUs, but if one side is configured for fast and the other side is configured for slow, will that prevent the link from functioning? Or does it only dictate how fast a failed member link is pruned once it fails?

Filthy Lucre
Feb 27, 2006
It will come up and mostly work.

I've seen some occasional random drops when the timers are mismatched.

Pile Of Garbage
May 28, 2007



As I understand PDU rate just affects how quickly the port-channel can respond to a link failure and how quickly it will recover. Having mismatched PDU rate would probably just increase both the time to respond and time to recover.

Best practice is to ensure that both sides are set to the same rate. If a connected device does not have a configurable rate then consult its doco to determine what its default rate is so you can match. Worst case I guess you can do a capture and figure out the rate.

Nuclearmonkee
Jun 10, 2009


guppy posted:

Does the LACP rate (for PDUs) have to match on both ends? I know that when you configure LACP on port-channel members, the rate you specify (fast or slow) dictates how fast the device tells its partner it wants to receive PDUs, but if one side is configured for fast and the other side is configured for slow, will that prevent the link from functioning? Or does it only dictate how fast a failed member link is pruned once it fails?

Slowest configured timer wins. It'll work regardless though.

Even if it's set to slow it'll behave as long as your problem isn't something where the physical carrier signal remains up while LACP is down, which is rare.

Nuclearmonkee fucked around with this message at 19:36 on Oct 9, 2023

guppy
Sep 21, 2004

sting like a byob
Thanks, this is about what I thought. Coworkers are telling me it won't work if they're mismatched, which does not comport with what I was reading. I plan to match them, they are already up, I'm just having issues that may require a change and I'm wondering if the link will be down after changing one until the other side matches. Sounds like it won't.

Partycat
Oct 25, 2004

For whatever reason I see more bugs related to fast BPDU over slow , but I never figured why it would matter.

guppy
Sep 21, 2004

sting like a byob
This seems like a basic question, but is there an easy way to capture LACP traffic (and only LACP traffic) on a Cat9k/IOS-XE? I'm aware of debug lacp and maybe that will be good enough, but I'm not sure which side of this port-channel is the issue. I'm already capturing it on the other side, which is where we have been assuming the problem lies, but now I need to rule this side in or out. Googling for this is almost impossible as you get all kinds of documentation for entirely different operating systems like IOS-XR and NX-OS.

The interfaces in question are extremely high-throughput and just capturing all traffic from the interface is going to be a problem. Some other Cisco OSes have a very simple lacp packet-capture, but that doesn't seem to exist here. I've seen monitor capture, but it's not clear to me how/whether I can capture L2 traffic with it; all their examples specify ports and the like that aren't going to apply here.

Partycat
Oct 25, 2004

Monitor capture on a port won’t include CPU handled traffic anyway when a process is involved - I don’t recall ever seeing LACP BPDU in there . Debug with a debug condition might be good , if the information you’re looking for isn’t shown in neighbor details in one of the various other output commands

guppy
Sep 21, 2004

sting like a byob
The specific issue I'm trying to troubleshoot is the LACP channel going down very, very briefly -- but it's enough to interrupt real-time stuff. The Catalyst side doesn't even report the issue because it's so brief. The other device says it isn't receiving the PDU in time, and we are trying to figure out if the Catalyst side isn't sending it in time/is sending something malformed, or if the other device isn't processing it in time.

Filthy Lucre
Feb 27, 2006
This is the type of behavior I've seen when the timers are mismatched.

tortilla_chip
Jun 13, 2007

k-partite
Is ELAM still supported on Catalyst 9000?

edit: https://www.cisco.com/c/en/us/suppo...-hId-1997310445

tortilla_chip fucked around with this message at 18:14 on Oct 11, 2023

guppy
Sep 21, 2004

sting like a byob
The timers were definitely not mismatched.

tortilla_chip posted:

Is ELAM still supported on Catalyst 9000?

edit: https://www.cisco.com/c/en/us/suppo...-hId-1997310445

Thanks! I'll take a look at this tomorrow.

We are actually thinking now that it is an issue with a scheduled task on the Catalyst side that is unexpectedly (but every time) screwing up and taking up excess CPU time. For now we have changed the LACP timers and we'll see how things go tomorrow.

Incidentally, for anyone wondering about the original question of whether the link would function with them mismatched, it will. I changed them on the Cisco side and the interface went down for a moment and then came back up, and continued to be up indefinitely. I am enjoying being the one who was right about this, mainly because it would have been a pain in the rear end otherwise (but also a little bit because I read up on the subject and am apparently the only one).

Partycat
Oct 25, 2004

LACP does fall apart if you have very high CPU, but that I don't think would be the problem.

If you have a fast/slow mismatch then sure just adjust it.

Like I said on IOS-XE there's "Wireshark" built in but if you don't see the BPDU in your edge port capture grab the control plane. I think you'll see it in the port capture, despite what I said before.

Tetramin
Apr 1, 2006

I'ma buck you up.
Upgrading our ISE deployment 2.7 -> 3.1 on the 10th… Im a little nervous about it as I’ve never performed a major upgrade, I’ve applied minor patches in a two node deployment. Ours is 2 admin nodes + 4 PSNs. Our current PSNs were too small, so we had new VMs built and preinstalled 3.1 to those, so I’ll need to figure out at what point to take the current nodes out of the deployment, and when to bring the new ones in.

Idk anybody been through this before? I’m just a bit concerned because I have a 2 hour maintenance window, and it seems like if something goes wrong, there’s a decent chance it will require a full re imaging, but maybe restoring the latest good VM backup would work?

Opening a pre emotive TAC case today to try and get some questions answered, but wondering if anyone’s got experience with this and can share any insight.

E: I really wish we were using a VIP on our F5 for ISE, and I could get all new VMs so I could build it standalone, and when the time comes just change the pool to point to the new build, but no luck

Tetramin fucked around with this message at 21:36 on Oct 31, 2023

uhhhhahhhhohahhh
Oct 9, 2012
2 hours is extremely ambitious. ISE always takes way longer than you'd hope and rarely go to plan. I once worked from 10am to 1am fixing a broken ISE deployment.

Cisco's written guides for how to do an upgrade are good (but they don't really help you if something goes wrong). I think you upgrade the secondary admin node first, which becomes the primary in its own 'deployment', then you upgrade the PSNs and put them on the new deployment, then finally the last admin node (and reboot it as primary if you want), there should be almost no actual downtime for end devices assuming nothing goes wrong. You'll probably spend at least 2 hours just waiting for all the nodes to shutdown and reboot without even counting uploading the image to the repositories and installing, which I've seen take an hour. This was doing upgrades from 2.x -> 2.x though. I don't know if anything changes going from 2 -> 3.

Prescription Combs
Apr 20, 2005
   6

Tetramin posted:

Upgrading our ISE deployment 2.7 -> 3.1 on the 10th… Im a little nervous about it as I’ve never performed a major upgrade, I’ve applied minor patches in a two node deployment. Ours is 2 admin nodes + 4 PSNs. Our current PSNs were too small, so we had new VMs built and preinstalled 3.1 to those, so I’ll need to figure out at what point to take the current nodes out of the deployment, and when to bring the new ones in.

Idk anybody been through this before? I’m just a bit concerned because I have a 2 hour maintenance window, and it seems like if something goes wrong, there’s a decent chance it will require a full re imaging, but maybe restoring the latest good VM backup would work?

Opening a pre emotive TAC case today to try and get some questions answered, but wondering if anyone’s got experience with this and can share any insight.

E: I really wish we were using a VIP on our F5 for ISE, and I could get all new VMs so I could build it standalone, and when the time comes just change the pool to point to the new build, but no luck

Don't forget to get your licenses converted. Not sure if ISE is kosher with VM snapshots but maybe if it goes poorly you could restore them that way? Have you run the upgrade readiness tool (URT)?

gooby pls
May 18, 2012



Back up and restore is the way to go for sure. This blog post hits on pretty much everything.

https://www.lookingpoint.com/blog/cisco-ise-3.0-major-upgrade

Upgrade your licenses, back up your system certs (they’re not in the config back up), and don’t forget to re-join to the domain if you’re doing anything with ad.

2 hours might be cutting it for the upgrade in total, but as long as your devices fail over properly then it should be pretty hitless for end users.

Kazinsal
Dec 13, 2011



gooby pls posted:

Back up and restore is the way to go for sure. This blog post hits on pretty much everything.

https://www.lookingpoint.com/blog/cisco-ise-3.0-major-upgrade

Upgrade your licenses, back up your system certs (they’re not in the config back up), and don’t forget to re-join to the domain if you’re doing anything with ad.

2 hours might be cutting it for the upgrade in total, but as long as your devices fail over properly then it should be pretty hitless for end users.

The local ISE expert at the VAR I work for is in agreement. With the additional caveat of "double your estimated time, and buy yourself a bottle of your favourite liquor for afterwards".

Tetramin
Apr 1, 2006

I'ma buck you up.

Prescription Combs posted:

Don't forget to get your licenses converted. Not sure if ISE is kosher with VM snapshots but maybe if it goes poorly you could restore them that way? Have you run the upgrade readiness tool (URT)?

I’m running URT on Thursday when I get back from some PTO.

My planned method is to bring in new PSNs which already have 3.1 installed, but I don’t have new servers for the SAN/PAN. I really wish I did have those but I doubt ill be able to get those in time..is this doable? I assume not needing to sit and upgrade the PSNs will help.

(On that note, do I want to remove the current PSNs from the deployment prior to upgrading the SAN, and then bring in the new ones after that’s done?)


I can probably expand the window if I need to. Should I shoot for like 4-5 hours?

Also we only are using ISE for TACACS and RADIUS for managing our network gear. No dot1x or MAB, if that makes a difference.

Thanks for the input!

Nuclearmonkee
Jun 10, 2009


gooby pls posted:

but as long as your devices fail over properly then it should be pretty hitless for end users.

At least for us, they did not. It flipped the hell out and the admin nodes poo poo the bed. Local PSNs kept working but it was disruptive to finally get it all stitched back together. Second time it's done this, and further reinforced my desire to remove ISE from the organization next year.

Tetramin
Apr 1, 2006

I'ma buck you up.

gooby pls posted:

Back up and restore is the way to go for sure. This blog post hits on pretty much everything.

https://www.lookingpoint.com/blog/cisco-ise-3.0-major-upgrade

Upgrade your licenses, back up your system certs (they’re not in the config back up), and don’t forget to re-join to the domain if you’re doing anything with ad.

2 hours might be cutting it for the upgrade in total, but as long as your devices fail over properly then it should be pretty hitless for end users.

Just want to make sure, if I’m doing an upgrade in place for my admin nodes, I don’t need to worry about rejoining the domain do I? My PSNs are fresh new VMs, but admin nodes will stay the same.

Ran URT today and it predicts an hour for each admin node, and one hour per PSN, but luckily I’m not upgrading the PSNs. I expanded my window to 5 hours.

Went to open a TAC case for the licenses and found that ISE ain’t on my account, so I’ll have to figure that out tomorrow and hopefully can get the licenses in time.

I think I’ve got a procedure close to complete, gonna ask Cisco about a couple of things when I can get a case opened, and then write out each step ahead of time. I still need to get clarity on if I need to do anything with certs for the new PSNs. It looks like my current PSNs are just using self signed, so do I need to generate new certs on the new nodes? They’re going to have different hostnames and IPs.

This is probably the “biggest” upgrade I’ve handled alone, and I tend to overthink and ask a billion questions when I’m learning how to do something… so thanks for the input y’all.

Tetramin fucked around with this message at 00:52 on Nov 3, 2023

Prescription Combs
Apr 20, 2005
   6
Worse comes to worse you rebuild the VM(s) and restore them from a backup.

MrMoo
Sep 14, 2000

https://coloradosun.com/2023/11/13/fastest-internet-service-terabits-denver-sc23/

This sounds pretty neat, 6Tb interwebs connection for a convention.

Pile Of Garbage
May 28, 2007



MrMoo posted:

https://coloradosun.com/2023/11/13/fastest-internet-service-terabits-denver-sc23/

This sounds pretty neat, 6Tb interwebs connection for a convention.

:holymoley: I need one of those for my apartment.

falz
Jan 29, 2005

01100110 01100001 01101100 01111010

MrMoo posted:

https://coloradosun.com/2023/11/13/fastest-internet-service-terabits-denver-sc23/

This sounds pretty neat, 6Tb interwebs connection for a convention.

Well, optical vendors use SC conference to test gear in production and put out flashy news releases, so it's a good thing to test waves between Denver and Chicago (starlight 710) per article. Pretty nice for press releases.

Also lol at whoever wrote this article

quote:

This year, the team installed the next generation of internet protocol technology, called IPv6.

Eletriarnation
Apr 6, 2005

People don't appreciate the substance of things...
objects in space.


Oven Wrangler
Yeah I laughed at that too, and the next line - "It’s not that new, though. It has similar speeds to the decades-old IPv4..."

Well, yes, IPv6 is also decades old at this point and your addressing system doesn't have much to do with how fast you're going.

Docjowles
Apr 9, 2009

That was goofy but I thought it was a reasonable article overall for a random local newspaper that’s not a trade journal or anything.

I used to work for a regional ISP (coincidentally also in Colorado). One time our marketing team sent a mailing with this sweet stealth bomber on it advertising our speeds in miles per hour :confused: The lead network engineer almost quit over that lol

Adbot
ADBOT LOVES YOU

MrMoo
Sep 14, 2000

Eletriarnation posted:

Well, yes, IPv6 is also decades old at this point and your addressing system doesn't have much to do with how fast you're going.

Facebook and others use IPv6 because the headers are smaller and less processing needed in routing, something like 10% faster?

China raised an order that all new equipment must support IPv6, so it helps to court their monies no doubt.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply