Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
the spyder
Feb 18, 2011
ASA question: Other then a static route, do I need a ACL to allow internet access in the setup below?

I have a 5515-x (8.6.2) and a HP 5604zl. I have ip routing enabled on the HP, I can ping between the vlans- all of that works. My problem is I can not get internet on any of the vlans. I have my ip route configured on the HP and a static route configured on the ASA. I do not have any ACL's configured and I have seen it mentioned that there needs to be.

Thanks!

Adbot
ADBOT LOVES YOU

n0tqu1tesane
May 7, 2003

She was rubbing her ass all over my hands. They don't just do that for everyone.
Grimey Drawer

Erwin posted:

Okay, another dumb question, but I want to confirm my understanding coming from the Procurve world:

I have a VLAN 10 for VOIP with Polycom phones. The phones are configured explicitly for that vlan, so they tag their traffic. On the Procurves, ports have a primary vlan, but can also be a member of other tagged vlans. Trunks are something entirely different (aggregated links).

With Cisco, it sounds like if I want to plug a computer into a port and have it be on vlan 1, then disconnect it and connect a phone tagged on 10, it needs to be a trunk, even though trunks are defined as connecting two switches in the Cisco world. Is that right? Is that the best way to handle it? We're not daisy-chaining the computer through the phone, so it doesn't need to recognize both at the same time, but I don't want to reconfigure anything if I swap a computer out for a phone, and I want the phones on their own VLAN. Not that that's the end of the world, but I'm lazy.

Like ragzilla said, the "switchport voice vlan 10" will accept tagged traffic intended for vlan 10. This allows you to use the auto-qos command, and traffic going over the voice vlan will be given higher priority.

On the newer switches, a port configuration for a phone with auto-qos would be done something like this:

interface GigabitEthernet1/0/14
switchport access vlan 100
switchport mode access
switchport voice vlan 10
auto qos voip cisco-phone

The "auto qos voip cisco-phone" command will add a few other lines of configuration to the port, and a bunch of lines of configuration to the global config. You could use this config on all your ports, and will allow you to move phones around without having to reconfigure the port every time. It'll keep the PCs on your PC vlan, and your phones on your voice vlan.

You can also configure the ports as a trunk, and connect a PC to it. Since PCs don't normally tag their traffic, it'll default to sending the untagged traffic over vlan 1. You can configure this on the port as well, using the "switchport trunk native vlan XXX" command, which will put all untagged traffic coming in on that port into vlan XXX.

Older Cisco switches, such as the 3500XL switches, didn't automatically trunk the port for a phone with the voice vlan command on an access port, and the port had to be configured to trunked. This way, if you had a PC hanging off the back of a phone, you had to use the "switchport trunk native vlan" command to put the data from the PC onto the correct vlan. For purposes here, this command functions the same on a trunk port as the "switchport access vlan" command works on an access port. Such a configuration would look like this:

interface FastEthernet 0/14
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 100
switchport voice vlan 130

With a trunked port though, by default it'll accept tagged traffic for any vlan, which isn't always desirable for security reasons. The voice vlan command only accepts tagged traffic for the voice vlan, and not for any others.

EDIT: Wow, that's a lot of :words:

BelDin
Jan 29, 2001

the spyder posted:

ASA question: Other then a static route, do I need a ACL to allow internet access in the setup below?

I have a 5515-x (8.6.2) and a HP 5604zl. I have ip routing enabled on the HP, I can ping between the vlans- all of that works. My problem is I can not get internet on any of the vlans. I have my ip route configured on the HP and a static route configured on the ASA. I do not have any ACL's configured and I have seen it mentioned that there needs to be.

Thanks!

Default outbound behavior is to drop, so you will have to set up a permit ACL outbound on your inside interface. Do you need to/Have you set up PAT?

the spyder
Feb 18, 2011
I should clarify-
I have internet if I set the default gateway on the hosts to the ASA's inside IP.

Can you give me a example? I believe we do, I will need to remote in later to answer that.

ragzilla
Sep 9, 2005
don't ask me, i only work here


BelDin posted:

Default outbound behavior is to drop, so you will have to set up a permit ACL outbound on your inside interface. Do you need to/Have you set up PAT?
Default ASA policy is to permit to lower security-level unless that changed in -X. The usual thing I see people missing is an outbound NAT/PAT if needed.

BelDin
Jan 29, 2001

ragzilla posted:

Default ASA policy is to permit to lower security-level unless that changed in -X. The usual thing I see people missing is an outbound NAT/PAT if needed.

You're right, my bad. My difficulties were when you had the same security level on two interfaces. Of course, when I inherited our firewalls they were being used as routers with that exact configuration.

I think I need to take a break. Been off lately.

the spyder
Feb 18, 2011

ragzilla posted:

Default ASA policy is to permit to lower security-level unless that changed in -X. The usual thing I see people missing is an outbound NAT/PAT if needed.

What should the outbound NAT/PAT look like?

ragzilla
Sep 9, 2005
don't ask me, i only work here


the spyder posted:

What should the outbound NAT/PAT look like?

Typically something similar to:

code:
global (outside) 1 interface
nat (inside) 1 192.0.2.1 255.255.255.0
But replace '192.0.2.1 255.255.255.0' with your actual inside range.

Jelmylicious
Dec 6, 2007
Buy Dr. Quack's miracle juice! Now with patented H-twenty!

the spyder posted:

What should the outbound NAT/PAT look like?

Since you can reach the internet if you are directly connected to the ASA, my guess would be that you don't have any routes back to your other subnets. Does the ASA know it should go back over the HP to reach your inside networks?

H.R. Paperstacks
May 1, 2006

This is America
My president is black
and my Lambo is blue
Anyone know of a good *nix based solution for introducing latency? I am looking for something cheap I can put into our small local lab to help simulation latency on a few lings to mimic our global stuff.

If anyone has a suggestion on traffic generation that's free as well, that would be great. We have all this stuff in our big lab in VA, but I am cobbling something together here for the team so we don't have to travel.

ragzilla
Sep 9, 2005
don't ask me, i only work here


routenull0 posted:

Anyone know of a good *nix based solution for introducing latency? I am looking for something cheap I can put into our small local lab to help simulation latency on a few lings to mimic our global stuff.

If anyone has a suggestion on traffic generation that's free as well, that would be great. We have all this stuff in our big lab in VA, but I am cobbling something together here for the team so we don't have to travel.

I think dummynet on fbsd is the defacto standard for packet mangling on the cheap.

Sepist
Dec 26, 2005

FUCK BITCHES, ROUTE PACKETS

Gravy Boat 2k

ragzilla posted:

Typically something similar to:

code:
global (outside) 1 interface
nat (inside) 1 192.0.2.1 255.255.255.0
But replace '192.0.2.1 255.255.255.0' with your actual inside range.

He's using 8.6 code

code:
 object network obj_any
   subnet 0.0.0.0 0.0.0.0
   nat (inside,outside) dynamic interface
You can replace the 0 subnet with your matching inside networks if you want, you get the jist I hope.

Doesn't look like your PAT is the issue though since internet is fine if the host gateway is configured as the ASA. Does the HP have a default route to the ASA inside IP?

Sepist fucked around with this message at 15:22 on Sep 21, 2012

Tasty Wheat
Jul 18, 2012

routenull0 posted:

Anyone know of a good *nix based solution for introducing latency? I am looking for something cheap I can put into our small local lab to help simulation latency on a few lings to mimic our global stuff.

If anyone has a suggestion on traffic generation that's free as well, that would be great. We have all this stuff in our big lab in VA, but I am cobbling something together here for the team so we don't have to travel.

D-ITG, Distributed Internet Traffic Generator
http://www.grid.unina.it/software/ITG/

I have used this in the past for some QoS projects

the spyder
Feb 18, 2011

Sepist posted:

He's using 8.6 code

code:
 object network obj_any
   subnet 0.0.0.0 0.0.0.0
   nat (inside,outside) dynamic interface
You can replace the 0 subnet with your matching inside networks if you want, you get the jist I hope.

Doesn't look like your PAT is the issue though since internet is fine if the host gateway is configured as the ASA. Does the HP have a default route to the ASA inside IP?

Thanks! I converted it last night, but did not have a chance to test it.

The HP is 10.20.28.254 and has: ip route 0.0.0.0 0.0.0.0 10.20.28.1
The ASA is 10.20.28.1. It has a static route: 10.20.0.0 255.255.0.0 10.20.28.254
It also has a existing network obj_any statement that fairly closely matches that one. I will check when I get to that site.

doomisland
Oct 5, 2004

Yo. What model router do people use for internet routing? I would need something that is able to hold full BGP tables. I was looking at the ASR 1004 and maybe the 7603-S as something of a possibility with the correct route processor cards. Anything else in the Cisco catalog that would fit the bill? I'm not familiar with Cisco product families so not sure if I'm missing some obvious solution without reading every data sheet.

ragzilla
Sep 9, 2005
don't ask me, i only work here


doomisland posted:

Yo. What model router do people use for internet routing? I would need something that is able to hold full BGP tables. I was looking at the ASR 1004 and maybe the 7603-S as something of a possibility with the correct route processor cards. Anything else in the Cisco catalog that would fit the bill? I'm not familiar with Cisco product families so not sure if I'm missing some obvious solution without reading every data sheet.
ASR1k with an ESP10 or greater I think.
ASR9k is also a good option (check 9001).
7600/6500 is getting a bit long in the tooth but is a good platform if you stay within its limitations.

doomisland
Oct 5, 2004

ragzilla posted:

ASR1k with an ESP10 or greater I think.
ASR9k is also a good option (check 9001).
7600/6500 is getting a bit long in the tooth but is a good platform if you stay within its limitations.

Thanks, looks like I was looking around the right products.

Bluecobra
Sep 11, 2001

The Future's So Bright I Gotta Wear Shades

doomisland posted:

Yo. What model router do people use for internet routing? I would need something that is able to hold full BGP tables. I was looking at the ASR 1004 and maybe the 7603-S as something of a possibility with the correct route processor cards. Anything else in the Cisco catalog that would fit the bill? I'm not familiar with Cisco product families so not sure if I'm missing some obvious solution without reading every data sheet.
We have two Cisco 3845's each with 1Gb of memory and this has been pretty solid for years. I just checked and saw that we have ~420K IPv4 routes and ~10K IPv6 routes.

CrazyLittle
Sep 11, 2001





Clapping Larry

Bluecobra posted:

We have two Cisco 3845's each with 1Gb of memory and this has been pretty solid for years. I just checked and saw that we have ~420K IPv4 routes and ~10K IPv6 routes.

The 1gb memory is kinda the part that matters the most. You can still get a Sup-720 that does not have enough memory to store full tables.

the spyder
Feb 18, 2011
It looks like I figured it out- it was my Hp 5406zl. I needed to assign a IP to the default vlan, use that as the default gateway for the edge switches, and restart everything. Came right up. This works great, thanks!

doomisland
Oct 5, 2004

Bluecobra posted:

We have two Cisco 3845's each with 1Gb of memory and this has been pretty solid for years. I just checked and saw that we have ~420K IPv4 routes and ~10K IPv6 routes.

Yeah that is about what the internet looks like for us. I checked out the 39xx chassis and it looks like it only can go up to 350Mbps. Yikes! Am I reading that wrong or do the the expansion slots matter? It looks like you'll only get line rate on-card with those expansion and as soon as it needs to be routed you run into the 350Mbps issue. For comparison the ASR 9001 can do 120Gbps and the smallest router we're buying now can do 20Gbps. I suppose I should've mentioned it being able to route probably at least 10 Gbps.

BurgerQuest
Mar 17, 2009

by Jeffrey of YOSPOS
You're reading it right. Any old router can in theory receive full routes with enough memory, but your actual load requirements may dictate spending more on beefier equipment.

ior
Nov 21, 2003

What's a fuckass?

CrazyLittle posted:

The 1gb memory is kinda the part that matters the most. You can still get a Sup-720 that does not have enough memory to store full tables.

Don´t forget TCAM on the hw forwarding platforms!

doomisland
Oct 5, 2004

BurgerQuest posted:

You're reading it right. Any old router can in theory receive full routes with enough memory, but your actual load requirements may dictate spending more on beefier equipment.

Yeah, the ASRlk2 looks pretty sweet since you can software upgrade to meet your needs. Though I'm trying to figure out why someone would get the dual height 10 port SPA over two 8 port SPAs which are single price. I'm going to assume price or niche requirement.

a world called z0r
Aug 31, 2012

jwh posted:

Nexus 5k experiences: yay? nay?

I'm thinking about bringing them in as replacements for a number of 3750s.

The driver is more affordable 10g density.

5510 = NO

5596 = maybe

a world called z0r
Aug 31, 2012

doomisland posted:

Yo. What model router do people use for internet routing? I would need something that is able to hold full BGP tables. I was looking at the ASR 1004 and maybe the 7603-S as something of a possibility with the correct route processor cards. Anything else in the Cisco catalog that would fit the bill? I'm not familiar with Cisco product families so not sure if I'm missing some obvious solution without reading every data sheet.

Juniper M and MX series
Cisco 7600 and 6500 series

BelDin
Jan 29, 2001

a world called z0r posted:

5510 = NO

5596 = maybe

I'm having some luck with a pair of 5548s right now, and other than the hoop jumping of converging our networks, they have been solid.

Definitely seconding that you don't want anything lower than a 5548. The 5548/96 also supports ISSU, so that when you want to do a firmware upgrade you can do it without a reboot.

Just remember that they don't do per interface MTU. *lol*

evil_bunnY
Apr 2, 2003

BelDin posted:

Just remember that they don't do per interface MTU. *lol*
Don't remind me :sigh:

ofwolfandan
Aug 13, 2004

FACE THE PIE THAT SHOULD NOT BE
Ok so I'm still playing around with this Cisco 860. I'm trying to get PPTP working so I can connect with my laptop through the windows 7 built in client. I can only get connected to it if I manually set the IP on my laptop. But even then I can only ping the inside local ip (192.168.1.1) of the router and nothing else on the inside of the network. If I try to connect without manually setting the IP in the client first, it tells me that their was a network error. Any suggestions?

code:
Building configuration...

Current configuration : 2409 bytes
!
! Last configuration change at 18:13:20 UTC Fri Jan 6 2006 by Admin
! NVRAM config last updated at 11:43:29 UTC Wed Jan 4 2006 by Admin
!
version 15.0
service config
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
aaa new-model
!
aaa authentication login defaut local
aaa authentication ppp default local
!
aaa session-id common
memory-size iomem 10
!
ip source-route
!
!
ip dhcp excluded-address 192.168.1.1
!
ip dhcp pool dhcp-pool
   network 192.168.1.0 255.255.255.0
   dns-server 8.8.8.8 8.8.4.4
   default-router 192.168.1.1
!
ip dhcp pool LAB-DHCP-POOL
   network 10.1.1.0 255.255.255.0
   dns-server 8.8.8.8 8.8.4.4
   default-router 10.1.1.1
!
ip dhcp pool PPTP_DHCP_POOL
   network 10.0.0.0 255.255.255.0
   dns-server 8.8.8.8 8.8.4.4
!
!
ip cef
!
!
vpdn enable
!
vpdn-group PPTP
! Default PPTP VPDN group
 accept-dialin
  protocol pptp
  virtual-template 1
 l2tp tunnel timeout no-session 15
!
license udi pid CISCO861-K9 sn FRX173191V0
!
username ofwolfandan privilege 15 password 7 11BA5D54351D5E100D38
!
interface FastEthernet0
!
interface FastEthernet1
!
interface FastEthernet2
!
interface FastEthernet3
!
interface FastEthernet4
 ip address inside.global.ip.address 255.255.255.240
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
!
interface Virtual-Template1
 description ## PPTP TUNNEL ##
 ip unnumbered Vlan1
 ppp encrypt mppe 128 required
 ppp authentication ms-chap-v2
 ppp timeout idle 360
!
interface Vlan1
 ip address 192.168.1.1 255.255.255.0
 ip access-group firewall in
 ip nat inside
 ip virtual-reassembly
!
interface Vlan10
 ip address 10.1.1.1 255.255.255.0
!
ip local pool PPTP_POOL 10.0.0.10 10.0.0.150
ip forward-protocol nd
no ip http server
ip http authentication local
no ip http secure-server
!
ip nat inside source list 1 interface FastEthernet4 overload
ip nat inside source static tcp 192.168.1.246 1167 interface FastEthernet4 1167
ip route 0.0.0.0 0.0.0.0 isp.gateway.ip.address
!
ip access-list extended firewall
 permit tcp any any
 permit udp any any
 permit ip any any
 deny   ip any any
!
access-list 1 permit 192.168.1.0 0.0.0.255
access-list 10 permit 10.1.1.0 0.0.0.255
!
control-plane
!
line con 0
 no modem enable
line aux 0
line vty 0 4
 transport input telnet
!
scheduler max-task-time 5000
end
**Update**
I just changed the PPTP_POOL to;
ip local pool PPTP_POOL 192.168.1.5 192.168.1.15

and added to the virtual-template 1 interface;
peer default ip address pool PPTP_POOL

Now I can get an address without having to manually enter it in the windows VPN client, but I can't resolve any host names on the network, access the internet while connected to the VPN. I can access network resources with the IP address.

Interface IP-Address OK? Method Status Protocol
Virtual-Access1 unassigned YES unset down down
Virtual-Access2 unassigned YES unset up up
Virtual-Access3 192.168.1.1 YES unset up up
Virtual-Template1 192.168.1.1 YES unset down down
Vlan1 192.168.1.1 YES NVRAM up up

ofwolfandan fucked around with this message at 04:48 on Sep 24, 2012

Mierdaan
Sep 14, 2004

Pillbug
QoS question:

Is it possible to enable QoS on a dedicated L2 switch handling only one traffic type, and end up with interfaces having more resources available to them than they did before QoS was enabled?

I ask because I'm troubleshooting a sporadic latency issue for ESXi hosts accessing iSCSI LUNs on a Compellent SAN, with a 3560 in between, and think it's tied it to output drops on the switch. That manifests for the VMware hosts as very short bursts of 200-500ms latency. Right now there's no QoS enabled at all, and Cisco's recommendation seems to be to enable QoS and give all the egress queue resources to one queue. Does that make sense?

jwh
Jun 12, 2002

I suppose that makes sense, sure.

Though I don't see how a different queue set would rectify a 200-500ms sporadic latency issue.

Mierdaan
Sep 14, 2004

Pillbug

jwh posted:

I suppose that makes sense, sure.

Though I don't see how a different queue set would rectify a 200-500ms sporadic latency issue.

I'm not sure either, unless microbursts are filling a buffer and this could help out?

Here's specifically what they're proposing:

code:
mls qos srr-queue output cos-map queue 2 threshold 3 0
mls qos srr-queue output dscp-map queue 2 threshold 3 0
mls qos queue-set output 2 threshold 2 100 100 50 3200
mls qos queue-set output 1 buffers 1 97 1 1
mls qos queue-set output 2 buffers 1 97 1 1
no mls qos rewrite ip dscp
mls qos

jwh
Jun 12, 2002

At 1mbit per millisecond, with default queue allocations, you're still servicing each queue well below each .25ms, or 0.25ms.

I just can't wrap my head around how the default queue allocations would be introducing such HUGE latency. I don't even think the per-port buffers are capable of storing much more than 2MB per 4-port bundle.

Is it possible the large increases in latency are the result of retransmission? I don't know how iscsi works. It can't be tcp drop retransmit, because that's much slower than what you've described.

Mierdaan
Sep 14, 2004

Pillbug
I don't know, I'll admit I'm over my head here - I've never even touched QoS before. Here's some debugging info I was looking at since the interface MTU looks like it's set to 9000, and since the hosts/SAN are set at 1500 I was worried I was misunderstanding when fragmentation would happen. Doesn't look like that's the case though, and my untrained eye doesn't see anything else really worrying here.

code:
!
interface GigabitEthernet0/14
 description Uplink to VMware host
 speed 1000
 flowcontrol receive desired
 spanning-tree portfast
end
code:
!
interface GigabitEthernet0/21
 description Uplink to Compellent controller
 flowcontrol receive desired
end
code:
     Transmit GigabitEthernet0/14             Receive
   1274627020 Bytes                       2783504370 Bytes
   1196222214 Unicast frames               976648783 Unicast frames
     10307309 Multicast frames                   908 Multicast frames
       815451 Broadcast frames                 60846 Broadcast frames
            0 Too old frames              2777708765 Unicast bytes
            0 Deferred frames                  82072 Multicast bytes
            0 MTU exceeded frames            5713021 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames            92269430 Minimum size frames
            0 8 collision frames            68636588 65 to 127 byte frames
            0 9 collision frames             3290717 128 to 255 byte frames
            0 10 collision frames            3308783 256 to 511 byte frames
            0 11 collision frames           10547064 512 to 1023 byte frames
            0 12 collision frames          798657963 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  8 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  -----> 0 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
    258744168 64 byte frames                       0 Valid frames, too small
    212610021 127 byte frames
      3177434 255 byte frames                      0 Too old frames
     12259823 511 byte frames                      0 Valid oversize frames
     30776869 1023 byte frames                     0 System FCS error frames
    689776659 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames
code:
#sh int gi0/14
GigabitEthernet0/14 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 588d.098d.268e (bia 588d.098d.268e)
  Description: Uplink to VMware host
  MTU 9000 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 1w3d, output 00:00:00, output hang never
  Last clearing of "show interface" counters 06:51:53
  Input queue: 0/75/0/0 (size/max/drops/flushes); -----> Total output drops: 81
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1308000 bits/sec, 131 packets/sec
  5 minute output rate 404000 bits/sec, 108 packets/sec
     4403681 packets input, 5210529306 bytes, 0 no buffer
     Received 162 broadcasts (0 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     3958684 packets output, 2139184582 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
code:
#sh int | inc output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 54
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 40
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 47 -----> VMware host
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 81 -----> VMware host
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 219 -----> VMware host
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 90 -----> VMware host
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 49 -----> VMware host
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 113 -----> VMware host
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 968 -----> Compellent SAN
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1730 -----> Compellent SAN
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2048 -----> Compellent SAN
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 768 -----> Compellent SAN
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 54
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

Mierdaan fucked around with this message at 00:55 on Sep 25, 2012

jwh
Jun 12, 2002

Troubleshooting queue drops is the worst thing in the world, in my experience.

But in this case, 81 queue drops over 4.4 million packets wouldn't worry me. I see much higher on ATM interfaces (pray for me).

I wish I could help more, but I'm it's hard to know exactly what to look at at this point. I would go out on a limb and say that more than likely, your infrastructure isn't the issue.

And as a general point about QoS, I'm fond of saying that it's not about making things better, it's about making other things worse. In your case, it doesn't look like there's much to make worse, so I'm not sure how much a robust QoS scheme is going to benefit you in the first place.

:(

Langolas
Feb 12, 2011

My mustache makes me sexy, not the hat

jwh posted:

Troubleshooting queue drops is the worst thing in the world, in my experience.

But in this case, 81 queue drops over 4.4 million packets wouldn't worry me. I see much higher on ATM interfaces (pray for me).

I wish I could help more, but I'm it's hard to know exactly what to look at at this point. I would go out on a limb and say that more than likely, your infrastructure isn't the issue.

And as a general point about QoS, I'm fond of saying that it's not about making things better, it's about making other things worse. In your case, it doesn't look like there's much to make worse, so I'm not sure how much a robust QoS scheme is going to benefit you in the first place.

:(

I am going to side with jwh on this too. That low of a drop count for that many packets isn't that much to worry about. If you had something like 200,000+ like I had on my 3750's a while back then you have to worry about it.

QoS may or may not help your issue there. It really could be a mixed bag for you and even cause more issues if its not done right.

ragzilla
Sep 9, 2005
don't ask me, i only work here


The QoS in this case isn't actual QoS, this is fixing the broken defaults in IOS. The way development seems to be going is that Cisco assumes everyone is running enterprisey QoS and sets up the hardware defaults that way which in the case of 3560/3750 leaves 75% of the buffer resources unused assigned to queues 0,2-3 while queue 1 services everything.

Sigh.

We had this same problem with Scale Computing backend network (don't touch them with a 10' pole if you have the chance) where the bursts would overflow switch buffers until TCP slowed down, it gets compounded by the fact you're typically looking at a many-to-one-port traffic pattern (lots of ESX boxes talking to a handful of SAN ports).

Try Cisco's suggestion (during a maintenance window) but keep in mind it may not help. We ended up moving to 4900/6500 to get some decent buffer depth.

Mierdaan
Sep 14, 2004

Pillbug
Thanks for the input. I'll go ahead with Cisco's suggestions this weekend and see if they have any impact.

So far the latency spikes don't really affect anything, but it's worrying to think about moving production Exchange/SQL over to this platform when there's :ghost: spooky performance issues :ghost: I can't explain. You guys have talked me off the ledge a bit though, I'm okay with the answer being that we move to a more robust switch for this fabric later if we really start to hit a performance wall.

edit:
They make performance graphs look like poo poo too :suicide:

Mierdaan fucked around with this message at 01:01 on Sep 25, 2012

jwh
Jun 12, 2002

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750e_3560e/software/release/12.2_55_se/command/reference/cli1.html#wp2144505

Here's a doc on queue set and buffer tuning, FYI.

Adbot
ADBOT LOVES YOU

nzspambot
Mar 26, 2010


that's a very good doc to read.

FYI a couple of pages ago I had the same issues with a 3750-X stack, no amount of tinking can really fix the issue with these switches.

The only thing which really worked for me was to look at which queue had the largest amount of traffic and make that have the most buffers. Even then I still see drops.

The boss found a Dell switch which has more buffers for a lot cheaper inc HBAs

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply