Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
bsaber
Jul 27, 2007
Not sure where to ask this so I'm just going try here as I work for a small shop (only IT person). We have a 1Gb fiber connection from the enterprise division of our ISP. Over the last month our bandwidth would drop significantly at random times. I called enterprise support and they keep saying everything is fine and refused to send out a tech to check. I finally called our account manager and a tech was dispatched. They did their tests and it shows they're getting full speeds (980Mbps). But you try to load anything on our end and it's super slow. Speed tests are showing as low as 100Mbps down/up at some times. They ended up replacing the edge router (it's an ADVA but uncertain the model number) and just said "your problem" and left basically. Our account manager promised to bring this up with engineering but...

I'm not a networking person so a bit out of my depth. The tech called engineering to take a look while he was onsite but the engineer he talked to said everything is fine. The tech did admit to me that something is definitely "not right" but nothing he can do. He replaced the ADVA as I mentioned above, he ran tests from the ADVA to their test point and everything is fine and getting the full 980Mbps. But when I attach ANY of our switches to their edge router, the ADVA, the tests tank and get down to 250Mbps (using his test meter plugged into our switch(es)). Here's the thing, when I disconnect everything from our switches and the tech plugs his test meter into it to run the tests it'll be slow. But if we plug directly into the ADVA unit, he's able to get full speeds. The ISP's utilization charts show we're not using much bandwidth at all either. They can see the spikes in utilization when the tech was running tests. So we changed to a different switch and the same thing happened. So shouldn't be a problem with the switches (?).

As mentioned this happens at random times. Today for example, I'm able to test full 900Mbps down/up in the morning. But this afternoon, I was getting ~300Mbps down and only ~200Mbps up. Now all these tests that I've been doing on my own are just using different speed test sites. So I decided to spin up a VM on our cloud provider and setup iperf3. I connected directly to their ADVA with a laptop and used the following parameters to do the iperf test: iperf3 -c IPADDRESS -u -b 5000m -l 65500

These are the results of the iperf tests:
sender: 807 Mbits/sec, 0 jitter, 0% loss
receiver: 83.8 Mbits/sec, 0.498 ms jitter, 2.5% loss

Again, I'm not a networking person so not sure how else to troubleshoot. The ISP is just saying it's something on our end and that the speed test sites don't mean anything. But loading a video or even webpages at times would take awhile so there's obviously something wrong. Nothing has changed on our end network wise when the problem started. What should I try next? What other troubleshooting sets should I take? Any help or suggestions would be greatly appreciated.

bsaber fucked around with this message at 01:50 on Jun 23, 2023

Adbot
ADBOT LOVES YOU

bsaber
Jul 27, 2007

bolind posted:

I don't have a ton of magic fixes, but could you run a semi-agressive ping towards a trusty host, and demonstrate packet loss over time (let it run over night and/or a weekend.).

Thanks for the suggestion.

I was able to graph both download and upload speed along with ping and sent it to the ISP. After looking at a 7 day period I noticed that the drops would happen about once an hour. Strangely the connection was extremely stable averaging ~900Mbps both down and up on the day July 4th.

I suggested that perhaps they have us on a shared circuit but the tech support insisted that “it’s not possible”. Still having this issue almost a month later but the drops are not as massive and it doesn’t interfere with day to day operations as much now. The ISP has stopped responding to my messages as well so… yeah…

bsaber
Jul 27, 2007

M31 posted:

2.5% packet loss is pretty high and will definitely impact your throughput as TCP will basically half your speed on packet loss. Try running iperf with TCP instead of UDP.

I don't know what kind of shop you are running, but see if there is any high interference machinery/cabling/construction nearby? And test the cables if you haven't done so already by now.

Reminds me a bit of all those stories where the issue was caused by a faulty microwave.

First thing we tried when the tech came out was replaced the cables. So it's not the cables. The server room doesn't have any high interference things nearby.

Just did another iperf test with TCP instead of UDP and there's no packet loss. I contacted our account manager at the ISP and he said he'll double check that we're not on a shared circuit but never got back to me...

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply