Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Thanks Ants
May 21, 2004

#essereFerrari


Any potential database performance issues are due to 'the network' or your lazy storage admin not putting the entire database on SSDs.

Adbot
ADBOT LOVES YOU

GreenNight
Feb 19, 2006
Turning the light on the darkest places, you and I know we got to face this now. We got to face this now.

Thanks Ants posted:

Any potential database performance issues are due to 'the network' or your lazy storage admin not putting the entire database on SSDs.

As if we have any SSD's.

anthonypants
May 6, 2007

by Nyc_Tattoo
Dinosaur Gum
Why worry about managing MS SQL when you should really be moving everything to a nice cloud-based NoSQL solution instead?

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

captkirk posted:

Make sure you don't know anything about disk, resource bottle necks, or the generally finite nature of physical memory. Those are all concerns for the systems team.

I'm so triggered right now.

GnarlyCharlie4u
Sep 23, 2007

I have an unhealthy obsession with motorcycles.

Proof

anthonypants posted:

Why worry about managing MS SQL when you should really be moving everything to a nice cloud-based NoSQL solution instead?

Because it's easier to make really big lovely and slow queries in MSSQL.
Be sure to nest views, avoid joins as much as possible, never partition DBs, and never truncate anything, ever.

When everything is hosed, blame "the server guys."

ChubbyThePhat
Dec 22, 2006

Who nico nico needs anyone else

GnarlyCharlie4u posted:

Because it's easier to make really big lovely and slow queries in MSSQL.
Be sure to nest views, avoid joins as much as possible, never partition DBs, and never truncate anything, ever.

When everything is hosed, blame "the server guys."

Who will, in turn, blame "the network".

Farts
Nov 4, 2005

ChubbyThePhat posted:

Who will, in turn, blame "the network".

And then the network guys blame the software vendor.

MrMoo
Sep 14, 2000

ChubbyThePhat posted:

Who will, in turn, blame "the network".

I'm surprised this is still a thing in nearly TYOOL 2016, but the majority of Cisco's product line up still cannot manage line speed so :shepspends:

psydude
Apr 1, 2008

99% of all network problems are sysadmins and DBAs trying to buy time to fix their gently caress ups. Throw in the developers if you're managing an F5.

BlueBlazer
Apr 1, 2010

Judge Schnoopy posted:

I don't know how anybody could go with meraki switches when they're easily 5x normal cisco gear. The pricing is out of their loving minds.

You could buy a truck of ubiquiti switches for the same price. Just throw them away if they cause any issue. The rep must be blowing your boss or something.

ZteleME
Dec 27, 2009
Was wondering if I truly need a bachelor's degree to advance in the IT field? I'm already working in IT for awhile now, but only have an associates.

And does it matter what school I go to. I mean I've been researching schools and noticed Western Governor's University and Southern New Hampshire University had some really great tuition. I can go to a local university (Boston area) but they're quite expensive. Thoughts?

Kashuno
Oct 9, 2012

Where the hell is my SWORD?
Grimey Drawer
Hey other Boston goon with an associates! I asked about WGU the other day and this might be relevant for you


skipdogg posted:

There are several of us here that have degrees from WGU. I finished my BS Oct 2014.


WGU is legit, I have an AAS from a CC, and took a couple semesters at Arizona State but never finished my degree. I worked just as hard at WGU as I did earning the AAS or taking classes at ASU. When I started I felt it was the best option for me at the time. There are definitely more options out there now that might be worth looking into*, but WGU gave me the opportunity to finish my Bachelors in a reasonable amount of time and I'm glad I did it.

* WGU, is fully accredited and no one has every questioned the validity of the degree I earned, but many traditionally brick and mortar colleges are starting to offer 100% online degrees that are the same as if you attended classes on campus. They can be way more expensive than WGU, most cost over 500 dollars a credit hour, but some might possibly find value in getting a degree from a nationally recognized university like ASU versus having to potentially explain what WGU is to someone. Several other 'regular' colleges have 100% online bachelor programs targeted at folks like us. Penn State World Campus, University of Texas System Online Consortium, and others.

AreWeDrunkYet
Jul 8, 2006

ZteleME posted:

Was wondering if I truly need a bachelor's degree to advance in the IT field? I'm already working in IT for awhile now, but only have an associates.

And does it matter what school I go to. I mean I've been researching schools and noticed Western Governor's University and Southern New Hampshire University had some really great tuition. I can go to a local university (Boston area) but they're quite expensive. Thoughts?

If you're past the junior level, the degree won't really matter for any more technical roles. You could probably be en enterprise architect with a high school diploma.

If your ambitions are in management though, you will want a degree. It's more of a check box than anything (especially for an internal promotion), but the reputation of the school can make a difference if you're interviewing with people who don't know you.

12 rats tied together
Sep 7, 2006

psydude posted:

99% of all network problems are sysadmins and DBAs trying to buy time to fix their gently caress ups. Throw in the developers if you're managing an F5.

I thought this was hyperbole when I was still a junior, but at my first gig acting as the primary network engineer I spent upwards of 80% of my time proving that <issue> wasn't the network. Sometimes the issue really was the network though, and it's such a transparent thing to Systems/DBA that I didn't really mind time spent investigating issues that were incorrectly configured systems or systems people not understanding how subnetting works.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Reiz posted:

I thought this was hyperbole when I was still a junior, but at my first gig acting as the primary network engineer I spent upwards of 80% of my time proving that <issue> wasn't the network. Sometimes the issue really was the network though, and it's such a transparent thing to Systems/DBA that I didn't really mind time spent investigating issues that were incorrectly configured systems or systems people not understanding how subnetting works.
At one of my prior engineering roles, I spent a lot of time having to prove that yes, issue X was the network. The network engineer would shrug off reports of problems with comments like "well, I can ping it" like a long-running string of 64-byte packets once a second must surely reveal issues with misconfigured Ethernet flow control or interface MTUs.

A lot of network admins just plain aren't good at that aspect of their jobs. It makes it hard to instill confidence in the network when the admins clearly have no idea what to even monitor to say the network looks healthy.

Virigoth
Apr 28, 2009

Corona rules everything around me
C.R.E.A.M. get the virus
In the ICU y'all......



AreWeDrunkYet posted:

If you're past the junior level, the degree won't really matter for any more technical roles. You could probably be en enterprise architect with a high school diploma.

If your ambitions are in management though, you will want a degree. It's more of a check box than anything (especially for an internal promotion), but the reputation of the school can make a difference if you're interviewing with people who don't know you.

This. I just got blocked going into management by not having a bachelors degree but I'm a senior software engineer.

Collateral Damage
Jun 13, 2009

Farts posted:

And then the network guys blame the software vendor.
Who will blame the database and complete the circle.

ZteleME
Dec 27, 2009
Looks like more schooling for me. Thx for the help guys!

WGU and SNHU are really attractive since I can use my 2 certs to knock off some classes.

H110Hawk
Dec 28, 2006
We recently had a database issue which was the network. :stonklol:

The vendor insisted we run their database on RHEL6 approved kernels only, and updated their "hardware support list" to not include the NICs which used to be on there, and which we happened to own. After a lot of gnashing of teeth we found out it really was the network. The newest mainline kernel fixed or improved a mountain of issues we were seeing, but no way no how that will void your contract.

Upgraded the NICs and it solved the issue. One down, 98 bugs on the wall... (Seriously it remained a pile of poo poo.)

22 Eargesplitten
Oct 10, 2010



Is there any way to get an alert when a new ticket comes in through zendesk? We haven't gotten any tickets since 3pm yesterday.

YOLOsubmarine
Oct 19, 2004

When asked which Pokemon he evolved into, Kamara pauses.

"Motherfucking, what's that big dragon shit? That orange motherfucker. Charizard."

Vulture Culture posted:

At one of my prior engineering roles, I spent a lot of time having to prove that yes, issue X was the network. The network engineer would shrug off reports of problems with comments like "well, I can ping it" like a long-running string of 64-byte packets once a second must surely reveal issues with misconfigured Ethernet flow control or interface MTUs.

A lot of network admins just plain aren't good at that aspect of their jobs. It makes it hard to instill confidence in the network when the admins clearly have no idea what to even monitor to say the network looks healthy.

This, but instead the response was "no errors on the port, must not be the network."

12 rats tied together
Sep 7, 2006

Vulture Culture posted:

At one of my prior engineering roles, I spent a lot of time having to prove that yes, issue X was the network. The network engineer would shrug off reports of problems with comments like "well, I can ping it" like a long-running string of 64-byte packets once a second must surely reveal issues with misconfigured Ethernet flow control or interface MTUs.

A lot of network admins just plain aren't good at that aspect of their jobs. It makes it hard to instill confidence in the network when the admins clearly have no idea what to even monitor to say the network looks healthy.

I can definitely attest to this. In said role my coworkers were surprised at how quickly I was able to pinpoint a problem in the network or able to come up with better confirmation that it isn't the network than "I can ping it". My predecessor(s) left me a shipping box full of 3 ring binders overflowing with various revisions of switch/router configurations in no particular order scrawled all over with pen -- this isn't the first time I've seen such a box either. I think "I can ping it" and "Let me check my encyclopedic collection of printed configs" are strongly correlated, and the first thing I did was get version control on the configs and, you know, "ack vlan 27" instead of flipping through binders for half an hour to find out if something changed.

My coworkers had absolutely no confidence that anything could be said to not be the network, and suspected that the network was involved in some portion of all problems because the prior response had been "I dunno, I'll call the vendor and get back to you in two weeks", so yeah I had absolutely no problem not even considering saying "it's not the network" until I had packet captures of both endpoints and the transitions at any intermediary layer 3 devices.

e: This role is also where it really hit home that "Any junior engineer can implement a feature, you hire senior engineers to tell you which features not to implement" is 100% correct. They managed to cajole one of the previous 7 network admins into implementing some really lovely form of asymmetric routing completely on static routes, but only through/out certain devices. It was also required of me to basically recreate internal/external dns zones except without using DNS because they didn't want to maintain the two zones: They needed a server farm to basically be able to make API requests against itself, through an external domain name, which needed to be accessed out of a specific static route on the boxes themselves, and then natted, policy-routed, and then finally leastconn load balanced back to the source.

12 rats tied together fucked around with this message at 18:23 on Jan 1, 2016

psydude
Apr 1, 2008

I have so many customers who use ICMP as a crutch even though it's almost meaningless in large, complex networks. It's hard to get them to realize that it's not an authoritative measurement of availability.

F5s are far and away the most headache inducing piece of equipment, though. The funny thing is that it has nothing to do with the device and everything to do with the organization: in non-DevOps shops, the developers and infrastructure team constantly punt issues to one another every time something breaks because nobody wants to take responsibility for the box. DevOps organizations are the only ones that make them work without mountains of stupidity.

psydude fucked around with this message at 19:18 on Jan 1, 2016

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

psydude posted:

F5s are far and away the most headache inducing piece of equipment, though. The funny thing is that it has nothing to do with the device and everything to do with the organization: in non-DevOps shops, the developers and infrastructure team constantly punt issues to one another every time something breaks because nobody wants to take responsibility for the box. DevOps organizations are the only ones that make them work without mountains of stupidity.

We had an F5 load balancer installed in place of our CISCO device that did round robin, and the F5 kept screwing up the load balancing on our Resin cluster, the F5 team insisted it was not due to their appliance, despite the fact that we could solve the issue in QA via a round-robin only load balance.

Finally got them to try it, and ta-da, issue disappeared.

Proud Christian Mom
Dec 20, 2006
READING COMPREHENSION IS HARD
if i have to deal with a CenturyLink tech one more loving time that insists pings are fine therefore everything is fine! i'll probably end up on a no-fly list

DigitalMocking
Jun 8, 2010

Wine is constant proof that God loves us and loves to see us happy.
Benjamin Franklin

ZteleME posted:

Was wondering if I truly need a bachelor's degree to advance in the IT field? I'm already working in IT for awhile now, but only have an associates.

And does it matter what school I go to. I mean I've been researching schools and noticed Western Governor's University and Southern New Hampshire University had some really great tuition. I can go to a local university (Boston area) but they're quite expensive. Thoughts?

Degrees help with two things:

1) getting your foot in the door with certain companies.
2) being promoted to management.

For the most part, beyond that, degrees are more useless in the IT fields than most other professional fields.

DigitalMocking
Jun 8, 2010

Wine is constant proof that God loves us and loves to see us happy.
Benjamin Franklin

Vulture Culture posted:

At one of my prior engineering roles, I spent a lot of time having to prove that yes, issue X was the network. The network engineer would shrug off reports of problems with comments like "well, I can ping it" like a long-running string of 64-byte packets once a second must surely reveal issues with misconfigured Ethernet flow control or interface MTUs.

A lot of network admins just plain aren't good at that aspect of their jobs. It makes it hard to instill confidence in the network when the admins clearly have no idea what to even monitor to say the network looks healthy.

I spend a lot of time proving that issue X isn't the network. The server/application/database administrator would shrug off reports of problems with comments like "well, I can reach the service" or "I can see the database structure in Manager" like opening an application in a web browser once must surely reveal issues with misconfigured services or databases that fail under load.

A lot of application/server administrators just plain aren't good at that aspect of their jobs. it makes it hard to instill confidence in the infrastructure when the admins clearly have no idea what to even monitor to say the services are healthy. :colbert:

PBS
Sep 21, 2015

DigitalMocking posted:

I spend a lot of time proving that issue X isn't the network. The server/application/database administrator would shrug off reports of problems with comments like "well, I can reach the service" or "I can see the database structure in Manager" like opening an application in a web browser once must surely reveal issues with misconfigured services or databases that fail under load.

A lot of application/server administrators just plain aren't good at that aspect of their jobs. it makes it hard to instill confidence in the infrastructure when the admins clearly have no idea what to even monitor to say the services are healthy. :colbert:

We can probably agree that a lot of people in IT aren't good at their jobs.

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online
I'm on the unix team at work. We take care of the F5s for some reason that only management can decipher. Recently some firewalls were life-cycled in our secondary datacenter. This caused a strange problem where traffic to nodes in our primary datacenter from the secondary data center's f5s is being intermittently dropped. The other way around is fine... Network security refuses to investigate the issue as there's "no possible way it could be the firewalls." Networking won't touch it either (yes networking is separate from network security). Meanwhile I get to deal with app owners whose apps are experiencing bizarre intermittent issues. I have logs that clearly show the problem did not exist with the old firewalls and as soon as they put the new firewalls in place, blammo. Security still won't do anything but blame everyone else. It's maddening.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Goon Matchmaker posted:

I'm on the unix team at work. We take care of the F5s for some reason that only management can decipher. Recently some firewalls were life-cycled in our secondary datacenter. This caused a strange problem where traffic to nodes in our primary datacenter from the secondary data center's f5s is being intermittently dropped. The other way around is fine... Network security refuses to investigate the issue as there's "no possible way it could be the firewalls." Networking won't touch it either (yes networking is separate from network security). Meanwhile I get to deal with app owners whose apps are experiencing bizarre intermittent issues. I have logs that clearly show the problem did not exist with the old firewalls and as soon as they put the new firewalls in place, blammo. Security still won't do anything but blame everyone else. It's maddening.

There was an issue they were tracking where F5s would randomly accept traffic and then never forward said traffic to the endpoint.

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online

CommieGIR posted:

There was an issue they were tracking where F5s would randomly accept traffic and then never forward said traffic to the endpoint.

F5? I don't think that's in play here. 2nd DC to 1st DC = dropped traffic. 1st DC to 2nd DC = Fine. It started immediately after they flipped everything over to the new firewalls. I think the F5's are triggering some kind of port scan protection on the new firewalls but I'm not sure.

psydude
Apr 1, 2008

Goon Matchmaker posted:

(yes networking is separate from network security).
This is a pretty common separation of powers in government and regulated industries (health and finance). It ensures that the network team doesn't circumvent security for the sake of making connectivity easier and also makes sure that an attacker can't get into the network and security equipment in the event a single account is compromised.

e: But it still adds one more level of rear end pain to any deployment.

psydude fucked around with this message at 22:37 on Jan 1, 2016

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Goon Matchmaker posted:

F5? I don't think that's in play here. 2nd DC to 1st DC = dropped traffic. 1st DC to 2nd DC = Fine. It started immediately after they flipped everything over to the new firewalls. I think the F5's are triggering some kind of port scan protection on the new firewalls but I'm not sure.

No, no, that is exactly what happens. DC on one side can communicate properly, but anything coming BACK through the F5, the traffic gets accepted but its fails to pass it onto the internal network.

There may be a configuration that was tied to the MAC/IP tagging for the long gone firewalls, and you may have to rebuild the F5 configuration to resolve this issue. If you have vetted your Firewall rules, I'd look further at the F5 itself.

https://support.f5.com/kb/en-us/solutions/public/12000/700/sol12703.html

CommieGIR fucked around with this message at 22:49 on Jan 1, 2016

abigserve
Sep 13, 2009

this is a better avatar than what I had before

DigitalMocking posted:

I spend a lot of time proving that issue X isn't the network. The server/application/database administrator would shrug off reports of problems with comments like "well, I can reach the service" or "I can see the database structure in Manager" like opening an application in a web browser once must surely reveal issues with misconfigured services or databases that fail under load.

A lot of application/server administrators just plain aren't good at that aspect of their jobs. it makes it hard to instill confidence in the infrastructure when the admins clearly have no idea what to even monitor to say the services are healthy. :colbert:

In my experience the network is always the first to be blamed because it guarantees at least one person will seriously investigate the problem. "the network has problems" is the sysadmin's way of saying "well everything else is probably broken as well so don't look at me"

Chickenwalker
Apr 21, 2011

by FactsAreUseless
fart

Chickenwalker fucked around with this message at 05:28 on Sep 23, 2018

Goon Matchmaker
Oct 23, 2003

I play too much EVE-Online

CommieGIR posted:

No, no, that is exactly what happens. DC on one side can communicate properly, but anything coming BACK through the F5, the traffic gets accepted but its fails to pass it onto the internal network.

There may be a configuration that was tied to the MAC/IP tagging for the long gone firewalls, and you may have to rebuild the F5 configuration to resolve this issue. If you have vetted your Firewall rules, I'd look further at the F5 itself.

https://support.f5.com/kb/en-us/solutions/public/12000/700/sol12703.html

Given we're on 11.5.3 I don't think that applies to us.

DigitalMocking
Jun 8, 2010

Wine is constant proof that God loves us and loves to see us happy.
Benjamin Franklin

abigserve posted:

In my experience the network is always the first to be blamed because it guarantees at least one person will seriously investigate the problem. "the network has problems" is the sysadmin's way of saying "well everything else is probably broken as well so don't look at me"

I don't disagree, just after 2 decades of hearing "it must be the network" I'm ready to stab a bitch some days.

CommieGIR
Aug 22, 2006

The blue glow is a feature, not a bug


Pillbug

Goon Matchmaker posted:

Given we're on 11.5.3 I don't think that applies to us.

:shrug: Don't be afraid to look into the F5 more anyways.

H110Hawk
Dec 28, 2006

Chickenwalker posted:

Speaking of this, we just got a dedicated 1 Gbps circuit for downloading footage from the field and wouldn't you know we're not actually getting 1 gig in either direction, even hooked directly up to the lovely ONT they provided.

So if I'm going to have to actually get some big boy enterprise network gear now, what's my best bet?

I'm a pretty big fan of Juniper's MX platform.

Adbot
ADBOT LOVES YOU

adorai
Nov 2, 2002

10/27/04 Never forget
Grimey Drawer

Chickenwalker posted:

So, I've been running Ubiquiti's highest-end router offering for a while, and I find out that maybe despite all the marketing hoopla they do that it can't handle more than 450 Mbps without hardware offloading enabled, and hardware offloading is only incompatible with, hmm, let's see: load-balancing/failover WAN links, policy based routing, and oh yeah this little thing called VLANs, just to name a few. Super.
If you are talking about their edgerouter line, it supports vlans in hardware offload. Has since like 1.2.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply