|
To be fair, it is usually DNS. Can't decide if terrible or best page snipe.
|
#
?
Jul 13, 2018 22:29
|
|
|
|
| # ? Jun 8, 2024 02:55 |
|
|
Always blaming DNS is a serious lack of creativity of how things can go wrong.
|
|
#
?
Jul 13, 2018 22:40
|
|
|
I've seen more issues with bad SSL/TLS certs than with DNS here.
|
|
#
?
Jul 13, 2018 22:43
|
|
|
Methanar posted:Always blaming DNS is a serious lack of creativity of how things can go wrong. http://www.bofhcalendar.com/
|
|
#
?
Jul 13, 2018 22:44
|
|
|
Well poo poo, this is my first corporate network where everything has a DNS hostname (I'm used to lists and lists of IP's) and it all works relatively well. DNS issues are pretty rare, which is weird considering we are a garbage network managed by garbage people. guppy posted:We have this problem too. It's even worse when wireless enters the picture. Just try convincing other groups that the wireless network itself isn't the cause of their wireless connectivity problem. AT&T handles our remote location switches and also handles the wireless AP's and an AT&T agent called and asked what was going wrong with their AP connected to their switch. I just enjoyed the minute or so of silence that followed.
|
|
#
?
Jul 13, 2018 22:50
|
|
|
"it's always dns" comes from the lack of effective and ordered troubleshooting. It is the first thing I check for any weird issue because it takes 15 seconds to check forward and reverse lookups for most things. If, on multiple occasions, I spent an hour on other troubleshooting, only to find out I would have known in 15 seconds, I would get frustrated and say poo poo like that myself. I work in finance and lately, our big excitement has been from applications which were statically compiled to use TLS 1.0. PCI-DSS v3.2 (finally) went into effect july 1st so any card data must be transmitted with 1.2 or higher, and statically compiled or configured TLS 1.0 has broken a number of things. So now, that's a good place to start.
|
|
#
?
Jul 13, 2018 22:50
|
|
|
dns is pretty easy to rule out. just use dig and see whats up
|
|
#
?
Jul 13, 2018 22:52
|
|
|
Contingency posted:^^This is what server admins actually believe. So glad I have whisky handy after reading this post. Also, I picked the hottest day I could to change brake pads with my dad.
|
|
#
?
Jul 13, 2018 23:02
|
|
|
It’s always network or whatever is just a symptom of how many IT organizations function where responsibilities are chunked up into discreet little domains and people generally don’t have much visibility, and thus much understanding, beyond their own little sphere.
|
|
#
?
Jul 13, 2018 23:20
|
|
|
Speaking of DNS, its incredible how out of the box dns/dhcp configuration isn't great for a workplace with laptops with docking stations. For example, if users undock their laptops and use wireless and back again, dynamic dns is flaky as gently caress. You can remedy this by changing the default dns/dhcp server configuration in not-so-straight forward fashion to make it a lot more bearable, but it is silly. It makes agent-less monitoring like pdq inventory a pain in the rear end if you dont know what you are doing.
|
|
#
?
Jul 13, 2018 23:50
|
|
|
I've actually been looking at running PDQ inventory and PDQ deploy. Anyone use these and have an opinion?
|
|
#
?
Jul 14, 2018 00:22
|
|
|
I have the enterprise subscription of both and my opinion is that they are great.
|
|
#
?
Jul 14, 2018 00:23
|
|
|
I have always enjoyed Cisco's 'solar bit flip' excuse when they can't find root cause. I didn't even know it was a thing until a co-worker told me about Cisco going to that well multiple times with his TAC cases. Now to me, every unexplainable problem is 'solar bit flip'
|
|
#
?
Jul 14, 2018 02:48
|
|
|
Counterpoint: the frequency and variety of ways in which people gently caress up, misunderstand, and misuse DNS despite it being critical to everything. "lmao DNS is easy, git gud" seems like an equal failure of imagination regarding ways things can go wrong. As we all grow more senior, we quickly catch the dumb DNS mistakes. But like, there is this issue open right now against the core Kubernetes project that we are facing. Sometimes DNS queries just take 5s to resolve instead of the usual single-digit ms. It's due to some race condition in the Linux kernel and no one has any idea how to fix it. I can't find the post (or maybe it was in a book?) cause it's Friday and beer exists, but there was some postmortem of a big Google outage a few years back. The root cause was automation without sufficient error checking that allowed a bad config push to delete a shitload of important records from DNS. I assume Vulture Culture either has the citation handy, or can prove I imagined this  Everyone messes up DNS all of the time, even the big girls and boys.It's not always DNS. But we should all have "did you check DNS?" carved into on our forearms. Vargatron posted:Despite having a cool exterior during critical downtime events, I'm actually panicking internally in such a way that I imagine the room and building are on fire. I can't recommend reproduction as a way to reduce stress. But I feel like after having kids, absolutely no outage situation at work can faze me anymore. I spend most of my time outside of work just trying to make sure my offspring don't literally kill themselves. Oh, the onsite Dell tech accidentally ripped out the entire drive tray from the MSSQL server instead of hot swapping the one disk he was sent to fix? This is fine. We can handle this. That insight about unsettling people by not panicking enough is interesting, though. Sounds like a good read.
|
|
#
?
Jul 14, 2018 03:54
|
|
|
Docjowles posted:Counterpoint: the frequency and variety of ways in which people gently caress up, misunderstand, and misuse DNS despite it being critical to everything. "lmao DNS is easy, git gud" seems like an equal failure of imagination regarding ways things can go wrong. As we all grow more senior, we quickly catch the dumb DNS mistakes. But like, there is this issue open right now against the core Kubernetes project that we are facing. Sometimes DNS queries just take 5s to resolve instead of the usual single-digit ms. It's due to some race condition in the Linux kernel and no one has any idea how to fix it. https://blog.quentin-machu.fr/2018/06/24/5-15s-dns-lookups-on-kubernetes/ Jesus christ
|
|
#
?
Jul 14, 2018 04:34
|
|
|
Docjowles posted:It's not always DNS. But we should all have "did you check DNS?" carved into on our forearms. This should always be one of those fundamental checks like “can I ping it?” Or “is the cable *physically* plugged in?”
|
|
#
?
Jul 14, 2018 05:16
|
|
|
In my interview today they commented on the fact that pinging was the first trouble shooting step i gave for almost everything, and the fact that it was rare to see
|
|
#
?
Jul 14, 2018 05:22
|
|
|
I find more people relying incorrectly on traceroutes than pings. Like no one ever pings, but they'll send horribly formatted tracert's all day long. Cisco, of all people tried to tell me that two different traceroutes to the same server, one with an extra hop, was responsible for consistent 20% packet loss on their video conference calls. Note, there was no latency. It was literally just an extra hop that was correctly configured in BGP between two sites in China. I'm pretty sure Cisco Telepresence is the devil and I'm in their hell.
|
|
#
?
Jul 14, 2018 05:37
|
|
|
wtf, ping and nslookup are step 1 and step 2 if anyone claims anything is "down"
|
|
#
?
Jul 14, 2018 05:37
|
|
|
Making a second post for my bad management takes since it has nothing to do with DNS (...or does it  ). I actually have a lot more feels about this than I expected to. To sum up:One on ones are not for status updates. They are to build relationships, talk short and long term career growth, address poor performance or reinforce great performance outside of the annual review (poo poo or nonexistent 1:1 and review setups go hand in hand). A good manager should tailor the meeting to the employee. If they are a big introvert (not exactly rare in our field), leading with "so wazzup?" and kicking your feet up to listen might not be the best tactic. At least until they are comfortable with you. I do like that as the default setup, though, if I can aggressively paraphrase AlternateAccount. Seniority doesn't matter. A junior engineer has very specific questions about how they can solve a problem or move from Level 1 to Level 2. A senior engineer probably also wants to move up (even if they can't in your org), but how will you ever find that out if you don't talk to them regularly? Do you want to be totally blindsided when your best talent quits? You can pick their brain and find out what projects the team might work on to keep them engaged, and connect them with the right people to make it happen. If one or both parties finds the meeting useless, talk about why, and fix it. I had a manager that talked a good game about 1:1s. Then he would cancel them 10 minutes beforehand every single week, "because something came up". gently caress you! I prepared for these meetings with topics to discuss. Of course things come up, but every week for a year? Thanks, you've shown me all I need to see about your priorities. He eventually left, tried to poach our whole team to his new and extremely choice gig, and every single person said "nah". It was amazing. Docjowles fucked around with this message at 06:07 on Jul 14, 2018 |
|
#
?
Jul 14, 2018 06:03
|
|
|
Proteus Jones posted:This should always be one of those fundamental checks like “can I ping it?” Or “is the cable *physically* plugged in?” Llaneza's Law of Network Engineering: Check the cables first. The Fool posted:wtf, ping and nslookup are step 1 and step 2 if anyone claims anything is "down" Those count under this Law, and also make sure the thingie is actually turned on. So we're finally upgrading the network to gasp ! 100MB hubs. Friday night we pull the old hubs out, rack the new ones, and re-punch everything for good measure. I say, "hey let's check some of the production workstations." IT Manager says "nah". Saturday morning comes. We turn everything on as if it was a weekday where we try and make money. An aisle of 8 workstations can see each other but not the rest of the network. We futz with the workstations for a few minutes, and then the boss says to leave it alone, he's calling in a buddy of his who's a top flight networking consultant. I am given a direct order to stay out of the server room where all the punchdown panels, new hubs, and what not are. We wait an hour and buddy shows up. He's making $200/hr. In 1998. He heads into the server room to fix our problem. An hour and half (and $300) later he comes out doing this: ¯\_(ツ)_/¯ He's explaining the to the boss. Besides those two, four of us are sitting around with the NFL playoffs on the loving radio because it's 1998; and Garrison Hearst has just suffered an injury that will knock the 49ers out of the playoffs and almost kill him. I duck behind the expensive consultant making excuses and head into the server room. I duck behind the rack and head for the hub the afflicted workstations are connected. I grab the uplink cable for that hub and push it home. *click* I walk out, check a couple of workstations, and announce that we're 100% online on the new network. I was on a lovely salary. The consultant made hourly. Since I hadn't yet heard the gospel that is the California Labor Code, I fixed that poo poo for free, and a motherfucker who didn't think to check Layer 1, collected two days worth of the OT that I should have had, and for not fixing a drat thing. Check. The. loving. Cables. Guillotine all executives
|
|
#
?
Jul 14, 2018 06:40
|
|
|
Docjowles posted:Counterpoint: the frequency and variety of ways in which people gently caress up, misunderstand, and misuse DNS despite it being critical to everything. "lmao DNS is easy, git gud" seems like an equal failure of imagination regarding ways things can go wrong. As we all grow more senior, we quickly catch the dumb DNS mistakes. But like, there is this issue open right now against the core Kubernetes project that we are facing. Sometimes DNS queries just take 5s to resolve instead of the usual single-digit ms. It's due to some race condition in the Linux kernel and no one has any idea how to fix it. I have that same issue, but for different reasons. Turns out if you configure resolv.conf to send both the v4 and v6 resolution requests using the same socket, and your border firewall eats the v6 response, then the kernel just gives up and waits until the timeout is reached before handing over the v4 address. Hilariously, everyone was blaming memcache and believed it was the cause of the app going to poo poo. Nope, it's DNS.
|
|
#
?
Jul 14, 2018 07:28
|
|
|
^^^Great story there. It's *always the obvious things that are missed* Decided to go back to my old job. The stories of unemployment are scaring me here, and the plan of being unemployed and looking for job a until I ran out of savings would be a gamble. I've gambled enough. I did that leaving my old job for that risky position. I was smart with my last job and did not burn bridges and they are ok with me coming back, so the plan is in the next two weeks. I said I'd at least stay another year if did return but no promises were made yet For the current job, I'm just going to go through the motions and do just enough work so I'll keep getting paid and not fired immediately. It was only a contact. My old job was FTE which I'm very glad to go back to. I thought I was making more money with the current contact job but every half-day and every holiday was adding up. Life without PTO and sick days is bad as well. I think I made the right choice. I'm 100 percent getting fired in the next 30 days from my current job. Going back to the old job may be a stepbackwards but it pays the bills and they like me and want me there at least so it's safe. Also for resume purposes I never left my old job so it will just look continuous. There is a new option I can now theoretically put into play: resign my current job Monday morning and start searching like hell for a new position within the next two weeks before I'm rehired to my old job. It's unethical though because I already told the manager there I'd return. Nonetheless I'm just putting it out there. DropsySufferer fucked around with this message at 07:45 on Jul 14, 2018 |
|
#
?
Jul 14, 2018 07:33
|
|
|
I personally wouldn't, but in your situation I think it comes down to "if the money is substantially better". You sound like you deserve a few days off anyways.
|
|
#
?
Jul 14, 2018 12:25
|
|
|
Docjowles posted:I had a manager that talked a good game about 1:1s. Then he would cancel them 10 minutes beforehand every single week, "because something came up". gently caress you! I prepared for these meetings with topics to discuss. Of course things come up, but every week for a year? Thanks, you've shown me all I need to see about your priorities. He eventually left, tried to poach our whole team to his new and extremely choice gig, and every single person said "nah". It was amazing.
|
|
#
?
Jul 14, 2018 14:27
|
|
|
Matt Zerella posted:Is "senior engineer" a decent spot to be in at a MSP? I landed a second interview at a shop on Monday. They seem ok and I'd get to touch a lot of stuff. They seem open to suggestions and listen to their engineers (or at least told me so). It depends a lot on the company. Some are more sales driven, lots of client turnover, few standards, 24 7 on call, using technology wrong. Some are great places to work with reasonable on call rotation, good customers, budget and time for training. You hear more about the first group. DropsySufferer posted:I need to get another job within 30 days. At larger companies 6-8 weeks between applying and getting hired, plus another week or two to get your first paycheck is not unusual. This applies more to mid career roles or higher, entry level roles should be faster. There are a lot of delays for scheduling phone interviews, background checks, drug test, screening interviews, in person interviews. Many of them are getting 1000+ applicants per open position and it can take a while before they get to your resume. You could apply with a relevant resume to 10 of the top 100 companies and never hear back if they hire someone else or otherwise never get to your resume. Federal jobs can take 6 months or longer. Look for people you know who can reccomend you for a job. You can cut through a lot of delays.
|
|
#
?
Jul 14, 2018 17:39
|
|
|
DropsySufferer posted:
Lookout for your self. Don't resign Monday morning. Continue looking for a better job during the next 30 days. If nothing is promising, go back to the old job.
|
|
#
?
Jul 14, 2018 17:44
|
|
|
https://circleci.com/blog/its-the-future/ https://circleci.com/blog/it-really-is-the-future/ Mandatory reading Methanar fucked around with this message at 05:40 on Jul 15, 2018 |
|
#
?
Jul 15, 2018 05:24
|
|
|
My head hurts.
|
|
#
?
Jul 15, 2018 05:30
|
|
|
Methanar posted:https://circleci.com/blog/its-the-future/ ...But who's on first?
|
|
#
?
Jul 15, 2018 05:38
|
|
|
quote:The software industry, contrary to what you might expect, is absolutely filled with people who hate progress. The sort of people who would walk into the Sistine Chapel after Michelangelo was done and declare that they already had a perfectly good picture of god, they prefer their ceilings to be white, and that frescos aren’t that cool anyway.
|
|
#
?
Jul 15, 2018 05:48
|
|
|
Finally,  !My letter of resignation and 2 weeks notice is ready and printed. I got an email and verbal offer on Friday, which I accepted. I then got asked if I wanted a MacBook or a Dell, did an online background check consent thing (Sterling)... was told on Monday I'd get the full formal offer with the CEO's signature, since I had done the verbal acceptance. Question is: do I wait for the formal offer letter, or am I safe to put in my 2 weeks notice tomorrow? I'm thinking shortly after lunchtime....
|
|
#
?
Jul 16, 2018 05:39
|
|
|
Always always always wait. Any hundred things could come up Monday morning. You'll lose nothing by waiting until you have a signed offer letter in hand. And congrats! Enjoy what I hope is a new pod!
|
|
#
?
Jul 16, 2018 05:40
|
|
|
Yea wait for the signed offer after negotiations. It’ll be another week but it’s so worth it just Incase something blows up the deal.
|
|
#
?
Jul 16, 2018 05:45
|
|
|
Also on the "i'd wait" train. What if the CEO wakes up covered in hookers and glitter before saying "poo poo, i can't afford another employee, i'd be able to do this one less time a year".
|
|
#
?
Jul 16, 2018 09:23
|
|
|
just came back from a 4 week vacation and the 1st thing that happens on my 1st workday is that i'm getting a raise. This is a good start.
|
|
#
?
Jul 16, 2018 10:11
|
|
|
Sefal posted:just came back from a 4 week vacation and the 1st thing that happens on my 1st workday is that i'm getting a raise. Just back from 2 weeks (all I get, gently caress this place) and lol I wish I was getting anything other than poo poo on.
|
|
#
?
Jul 16, 2018 12:46
|
|
|
 If its any consolation. Reality is starting to hit me that I only have 1 more PTO day left for the next 6 months. using 1 week of PTO in January was perhaps a mistake.. And the next days that the company is closed is during Christmas
|
|
#
?
Jul 16, 2018 12:59
|
|
|
Sefal posted:
My boss used to be head of HR so she is anal about documenting every hour of PTO, even if it’s like leaving at 3 o clock on a Friday to collect my child. I have 9 hours of pto left until dec 31st because I used the majority of it when my daughter was born in April. Hooray for first world countries with no paid family leave.
|
|
#
?
Jul 16, 2018 13:06
|
|
|
|
| # ? Jun 8, 2024 02:55 |
|
|
Methanar posted:https://circleci.com/blog/its-the-future/ just loving mainline this poo poo right into my veins.
|
|
#
?
Jul 16, 2018 13:08
|
|