Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
devmd01
Mar 7, 2006

Elektronik
Supersonik

Farking Bastage posted:

I can tell you that the expensive firewalls have quirks of their own. 15 minutes before quitting time today, something went wonky with our *VERY* expensive Checkpoint 15000 HA pair. We pushed an application rule to block snapchat and something glitched. The primary poo poo itself completely, but not in such a way that the HA backup would sense it and take over. Forced a failover to the secondary, but it still didn't want to work right. We ended up having to revert the config on the primary, fail it back over then revert the secondary. I'll be on the phone with someone in Israel a lot tomorrow looks like :(

That sounds about right for Checkpoint!

Adbot
ADBOT LOVES YOU

devmd01
Mar 7, 2006

Elektronik
Supersonik

Dick Trauma posted:

gently caress it. <approves all patches in WSUS>

This but unironically for all servers. Desktop team uses manage engine.

devmd01
Mar 7, 2006

Elektronik
Supersonik

minusX posted:

Speaking of, we're moving to ServiceNow. Anyone have a good resource to learn it?

drat near our entire business runs on it, from IT and HR to all of the field service technicians. Thankfully all I have to do is deal with change/incident, but at least half of our developers are servicenow focused.

The UI is a mess and nothing makes any god drat sense, so I’ve learned how to do the bullshit I need to do and that’s it.

devmd01
Mar 7, 2006

Elektronik
Supersonik

The Fool posted:

I haven't stayed in the office past 5:30 in 2 years, I've earned this poo poo.

Seriously. 4pm rolls around and I’m out the door unless there’s an ongoing emergency. Any after hours maintenance that could have an impact, I schedule for when it’s convenient for me.

devmd01
Mar 7, 2006

Elektronik
Supersonik
A couple of days ago the CIO was in the next row over, discussing an issue with the operations people where office applications are crashing randomly, probably due to last months lovely patches.

Felt so good to know that dealing with it was definitely Not My Problem, I did my time with desktop management.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Renegret posted:

I mean, teachers have lost their jobs before for taking pictures with alcohol so....who knows.

This is why when my wife and her teacher friends are planning drinks after school, it’s “poetry club,” because their email can be FOIA’ed.

devmd01
Mar 7, 2006

Elektronik
Supersonik
A ticket came in yesterday afternoon:

2:35PM: ldap server for our ServiceNow instance is down, nobody can log in. Start investigating.

2:45PM: isolated the issue to an edge router at our main datacenter, it affects everything, have service desk start drafting an outage notification

2:50PM: get service interruption notice from our data center

3:16PM: services restored, get all clear from data center, the team fucks off to the microbrewery down the street

devmd01 fucked around with this message at 15:01 on Apr 21, 2018

devmd01
Mar 7, 2006

Elektronik
Supersonik
My parents were using a wrt54gl with a custom firmware for years, I could ssh in through it to fix issues.

devmd01
Mar 7, 2006

Elektronik
Supersonik
With every job I’ve switched to in my career I’ve managed to move one exit closer to my house around 465 each time. If there’s little or no traffic and I haul rear end (most of it is interstate where there’s never cops) I can make it to work in 18 minutes, which is a pretty goddam good commute for living in an Indianapolis suburb. The interstate exit basically shoots straight into the parking lot, it’s great.

devmd01
Mar 7, 2006

Elektronik
Supersonik
A ticket came in this morning: “work order notifications are not being delivered to technicians from ServiceNow.” Kind of a big deal for us since our main business application is built in servicenow, and we are a service provider type organization.

Due to an “administrative error” aka I hope someone got fired today, our data center provider somehow managed to kill the dns records for their domain. This had some rather unpleasant side effects as we host our external dns with them (in six different datacenters), however when your own domain whois records point to ns1-6.provider.com....

Oddly enough servicenow was the only application that reported issues, it queried and cached a bad response while they had dns hosed up. Time to export the zones to csv and write a little script to import them into Azure DNS!

devmd01
Mar 7, 2006

Elektronik
Supersonik

MJP posted:

Does your provider's name rhyme with "ingredient" because I'm getting "we hosed up our DNS" incident alert emails from our cloud backup service provider.

Bingo! Not for long lol, we are getting the hell out of there in the next 7-8 months. This isn’t the first “administrative error” to cause a critical service outage for sure.

devmd01
Mar 7, 2006

Elektronik
Supersonik
A ticket came in yesterday: “all of the windows 7 machines in AP bluescreened simultaneously”

:psyduck:

devmd01
Mar 7, 2006

Elektronik
Supersonik
And now is a good time to push for a revamp of your patching policies and procedures so this doesn’t happen again!

With the right GPOs, WSUS approval groups, and a clearly defined monthly patch testing and rollout schedule it’s pretty straightforward to patch every month and not blow things up, as long as you pay attention to the r/sysadmin patch Tuesday megathread, it’s saved my rear end nearly every month this year.

devmd01 fucked around with this message at 23:15 on Jul 27, 2018

devmd01
Mar 7, 2006

Elektronik
Supersonik
Here's a friendly reminder to prevent some tickets coming in if you still have Windows 7 machines and O365, don't forget O365 is going TLS 1.2 only on October 31st.

https://support.microsoft.com/en-us/help/3140245/update-to-enable-tls-1-1-and-tls-1-2-as-a-default-secure-protocols-in

Hope you have KB3140245 installed along with a GPO to push the registry key out!

devmd01
Mar 7, 2006

Elektronik
Supersonik

Jaded Burnout posted:

Keeping my personal number out of the hands of employers and clients has been a heroic struggle, but so far I’m holding out.

Google voice number yo

devmd01
Mar 7, 2006

Elektronik
Supersonik
All of the phone calls came in: I managed to break external SSO :suicide:

devmd01
Mar 7, 2006

Elektronik
Supersonik
Patched the domain controller that has the PDC emulator role and it somehow completely hosed the OS and it’s coming up in safe mode.

ADFS only authenticates to the PDC role.

Triaged for 30 minutes, said gently caress it, and seized the FSMO role.

Guess who gets to rebuild a domain controller today!

devmd01
Mar 7, 2006

Elektronik
Supersonik

The Fool posted:

I don't think this is right but do not have a reference handy.

It was hosed and booting into safe mode, even trying to uninstall the patch would fail and try to revert. Since I had to seize the role to get ADFS working I wasn't gonna bring that server back in without flattening it and cleaning up ntdsutil, srv records, etc. Honestly easier than trying to figure out why the OS was fubar and fixing it.

When you have extranet lockout enabled it requires the PDC: https://blogs.technet.microsoft.com/pie/2015/10/11/adfs-extranet-lockout-and-pdc-requirement/

devmd01
Mar 7, 2006

Elektronik
Supersonik

Thom and the Heads posted:

that lady probably goes home and doesn't think about work until the next time she goes in. tbh i identify with people like her more than i identify with the people that get off work and spend their time off studying for certs

I don't think about work from the moment I leave until I get back, unless there is an outage or a 15-30 min planned change (that I get to pick the time after the kids are in bed). Once I leave work the hardest part of my day begins, with is being a good parent and husband and handyman. (new house, lots of projects to do)

Granted, i'm a glorified sysadmin, but I can't complain about my compensation and benefits at this company. They take care of their employees and understand there is life outside of work. If I have a good need I can WFH/come in late and it doesn't hit my 5 weeks of PTO. They invest good money in IT and people. We just doubled the size of the company with an acquisition and are doing integration/cut over planning to bring all their employees on board our systems.

devmd01
Mar 7, 2006

Elektronik
Supersonik
If they’re on the right version of IOS you can generate a QR code that the camera app will read and create the WiFi profile.

E;f;b

devmd01
Mar 7, 2006

Elektronik
Supersonik
A ticket came in to Verizon, it looks like texting/mms is down for a good chunk of the US. Hate to be the one fixing that issue!

devmd01
Mar 7, 2006

Elektronik
Supersonik

kensei posted:

User submits ticket that mail from a domain is blocked by the anti-spam solution despite his whitelisting it and needs to be fixed ASAP
Log in as user to anti-spam solution
User has blocked domain
Screenshot
Advise user of error
Fix
Fin

An email came in from the security team, a toss over the fence forward of a logrhythm alert for too many failed logins for some random account getting locked out: “hey guys can you look into this?”

*does an nslookup on the source IP*

Reply-all: “dude, it’s your own nexpose scanning tool doing the login”

*crickets*

Our security team is worthless, especially the manager.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Schadenboner posted:

The mistake was in accepting mail from Russia, hth?

Or any IP traffic at all. We literally block the world on our edge firewalls except for the US and the Cayman Islands. Exec traveling outside the US for some reason? Better put in a ticket with what country and the dates and we will allow it only for that period.

Between making sure O365 is using modern auth across the board and disabling pop/imap, this has greatly reduced incidences of any successful phishing attacks causing further damage. If they can’t get to our ADFS to authenticate in to O365, then they can’t get in period. Yes, this doesn’t help if they are proxied through a US host, but it’s pretty effective otherwise.

devmd01
Mar 7, 2006

Elektronik
Supersonik

stevewm posted:

Its run by a large software company some here may know... Epicor.

In a past life we had Prophet21 on-prem, this doesn’t surprise me in the least about Epicor. What a garbage piece of software and company.

Are there any market niche software apps for specific industry verticals that aren’t absolute garbage?

devmd01
Mar 7, 2006

Elektronik
Supersonik
....if you’re using a third party email filter provider why isn’t the tenant locked down to only allow email coming from the filtering provider?

The only way in or out of our tenant (or on-prem smtp relay) is through proofpoint.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Yep, we have the transport rules locked down to only allow email inbound from proofpoint, works like a charm.

devmd01
Mar 7, 2006

Elektronik
Supersonik
We are going all in on telephony with Teams, if you don’t have a reason/need to be in a call queue (customer call center, service desk, various internal call queues for our techs) that’s what you’ll get. Anyone in the call queues will get an actual desk phone tied in with a cloud SIP provider, with separate internet trunks/failover in our two data centers.

devmd01
Mar 7, 2006

Elektronik
Supersonik
The worst part about that virus is gaining a sexual attraction to purple monkeys.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Wizard of the Deep posted:

I quickly hooked up the computer, collected payment, and left. I did not get a tip.

But you could have!

devmd01
Mar 7, 2006

Elektronik
Supersonik
I set up our service desk with the proper delegated access to our employee OU, and they run-as mmc with their admin accounts to do what they need to do. After talking to some of the ignite tech people, I’m very interested in switching their processes to use the window admin center gateway, there is definitely some opportunity there to streamline their work.

devmd01
Mar 7, 2006

Elektronik
Supersonik
We’re all in on teams telephony unless you are in a call queue of some sort I.e. our call center, they’re staying on interactive/genesys. Everyone else at the corporate office is getting ported. I have no complaints, I’ve been using a teams number for months at this point. I really, really, don’t get the Teams hate. Is it perfect? No. But it works perfectly well enough for what we do, so whatever!

devmd01
Mar 7, 2006

Elektronik
Supersonik
Thankfully we have full support of the service desk manager to kick any escalated tickets back down if they didn’t do their due diligence. Half the time I’ll just link a KB article that I’ve already written for some GPO exclusion process or whatever, their admin accounts have appropriate delegation to the right groups.

devmd01
Mar 7, 2006

Elektronik
Supersonik

Dirt Road Junglist posted:

Speaking of Windows patching, anyone else's security teams having heartburn over this right now?

https://www.zdnet.com/article/microsoft-fixes-windows-crypto-bug-reported-by-the-nsa/

No because our security team is loving worthless and it starts at the top. They finally hired someone that seems to know what they’re doing but the jury’s still out until I know what he can do.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Even better, I had a SAML cutover for a SaaS app from ADFS to Okta that they wanted $5000 for. It took literally 10 minutes. Ludicrous.

devmd01
Mar 7, 2006

Elektronik
Supersonik
No WFH policy yet but 90% of our employees provide support services to hospitals so lol.

Boss is WFH today so I hosed off work at 1030 to go drink with a former coworker. :getin:

devmd01
Mar 7, 2006

Elektronik
Supersonik
what I really need is an easy way to mix in the audio from conferencecall.biz to my outgoing teams audio stream. Especially for cold callers.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Someone in the IT team sent out an invite to the team distro for a teams happy hour from 4-6 today. I couldn’t hit decline fast enough.

devmd01
Mar 7, 2006

Elektronik
Supersonik
Got dragged in to a call right after lunch for a P1 incident, 1/3 of the company had their samaccountname changed inadvertently. Our user provisioning/access person was setting up Workday import to our prod Okta tenant. Apparently when you add another provisioning source, Okta will flip attribute mappings to AD to all “create and update,” instead of on create only like we had set for samaccountname. Since 1/3 of our users had our legacy username format, well...

Thank god we had an HRIS export with the old samaccountname that we could run a script against to revert! I’m glad I’ve been pushing for us to use UPN/email for nameid in all of our SaaS apps since I’ve been here, otherwise it could have been much, much worse.

devmd01
Mar 7, 2006

Elektronik
Supersonik
So, related to the samaccountname disaster on Wednesday, they went live with the cutover of the user provisioning data source yesterday morning. It all went well so everyone’s updated title started flowing through to AD.

I get contacted at around 1:30PM, two of our applications pull data from a separate LDAP server that syncs data from AD and weren’t seeing the new titles.

While there were three overall root causes, one of the main ones was that a VP’s new full title is 68 characters long. The attribute in the LDAP schema was set for max 64 char, so it crashed the ldap sync as soon as it reached that user record.

Adbot
ADBOT LOVES YOU

devmd01
Mar 7, 2006

Elektronik
Supersonik
Marketing wants to force *everyone’s* homepage for Chrome to be SharePoint.

Including the setting that does it for every. New. Tab. Which also gets rid of the normal Chrome startup page that keeps track of your most frequently used sites.

People are not gonna be happy with this lol.

  • 1
  • 2
  • 3
  • 4
  • 5