Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
Nuclearmonkee
Jun 10, 2009


Moey posted:

What the hell is going on over there?

Great things. Post back in here if you haven't changed jobs by the time someone fucks up with snapshots due to letting them all grow until they fill the volume.

Bonus points if it's all thin provisioned without proper alerting thresholds.

Adbot
ADBOT LOVES YOU

Nuclearmonkee
Jun 10, 2009


Orcs and Ostriches posted:

Lightning came in:

It looks like our fire alarm panel and everything hooked up to it / security are gone, but that's none of my problem. I thought I only lost a stack of switches, but after replacing them everything is running so loving weird here. I can't even begin to troubleshoot because none of it makes sense to me.

Just for some examples: the computer in port 10 sees a connection, but can't get an ip address. I plug it into 11, and it starts working. The computer that was working in 11 moves to 10, and it still works. Switching them around stops them both from working.
The computer plugged into port 20 shows the same problem, but I can't move it to any known good port and have it work. I plug my laptop into that port, and it works. I plug the computer back in, and it starts to work.
The computer in 30 doesn't work, so I unplug it / plug it back in the same port. It works until I unplug it / plug it back in. I have to unplug it again, and after plugging it in again it works. I assume I'll need to repeat the whole process again.

There are a couple computers that look like the network cards have died on them as well, so good times all around.

Sounds like there's something weird with mac learning? If you put in a static IP does it work? If you show the mac address table on the switch does it have the correct macs on the correct ports?

Definitely sounds pretty hosed though. Probably need to setup a SPAN port and try to figure out wtf is actually going on.

Nuclearmonkee
Jun 10, 2009


TheShazbot posted:

companies buy toshibas? Ugh.

our latitudes get trashed but still hold together well.

I dropped my latitude about 8 ft off of a ladder the other day and the only part that broke was the usb console cable I was using at the time. I even got that working again with a hammer and some pliers.

Pretty good for a budget notebook or maybe I'm just lucky.

Nuclearmonkee
Jun 10, 2009


Bigass Moth posted:

The steps needed to recover root passwords from a Cisco UCS and from vsphere 5 are infuriating. Yeah just let me shut everything down, no problems there.

Of course the guy who worked here that I replaced went radio silent.

Can't you recover one fabric at a time? I vaguely remember having to do this due to me being a retard and somehow either deleting or not saving the password in the password manager.

edit: nevermind it was done during a maintenance window so I guess I did have to shut the whole thing down. Would it be too much to hope you have someplace else to vmotion everything to or is this a "welp everything's hosed and can't get in until we shut the whole thing down"?

Nuclearmonkee fucked around with this message at 19:28 on Sep 12, 2016

Nuclearmonkee
Jun 10, 2009


MF_James posted:

Sounds like someone else that lives the hell I am in.

I am so glad that my current employer doesn't blink twice when I ask for things like console servers for places where it's important and/or datacenter access is annoying.

Nuclearmonkee
Jun 10, 2009


flosofl posted:

Holy poo poo. A network guy in one of the ops groups was bitching about this over lunch. "It's like the modem all of a sudden defaulted for no reason we can see. ATT said they didn't touch it, and won't send someone out to fix the config since they say it's the customer's responsibility..."

I foresee a REALLY angry phone call in his future.

ATT, Comcast and TWC have equally useless small business circuit service. I just default to the assumption that it will be down whenever it matters so if it matters probably provision another small business circuit from another lovely provider. Though I have seen ATT and Comcast go down at the same time for different reasons at a single site that was only once which is p good for the price.

If the customer balks at it then welp. Just get that idiocy in the CYA folder for when they bitch that their internets are down and they need it back RIGHT NOW and why did you implement this terrible setup anyways?!

Nuclearmonkee fucked around with this message at 23:05 on Sep 12, 2016

Nuclearmonkee
Jun 10, 2009


Thanks Ants posted:

Are fibre circuits with real SLAs prohibitively expensive stateside, or are the build costs what push it out of the realm of affordability? Is it just a case of people wanting to pay as little as they can get away with?

It depends heavily on location but more often than not it's just business folks cheaping out in places they shouldn't.

Nuclearmonkee
Jun 10, 2009


GreenNight posted:

We've waited 9 months for the loving permits to build under some railroad tracks to get fiber to one of our manufacturing facilities.

I'm dealing with the same thing right now. Everything having to do with the railroad moves at such a truly glacial pace that big ISPs and the government look amazingly fast by comparison.

Nuclearmonkee
Jun 10, 2009


anthonypants posted:

We're coordinating a building move and Integra ensured us that the equipment at the future building will be able to handle the circuit we want. We had TWC out lately, and they're saying they need to install a bunch of equipment that won't be ready until December. Which service provider was correct? Who should we be pissed off at? :iiam:

Integra asked TWC and TWC said ya sure without actually checking is the most likely scenario.

Nuclearmonkee
Jun 10, 2009


stevewm posted:

Still waiting on a response from our processor on how they are going to compensate everyone.
They will most likely do whatever legal determines is the bare minimum required.

Nuclearmonkee
Jun 10, 2009


anthonypants posted:

In my experience, local/municipal governments excel at extremely short-term thinking.

I wouldn't really call it thinking. It's more like flailing reflex actions.

Nuclearmonkee
Jun 10, 2009


GnarlyCharlie4u posted:

A ticket came in:
Blackberry finally shut off their cloud services today. So all those people who didn't follow the instructions we sent out to enable IMAP emails can't send us tickets telling us that their email doesn't work. :D

Let's not kid ourselves here. No one followed the instructions.

Nuclearmonkee
Jun 10, 2009


Dr. Arbitrary posted:

Why?
Because that's how it works.

Silence heretic. Do not question the holy ritual of the Subn'Et.

Nuclearmonkee
Jun 10, 2009


Thanks Ants posted:

:cheers:

If you can't get something to link up when set to auto negotiate then it's loving broken, replace it.

Yeah though I still see a lot of "well it didn't work on auto but when I set it to 10 half it worked fine and then we immediately forgot about it forever"

Ideally you should replace your hosed up cable/nic

Nuclearmonkee
Jun 10, 2009


larchesdanrew posted:

A new job responsibility almost came in.

A stomach virus is going around and students are puking and making GBS threads all over the restrooms. Since we have no janitorial staff, we can't expect students to clean bodily fluids as part of their work service.

The director called me into his office.

:v: We've decided that you and the facilities coordinator will be trained in bodily fluid cleanup so you can clean up any messes.

I just looked him in the eye for a solid 20 seconds, and laughed as I walked out the door.

gently caress that poo poo.

Literal IT Janitor.

Nuclearmonkee
Jun 10, 2009


larchesdanrew posted:

So much accomplished. Installed some document cameras, imaged the computers, researched and filled out PO requisitions I've been putting off, replaced some faulty UPSes in the dorms, and finally got around to being able to look over some applications that have come in. All before lunch.

I never realized how much I keep getting pulled away from my duties by everyone else's issues

This is why help desk exists in most places :v: constant interruption destroys productivity and kills your ability to prioritize or complete work.

Nuclearmonkee
Jun 10, 2009


So today I was working on straightening out a horrible mess of a fiber setup at one of the larger facilities. There's large amounts of OM2 strung all over nearly at random, looped through various patch panels, also seemingly at random. Example patch panel:



Note the excellent labeling. The minimal labels that at least tell you which loving building they go to were recently added by me as when I started this there was no labeling and no documentation whatsoever. Also lots of those cables come out of the patch panel and are patched right back in. So I figure first thing I am going to do is build the mythical VEE LAN on an aggregation switch in the worst congested distribution point so that we can consolidate our retarded sprawling parallel connections to free up some fibers. First candidate is traced out through no less than four fiber patch panels. It's a low impact connection (internet for a facility in the complex, which can be down for up to a few hours without affecting production) and once I free it up, it will be easy to pull all of the other connections back into the aggregation switch which will be handling all of the internet connectivity, which by the way is already shared but runs separately on different pairs in parallel eating tons of fiber. The freed fibers will be used for some etherchannel and redundant connectivity (did I mention that there is zero redundancy whatsoever in the current setup?)

Pulled the cable, and corporate data goes out for half of the mile or so long facility which isn't part of the network I manage. I'm just trying to clean up the manufacturing networks that serve the robutts and things which I support. Once guy on the other end tells me that people are asking him why all of their ERP and email connectivity died I immediately reverted the change since it was obvious this fiber was not what we thought it was. Some of the network comes back but other pieces stay dead. Now I don't officially support this network since that's another team's responsibility but I broke the loving thing so I figure I should help fix it. Their network guy is on a plane and can't look at it and there is no backup. Helpdesk doesn't have a NOC screen or can tell me anything beyond "So and so in office 491 in Building 3 can't work" etc etc :suicide:. After walking around and looking for a while I figure out the resulting broadcast shitstorm on their beautiful single /16 broadcast domain err-disabled a couple ports on a few random switches which were not set to autorecover. I eventually track the offenders down throughout the facility and reset them, guided by a map and a panicky helpdesk guy on the phone pinging poo poo and guiding me around.

2 hours later, it's all working thank god. I go back and retrace our fiber and locate the fifth patch panel which the local guy forgot existed. 1 connection down, god knows how many to go. Going back tomorrow with the labeller and going ham on the place.

Nuclearmonkee
Jun 10, 2009


Neddy Seagoon posted:

I dare you to run an OTDR through it to see how long that nightmare runs.

The first one is about 1100 meters 1000base-lx over OM2 without mode conditioning but that wasn't tested just measuring on a CAD drawing. They didn't know what mode conditioning was when I asked and I'm frankly impressed these links come up at all.

Nuclearmonkee fucked around with this message at 04:02 on Oct 12, 2016

Nuclearmonkee
Jun 10, 2009


pr0digal posted:

A flat /16? I'm surprised it didn't cause them trouble before. Why yes I would like my broadcasts to go to 65k other machines!

Also I commiserate with you on the fibre. I love tracking down what port a certain machine is on in this rats nest of a patch panel with no labeling :eng99:

It causes trouble all the time but welp that's not my side of the house I'm just trying to fix the dumpster fire on my side of the fence which is just as bad.

Nuclearmonkee
Jun 10, 2009


I am currently virtualizing an incredibly decrepit and ancient collection of physical 2003 servers at a site running CitectSCADA version Oldasfuck.outofsupport service pack 3. This piece of poo poo software uses USB software license dongles and requires version Slightlylessoldasfuck.outofsupport in order to move to a software key.

This allowed me to virtualize this piece of poo poo so the company can put off upgrading for another 5 years or ideally ever since it will be like 300k dollars to upgrade it.

Thank god for network attached usb

Nuclearmonkee fucked around with this message at 19:51 on Oct 25, 2016

Nuclearmonkee
Jun 10, 2009


Wibla posted:

You're actually upgrading something automation-related. That's not supposed to happen. What's supposed to happen is for that poo poo to keel over at oh dark thirty and a frantic engineer calling me to fix it.

I work in industrial automation and the sheer amount of old / lovely systems out there running critical services that we all depend on scares me.

Some of the newer crap being internet-accessible scares me even more. There's literally no reason why the SCADA system for a water purification plant should be reachable on the public internet.

That happened when one of their 2003 servers crashed and they frantically moved load around to the other servers. That's why I was called and this infrastructure is now being modernized and virtualized.

PLC "security" is a joke and we just flat out tell people nope if they want stupid poo poo related to punching holes through firewalls, which is exactly what one of these vendors wanted me to do. They are pitching a fit due to having to use a VPN to remote into a specific remote access terminal for talking to their PLCs.

No but sorry I don't want to put equipment that handles volatile chemicals and moves at high speeds directly on the internet so that your support team can get into it at will with no access log.

Nuclearmonkee
Jun 10, 2009


bpduguard is your friend if you don't have nac and even if you do.

Nuclearmonkee
Jun 10, 2009


Powered Descent posted:

Excellent advice, if you have actual enterprise gear. But not particularly relevant if your infrastructure is built out of whatever 24-port switches happened to be on sale at Fry's when you did the office move. :)

Sounds like a real joy to maintain.

Nuclearmonkee
Jun 10, 2009


Judge Schnoopy posted:

What are people using in place of Cisco Anyconnect on Mac? My predecessor let them use PPTP until Apple (rightfully) pulled it, now these people want back on the VPN but Anyconnect isn't in the ITunes store.

These are personal devices so I'm not installing any cisco packages on there unless I have 0 other options.

Any easy solutions that are Anyconnect compatible?

Mac built in VPN client works for just fine and does IPsec VPN with minimal pain.

Nuclearmonkee
Jun 10, 2009


Wilford Cutlery posted:

Having sent her the instructions, what's happened here is that the IT guy who's been here less than 1.5 years has just shown the company's longest tenured employee (20+ years) how to do her job.

This isn't that unusual. Lots of people would rather just give someone else any problem involving critical thought.

Nuclearmonkee fucked around with this message at 20:05 on Feb 10, 2017

Nuclearmonkee
Jun 10, 2009


MF_James posted:

hhahahahah you think places like that mandate password changes?

Forcing users to change passwords with a complexity requirement that they are allowed to pick is so pointless that I don't know why people even bother. If people care about security use 2FA or it's just pretending to care.

They will use sticky notes or figure out some retarded system for keeping track of their password which defeats the purpose.

Nuclearmonkee
Jun 10, 2009


Collateral Damage posted:

Unpopular opinion; It's better that users keep their passwords on a piece of paper in their wallet than using "Password123!" as their password everywhere.

Whatever stupid thing they pick to replace it will be just as bad and pointless. Usually complexity requirements will preclude them from using their AOL password anyways and their corporate account name will likely not match that username either.

Flatscan posted:

PCI compliance in my case.

This is one of the "we must pretend to care because it's the rule" scenarios.

edit: also you can use freeRADIUS with google authenticator and have 2FA for free if the place you work doesn't want to pay for RSA tokens or whatever (they never do).

Nuclearmonkee fucked around with this message at 18:31 on Feb 15, 2017

Nuclearmonkee
Jun 10, 2009


Takkaryx posted:

When I went to university and worked their helpdesk part time to pay for rent/food/booze, our password requirements changed from the normal upper/lower/number/special to include cannot contain words longer than 2 letters found in the dictionary. We had manifesto length complaints from faculty, staff and students.

Those are the best particularly when they say "I can't use $dogs_name+$street_address which I use to password everything including these twenty services which were all hacked in the last year or two and now I can't remember my password thanks to you :argh:."

Nuclearmonkee
Jun 10, 2009


Ok that works in communist europa but here in 'murica a national ID is the mark of the beast dontcha know. Can't have that.

Nuclearmonkee
Jun 10, 2009


MJP posted:

After so many years of bosses that were mediocre at best, lovely awful garbage piles at worst, my current boss is a freaking saint and a joy to work for.

Yesterday: I did a Move instead of Copy of VHDs for a powered-off VM. Upon the copy failing, I deleted the source files, including VHD, and realized I'd have to rebuild the VM from scratch (no, we weren't backing it up - in fact, I was moving it to make room on the drive to be able to back up said VM). He was jovial about it: "It's not super critical, as long as we can get it back up, and I have faith that you know how to put Humpty back together again."

Straight Win 7 desktop build, did a few extra hours last night to make it happen, and we're good for 9 AM when it starts becoming needed again, got an attaboy for my efforts.

Today: our phone system dies. 12:31 PM. I start troubleshooting, don't quite get it down pat, run some quick plans of action past the boss, and he's equally non-perturbed: "You're on it, you know what you're doing, and even though I have to bail to head up to a remote site for a pre-existing appointment, it's not the end of the world. People will use their cells."

It turned out the cert on our PSTN gateway expired. Worked with the corporate office IT people, generated a new request, got it signed, back in business three hours later.

At no point did I fear for my job, or for political retaliation.

There's no real vertical room unless I wanna move to the corporate headquarters in the Midwest (I'm in the NYC metro area... yeah, no) so maybe I could just stick out the rest of my career in this kind of really nice environment.

I'm not used to being treated with... respect? Is that the word here?

It's nice isn't it. I will never work at another shitshow place again. I don't even pretend in interviews I just flat out tell them "If your culture sucks I'm out. Don't even bother hiring me if it does. You all know what I mean."

Nuclearmonkee
Jun 10, 2009



This is the retention policy at my current place of work



And the official way to archive things is to make a pst and stick it on your network share. I am fortunately not responsible for email/file shares in any way shape or form but the people who are are extremely bad at their jobs.

Nuclearmonkee
Jun 10, 2009


IronSaber posted:

"Yeah, I just got an email from *wheezing noise* at mailinator dot com. Do I open it?"

The answer of course is yes for the vast majority of users because the "someone has shared a document with you" thing is pretty close to the internet equivalent unwrapping a present and human beings are insanely dumb.

Nuclearmonkee
Jun 10, 2009


Sefal posted:

I made the mistake of saying that I did some stuff with linux in school.

Now i'm the guy who people occasionally ask to fix issues on a linux server
I can barely navigate through vim.

If they are only asking for basic administration a functional brain along with google skills and a little 'nix familiarity is enough. I always just give the general disclaimer of "I am not an expert at this and if you want it to be really secure and good you need to pay someone who can do that."

I honestly don't mind though as it's a useful skillset to have and it gets rusty if you don't use it like all things.

Nuclearmonkee
Jun 10, 2009


GnarlyCharlie4u posted:

In my experience, Lunix = free, so use that. Also, we're too cheap to actually pay you anything to do it, that's why we want the free thing in the first place.

Use them to build resume, get certs, and go work somewhere that will pay.

Nuclearmonkee
Jun 10, 2009


sixth and maimed posted:

Any other measures I can take (apart from user education)?

That's basically like trying to train emu to remember complex tasks. Same caveats apply; you will require a reward and training must occur daily

Nuclearmonkee
Jun 10, 2009


One of my links went down to a remote manufacturing facility. I start putting in a ticket with the provider and get a highly entertaining call from the on site electrician asking why no remote connectivity works and also by the way we were doing some work back there and moved a white box about two feet on the wall.

Had them send me a picture it was the ATT demarc box, complete with all of the labels covering it saying don't loving touch this call this number etc. :suicide:

Nuclearmonkee
Jun 10, 2009


A ticket came in.

Hey Windows won't activate on any of those new servers you guys built for us a few weeks ago.

Check, determine that KMS is real pissed because none of them went through customization when they were deployed because the guy who deployed them "Didn't realize that mattered I've never done that before. I just set their hostnames/IPs/joined domain manually after cloning from template."

They've been in use for a couple weeks so I can't just blow them away and have them redone without people getting mad. I'm thinking best option at this point is just telling retard to use the MAK key and making a note that we have 32 servers not using KMS because of stupid, and then telling him to migrate them all over time as maintenance windows permit.

Nuclearmonkee fucked around with this message at 01:13 on May 19, 2017

Nuclearmonkee
Jun 10, 2009


GreenNight posted:

I don't understand. I've gone into KMS and sent servers setup with MAK a KMS key and activated it to KMS and it worked fine.

If they all have the same CMID because they didn't go through generalization.

Nuclearmonkee
Jun 10, 2009


GreenNight posted:

Oh they didn't sysprep. They cloned from template not deployed from template. Nice.

I don't even understand it at all. The guy was shown how to deploy a huge pile of vms very easily through a pre-existing script. Just had to dump in the pile of names, let vmware do its thing for a bit and then setup permissions for the appropriate people to get into them and do whatever. He did it all manually after being showed the lazy way.

Adbot
ADBOT LOVES YOU

Nuclearmonkee
Jun 10, 2009


Rudager posted:

It depends, I mean like a bit of bank software hacked together to stay running on legacy hardware/software probably won't kill you, but a forced bit in a PLC program to override a dodgy sensor that's part of the safety circuit very easily could.

I just had an outage where a local plant electrician/plc programmer bypassed a temperature alarm and melted a rather expensive piece of equipment.

  • Locked thread