|
Thanks, I think I'm actually just frustrated that my linux skills are completely wasted at this job.
|
#
?
Sep 29, 2014 18:47
|
|
|
|
| # ? May 10, 2024 08:59 |
|
|
evol262 posted:This is actually true. If it's not exposed to the internet or not running CGI (or, not and -- it doesn't matter if it's running CGI inside your network unless you expect people to try to break a vendor appliance internally), it's not a big deal. Lots of people are convinced their appliances aren't touching the Internet. Then someone finds a Netgear switch from 1998 in a broom closet.
|
|
#
?
Sep 29, 2014 19:15
|
|
|
I'm desperate for my systems to connect to the Internet 
|
|
#
?
Sep 29, 2014 20:01
|
|
|
Ynglaur posted:Lots of people are convinced their appliances aren't touching the Internet. Then someone finds a Netgear switch from 1998 trapped between two desks/between the back of a desk and the wall/strung from the bottom of a desk with zip ties/in a goddamn drawer.
|
|
#
?
Sep 29, 2014 20:15
|
|
|
Didn't someone have some pictures/stories of finding working switches and/or servers drywalled in between walls? Or a room got drywalled over, still containing working equipment.
|
|
#
?
Sep 29, 2014 20:22
|
|
|
Some moron at a previous job drywalled a fiber breakout cabinet thing into a ceiling. I don't know what goes through people's heads when they see that and decide that it would be much better off if it were inaccessible.
|
|
#
?
Sep 29, 2014 20:35
|
|
|
Thanks Ants posted:Some moron at a previous job drywalled a fiber breakout cabinet thing into a ceiling. I don't know what goes through people's heads when they see that and decide that it would be much better off if it were inaccessible. "It's 3 o'clock. I sure wish I were drinking a beer right now. Let's wrap this up, shall we?"
|
|
#
?
Sep 29, 2014 20:51
|
|
|
I seem to remember someone with a story of an elderly user requesting that her tower be walled off (and this somehow actually being done), and that over time everyone with any knowledge about it either retired or moved on. Then her computer finally bit the dust, and no one could find it, forcing them to break out blueprints or something to get to the bottom of it.
|
|
#
?
Sep 29, 2014 20:52
|
|
|
stubblyhead posted:I seem to remember someone with a story of an elderly user requesting that her tower be walled off (and this somehow actually being done), and that over time everyone with any knowledge about it either retired or moved on. Then her computer finally bit the dust, and no one could find it, forcing them to break out blueprints or something to get to the bottom of it. This sounds like the opening of a D&D module. Yes I am a geek.
|
|
#
?
Sep 29, 2014 21:44
|
|
|
SubjectVerbObject posted:This sounds like the opening of a D&D module. Yes I am a geek. I remember this. This was at a university, I believe.
|
|
#
?
Sep 29, 2014 21:45
|
|
|
One of the staff that worked this weekend emailed me a kudos for coming in so quickly to get all the systems back up. She copied in the HR VP. The HR VP replied with a kudos of her own and copied in... the CEO!  Guess she didn't get the memo.
|
|
#
?
Sep 29, 2014 21:48
|
|
|
Hell yes. I just got to tell one of our sales guys to pound sand. According to my coworker, it was magnificent- The rear end in a top hat was trying to pull back hardware we had already purchased for a customer that was in staging to ship out. His reasoning? Using current gen hardware would cost us $50k in profit and we should only send them half the servers i ordered. My boss told me specifically to order current gen hardware and that we needed to put our best foot forwards for this customer. I could feel the sales guy squirming as I'm sure this came straight out of his commission. Screw you buddy, out of everyone here, you've earned this. 
|
|
#
?
Sep 29, 2014 22:36
|
|
|
mewse posted:Sysadmin bravely navigating an outage: Haha, best thing about that .gif is that he ejects into what would be Day 6 or something - It just got too much. "This is just unsalvageable. gently caress it, bye!"
|
|
#
?
Sep 29, 2014 22:43
|
|
|
Our EHR has been down since this morning. No one can do any work and I can't do a drat thing to fix it since we just contract with another company for access, and they are the ones that talk to the actual vendor.
|
|
#
?
Sep 29, 2014 22:52
|
|
|
Thing currently pissing me off: NetApp So I go to their Update Advisor (which is very handy) and find out we are 2 point releases behind, which requires upgrading to each point release before going to the next one. So that means I have to go from 8.0.2 to 8.1.3 to 8.2.2. Also, apparently the intermediate upgrade breaks a lot of stuff, so we are using vendor support to hold my hand because I don't know dick about storage or SANs outside of "Storage Area Networks for Dummies" and 10 hours of boring NetApp videos that were more of a sales pitch than anything actually beneficial.
|
|
#
?
Sep 29, 2014 23:48
|
|
|
Enterprise SANs are one thing you really shouldn't be touching unless you know what you're doing or are under the close supervision of someone who does (and being on the phone with a vendor support tech who might well be almost as clueless as you are doesn't count). There are a lot of potential pitfalls with enterprise storage and when the consequences can be "welp all of your company's data is gone forever," it's worth paying for someone who knows what they're doing to come in and help, even if it's just for long enough to get you up to speed.
|
|
#
?
Sep 30, 2014 00:51
|
|
|
gfsincere posted:Thing currently pissing me off: NetApp dennyk posted:Enterprise SANs are one thing you really shouldn't be touching unless you know what you're doing or are under the close supervision of someone who does (and being on the phone with a vendor support tech who might well be almost as clueless as you are doesn't count). There are a lot of potential pitfalls with enterprise storage and when the consequences can be "welp all of your company's data is gone forever," it's worth paying for someone who knows what they're doing to come in and help, even if it's just for long enough to get you up to speed. I might be inclined to hold off on the updates until this Shellshock thing is sorted. All DataONTAP is affected and I haven't heard yet exactly how NetApp plans to patch. No sense planning a huge upgrade only to learn that next week you have to do it again. e: this was apparently my thousandth post 
Aunt Beth fucked around with this message at 01:00 on Sep 30, 2014 |
|
#
?
Sep 30, 2014 00:57
|
|
|
We had some consultants come in once who did IT work all over the world and had plenty of stories to go along with it. They were doing an installation somewhere in the midwest doing network design or some such and found a workstation completely cabled, running and drywall enclosed in a wall in the drop ceiling. Why would anyone make so much effort to make sure a mystery workstation continued to exist?
|
|
#
?
Sep 30, 2014 02:31
|
|
|
Simpleboo posted:Why would anyone make so much effort to make sure a mystery workstation continued to exist? Seedbox? Cpt.Wacky posted:Our EHR has been down since this morning. No one can do any work and I can't do a drat thing to fix it since we just contract with another company for access, and they are the ones that talk to the actual vendor. Who's your contractor? Perot? Cenodoxus fucked around with this message at 03:31 on Sep 30, 2014 |
|
#
?
Sep 30, 2014 03:25
|
|
|
The loving service database on an old-rear end Server 2003 machine is locked. It has been locked for what seems like forever. Neither our technicians nor the on-site IT Director can figure out how the hell to get them unlocked. Everything we've tried is a circular pattern right back to "Sorry, Error 1055: cannot do that because the service database is locked" I just stayed awake an extra hour to reboot this machine after all users who need VPN file access have finished for the night... and it is still locked. One of the engineers who is a computer wizard said "this should do it, if it doesn't, who knows what the gently caress" E: HOLY gently caress, I ran sc lock on a whim and it worked! It loving worked! I'm going to buy that engineer a beer and some roses, this ticket has been a pain in my rear end for as long as I can remember. Looks like he manually scanned through the registry and found a service that was disabled, but still set to start automatically in its regvalues. I don't even want to imagine sifting through seventy services to find the one service that is loving up that badly. E2: ahaha holy poo poo, the filepath for the installer I need to run now i just c:\users\badmin. This is almost as good as the client we manage whose entire server fleet is named after Game of Throne characters. bawk fucked around with this message at 04:43 on Sep 30, 2014 |
|
#
?
Sep 30, 2014 04:37
|
|
|
Aunt Beth posted:I might be inclined to hold off on the updates until this Shellshock thing is sorted. All DataONTAP is affected and I haven't heard yet exactly how NetApp plans to patch. No sense planning a huge upgrade only to learn that next week you have to do it again. The vendor support isn't coming in to help us with the upgrades until mid-October, so hopefully Shellshock is patched by then.
|
|
#
?
Sep 30, 2014 05:24
|
|
|
Cenodoxus posted:Who's your contractor? Perot? Our contractor is just another business like us but 5x bigger. It's a real shitshow since they are not professional SaaS providers, or professional anything really. About 30 minutes after I posted that they gave us the all-clear to start using it again. The only explanation provided: "vendor installed a new product for testing that was causing the issue"
|
|
#
?
Sep 30, 2014 05:30
|
|
|
Johnny Aztec posted:Didn't someone have some pictures/stories of finding working switches and/or servers drywalled in between walls? Or a room got drywalled over, still containing working equipment. stubblyhead posted:I seem to remember someone with a story of an elderly user requesting that her tower be walled off (and this somehow actually being done), and that over time everyone with any knowledge about it either retired or moved on. Then her computer finally bit the dust, and no one could find it, forcing them to break out blueprints or something to get to the bottom of it. "I reiterate: what janitor's closet?"
|
|
#
?
Sep 30, 2014 05:31
|
|
|
I'm impressed with how well I remembered the details of that story.
|
|
#
?
Sep 30, 2014 06:23
|
|
|
death .cab for qt posted:This is almost as good as the client we manage whose entire server fleet is named after Game of Throne characters. Are their servers abruptly powered off and destroyed without warning at the CIO's whim?
|
|
#
?
Sep 30, 2014 13:56
|
|
|
stubblyhead posted:I seem to remember someone with a story of an elderly user requesting that her tower be walled off (and this somehow actually being done), and that over time everyone with any knowledge about it either retired or moved on. Then her computer finally bit the dust, and no one could find it, forcing them to break out blueprints or something to get to the bottom of it. In the beginning of my career, I was on a project to upgrade infrastructure from Novell 3.11 to Novell 4, and MSMail to Exchange. Our first site to convert was my home site. We figured we'd learn tons from it, get practice, and become well oiled team from it. Boy, were we right. We were upgrading a novell 3.12 server (phxfs026, I remember this beast to this day) that was a small departmental server for a helpdesk. The last decom step was after we shut down the novell fileserving bits, we were to walk up to the server and power it off and remove it. Realize, this was 1996, the dark ages as far as datacenter management went. The best you could do in these days, without spending TONS of money was labeling the hardware, and walking around with a crash card. So I walk up and down the racks, looking for this piece of hardware. It's not there. I figure, "ok, one of these servers is mislabeled. I'll just attach the crash cart to each and verify" Nope. Everything was as it should be. So I ask one of the grizzled old novell guys. "OH yeah, that server is in rack 2-B!" We walk over, and of course not. Our new test SNA gateway is there. (which ran Windows NT server) So I had a project. I was a young kid, with a lot to learn, so I figured chasing this down would be fun. I got MAC addresses for everything I could grab in the datacenter and then went to the network guys. I Got a nice list of switch ports with known entities on them, and a list of six switchports which had things not on my list. Friday night, I got the OK from change management to unplug each of those six, to see which port would make the hidden server fall off the network. Of course, it was the last port I tried. So I find the port, and it's this mangled yellow cable that is wrapped around the power distribution cable down into the false floor. I start pulling tiles to follow it, and find the server underneath the floor, under the rack it should have been in, dust caked all over the intakes. I sneezed just looking at it. EDIT: The most amazing thing just happened. Remember my rant about UTC vs GMT? (its back in here somewhere.) I had a convo directly with the customer yesterday, adn emailed him the details. I had an email this morning: quote:Ok, that makes sense. It's fine the way it is then.
|
|
#
?
Sep 30, 2014 14:14
|
|
|
I've been working on this loving PBX call connection/audio delay for months now. Everything I want to do to try to fix it hasn't been possible because the CIO never wants us to do work outside of business hours (what the gently caress) and I'm absolutely not allowed to take the phones down ever for more than about 5-10 minutes at a time, and even then only after 4pm with ample warning (so usually only on Fridays). After a lot of struggling in vain, I realized that this problem started when a couple of sales reps moved and we accommodated them by sending them Polycom phones and allowed remote access to the PBX... by turning on the PBX's NAT Port Forwarding. It barely makes sense, but as soon as we turned that on, the reports of the delay started. So I suggest that we turn that setting off, as a test. Obviously the issue with that is that it will sever the connections of the remote phones. I insist that it's a necessary evil. First, the CIO doesn't want to tell the remote users to use their cell phones for a couple days. Fair enough. Then, nobody wants to consider the option of doing what a lot of people recommend and providing VPN routers to the remote employees to connect their phones with. Sigh. So I suggest that we try out a cloud PBX from the same company, literally a cloud version of our current PBX. The remote users can connect to that, and the two systems can be peered. With the Cloud PBX functioning as a VoIP provider, we can turn off the NAT Port Forwarding setting and see if it helps! This is acceptable to management, as a quick test because I work for crazy people. I had to send a proposal, which my boss sat on for the remainder of the week until saying "let's do the 30 day trial" the following Monday. We got the trial process started with the vendor. They say it will be spun up in 10 days. They insist that they need to send us the 5 phones included with the trial and allow time for that. We talk them down to 3 business days to spin up the VM. Then they fail to get the server spun up in the promised 3 days; it's the following Monday (yesterday) instead. The entire time I'm waiting, I'm getting emails from my boss asking me what the status is because the CEO, who is basically the only person truly upset by this delay, is going apeshit about it. The server is finally up, but WAY before they send me the credentials, which I get 10 minutes before the end of my day. Great, I can finally log in, time to at least add our PBX as a VoIP Provider and finally get this show on the loving road. "You do not have write access to this section." I'm going to throw my chair through the window. Then I can escape. Fortis fucked around with this message at 15:52 on Sep 30, 2014 |
|
#
?
Sep 30, 2014 15:49
|
|
|
poo poo that really pisses me off: "Tech support" that admits they see the problem in a way that makes it clear it's nothing on my end but then reverts right back to the script for two hours of going in circles doing nothing useful. My office has "business class" cable (contract and everything) with two static IPs. The way this ISP works I just have two sequential addresses out of one of the many /24s associated with the local node, so I have x.x.x.xx0 and x.x.x.xx1 with a gateway of x.x.x.1. This morning I get in and one of my coworkers tells me the main IP isn't working but they switched our primary NAT over to the second one and we're online. Great... A few months ago the ISP accidentally gave away our main IP and it took a few days to get it back, so I'm hoping that hasn't happened again. Unfortunately we still have a lot of IPsec tunnels built on IP addresses, so without the main IP working we're crippled as far as working on certain customer networks. Log in to our router and set it back to the main one, I can ping our gateway address but no further. A bit more testing using one of my VPSes and I can see that my pings are getting out just fine but are not making it back in. Traceroute to the problematic IP and I see this: code:Call the ISP up, explain the problem, go through the motions of setting the IP on a laptop plugged directly in to the modem, get put on hold. They come back, ask me to do the laptop thing again. Go on hold again. Repeat another two times. They confirm they're seeing the same loop as I am, even when my modem is entirely powered off, yet the ask me to do the laptop thing three more times and then start asking about DNS settings. I really hate service providers where the "Tier 1" guys act as interfaces between the customer and higher levels in cases like this. There was clearly someone on the other end of a chat session from the guy I was talking to who had half of a clue, but this guy just kept returning to his script even when we had gone far beyond proving the issue was in the ISP network. I'm no DOCSIS expert but probably know a lot more about it than the average person and I'm pretty sure there's no possible way these symptoms can be caused by anything at my end of the coax. edit: Hey, it's finally back up! Wonder if it'll last... Still haven't heard back from the ISP. e2: stevewm posted:As much as I would like to say gently caress you to Comcast, for some of our branch locations they are stuck with them. Frontier is often the only other game in town. And around here their craptastic "6Mbps" connections barely manage 1Mbps most of the day. Same basic feeling in Comcast and TWC territories. As bad as either of those companies are to deal with, AT&T and Frontier are worse and DSL almost never even delivers the rated speed where cable usually is capable of exceeding it off peak. wolrah fucked around with this message at 17:03 on Sep 30, 2014 |
|
#
?
Sep 30, 2014 16:54
|
|
|
Comcast..... To make a long story short, after taking multiple months to get service activated at one of our branch locations, we find out Comcast somehow setup 2 accounts for the same location. We start getting billed for this fictitious account accordingly and eventually multiple collections calls per day since we refused to pay the bill. After weeks of multiple phone calls, hours on hold, getting hung up on, transferred to silence, getting multiple confirmation numbers, being told those tickets/confirmations didn't exist/where done wrong/etc.. I blew up and fired off a string of obscenities resulting in the agent hanging up on me, again. Obviously calling them was never going to work to get this fixed. So I took to their @comcastCares twitter account. It took them several days, but surprisingly someone actually replied to me. They promised to escalate the issue to the highest level. Today I received a phone call from someone at the Comcast Corp. Office. They said the matter had been resolved; the account had been deleted and all charges wiped out. They apologized for all the problems. Gave me a confirmation number (ugh) and also their contact info. We shall see if the account was actually wiped out. I don't trust them, it is Comcast after all. As much as I would like to say gently caress you to Comcast, for some of our branch locations they are stuck with them. Frontier is often the only other game in town. And around here their craptastic "6Mbps" connections barely manage 1Mbps most of the day.
|
|
#
?
Sep 30, 2014 16:58
|
|
|
wolrah posted:I really hate service providers where the "Tier 1" guys act as interfaces between the customer and higher levels in cases like this. There was clearly someone on the other end of a chat session from the guy I was talking to who had half of a clue, but this guy just kept returning to his script even when we had gone far beyond proving the issue was in the ISP network. I'm no DOCSIS expert but probably know a lot more about it than the average person and I'm pretty sure there's no possible way these symptoms can be caused by anything at my end of the coax. Having been the tier one person--they could lose their job for skipping the script and escalating. Pray for their souls. Tier1 is purgatory.
|
|
#
?
Sep 30, 2014 17:12
|
|
|
I've had so much better luck with Comcast support by saying something snarky about a problem on twitter and adding @comcast and some variation of #fail at the end than with calling them it isn't even funny.
|
|
#
?
Sep 30, 2014 18:19
|
|
|
wolrah posted:
DSL can actually deliver decent and rated speeds, however the ISP running the system has to be worth a poo poo. AT&T and Frontier do not meet this criteria. Some of our branches and our corp. office are served by a local phone company turned ISP that uses multiple technologies. They rate their ADSL connections at 5/1 and they actually achieve this at all times. If you are not getting your rated speed, call, and they won't hesitate to fix it. They also just recently started switching to VDSL in all areas. Our corp office was able to go from 5/1 to 20/3 on VDSL.
|
|
#
?
Sep 30, 2014 18:25
|
|
|
meanieface posted:Having been the tier one person--they could lose their job for skipping the script and escalating. Pray for their souls. Tier1 is purgatory. Yeah I'm pretty sure that when most of us bitch about "Tier 1 Support," they don't mean any individual but rather what amounts to the castle gates and the ironclad, inflexible policy of "you read script or you no job" in all circumstances. They don't want people working for them at that level, they want robots capable of parsing human speech halfway decently.
|
|
#
?
Sep 30, 2014 20:45
|
|
|
There's a dog in our server room. That is all.
|
|
#
?
Sep 30, 2014 21:22
|
|
|
I had someone bring in a toddler once and they put them in the server room (not alone) because it was the only room with a door.
|
|
#
?
Sep 30, 2014 21:53
|
|
|
I can't think of a better place to put a pet/toddler than a room specifically set aside for extremely expensive business electronics
|
|
#
?
Sep 30, 2014 21:55
|
|
|
Just found out our main clinical application will let users enter invalid characters into fields without warning them (i.e. letters in to numbers only fields) until they finally go to "sign" the note, at which point the database will reject the entire note. These are branching path notes that can take 30+ minutes to fill out. Oh and the vendor blames our "network" (read: citrix application on a VM talking to a DB on a VM in the same environment) whenever it flips out and takes up 100% of a vCPU or locks the DB. Bob Morales posted:There's a dog in our server room. That is all. I have deja vu...
|
|
#
?
Sep 30, 2014 22:03
|
|
|
Bob Morales posted:There's a dog in our server room. That is all. e: LANs best friend
|
|
#
?
Sep 30, 2014 22:35
|
|
|
mewse posted:I can't think of a better place to put a pet/toddler than a room specifically set aside for extremely expensive business electronics Well, if you let C-levels in...
|
|
#
?
Sep 30, 2014 22:40
|
|
|
|
| # ? May 10, 2024 08:59 |
|
|
stuxracer posted:e: LANs best friend DHCP leash VPN kennel arf table
|
|
#
?
Sep 30, 2014 23:00
|
|