Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

At least you were doing replication as well as your regular mysqldumps/XtraBackups, so you didn't lose much, right?

Adbot
ADBOT LOVES YOU

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Return Of JimmyJars posted:

Of course that doesn't help when the colo customer below you comes in and rips all the cables out of your servers, removes them from the racks and just tosses them on the floor because "it was easier for me to work on my server."

Are there seriously data centers who allow colo customers to have unsecured hands-on access to other customers' systems these days? :psyduck:

A web hosting company I used to work for actually did have that kind of access to the facility we rented space from, but our situation was unusual; we actually used to own the entire data center, then we sold off our dedicated server business and the data center itself to another company. We still had hundreds of shared hosting systems scattered across the 300+ racks in the DC, though, which were on a wide variety of ancient hardware platforms and required a lot of regular hands-on attention, so we arranged to retain full access to the DC. Eventually our stuff was consolidated down to a few dozen racks, but since the DC was never built for shared access, we could still have laid hands on any of the other systems in there. (At that point, the DC owner decided to beef up security by having us sign in on a sheet before going into the DC... :v: ) The two new DC facilities we ended up moving most of our stuff to had the typical individual secure cages, though.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

doomisland posted:

Pretty standard server though low on RAM. We have random R410s chillin since their projects got side tracked but they can just be repurposed for testing stuff. Only issue is they usually only have 12/24GB of RAM so there isn't much you can do with them. And that's the stuff that doesn't get shipped somewhere and racked.

Yeah, having servers sit around unused for a while isn't all that uncommon in any place with a decent-sized IT footprint. At that web hosting company, we had a good-sized provisioning room that was pretty much stacked floor to ceiling with unused hardware. Most of it was old stuff (either kept around to be reused for new builds because we were too cheap to buy new hardware, or to be parts donors or chassis swaps for our ancient production systems if they failed), but we had some new R620s and such sitting around for a few months here and there. Usually the new stuff would be "earmarked for a project" that wouldn't actually happen, then eventually someone would say "gently caress it" and put them to use when some new critical "OMG we need this now!" build came down the pipe and we didn't have any other suitable hardware.

At my current place, we actually have no spare hardware to speak of, mostly because almost all of our systems are still under warranty. We basically order new hardware as we need it. Most of our new server builds are VMs on one of our big blade clusters, unless they're just too big for a VM or have some other reason to require their own hardware, so we can still spin up most new projects pretty quickly. We've still had a few servers that have sat around for a year or two before they were actually put into production, though they were racked and running the whole time and it was more because the project they were for kind of stalled out for a while.

And yeah, that R710 should do fine, but probably does need some more RAM to really justify it, unless you guys don't have a VM cluster for smaller builds. Hell, we're running a bunch R710s and R720s that have 192-256GB of RAM (basically they're huge search index cache boxes). Actually, that was one of my first projects when I started here; the servers only had 96GB of RAM and would start swapping heavily and die after running for a few days. Apparently several folks had looked at the issue before me and said "Add RAM", so I did some more in-depth analysis and said the same thing, and now they have 192GB each and are perfectly happy (except when the occasional drive dies and it makes the drat PERC controller blow a gasket and drop other random drives from the array... :v: )

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

couldcareless posted:

I had to nicely explain to her that the backups aren't there for her personal convenience and that loading up our old journaling databases is tedious and that she can't just delete everything willy nilly and send us multiple requests a week asking us to pull it for her. I'm not sure I got through to her.

You probably didn't. It's likely that she neither knows nor cares how much trouble you have to go through to get her backups. The correct solution to this is to get your boss to implement chargebacks for your IT work and then bill her department/manager for every backup you have to pull. The only way anyone gives a poo poo how much work their requests are going to require someone else to do (unless they're actually good people, of course) is when an amount of money reflecting that work gets taken out of their department's budget.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

xThrasheRx posted:

No idea why the hell its standard to have those silly holes, almost all balconys have them here, potential draining from water or something?

Yes, exactly. If the water can't go over the side of the balcony, then it goes into your house. You really don't want that.

As for beer, all the beers I like are pretty expensive, though at least beer prices in general in Georgia are decent. Luckily I usually only drink a couple beers a week, so I ain't gonna go broke indulging in the good stuff. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

skipdogg posted:

Do always plug in your iLO cards people.

Judging by past experience, if there is one server without remote access in your data center, that server will always be the next to crash, without fail, leaving you with the choice of paying $20 out of your own pocket for parking or walking twelve blocks through South Murderville to the data center at 2AM to reboot it.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse
We've pretty much doubled our department's staff this month at my place, which is awesome. For quite a while we've had one AIX admin (managing well over a hundred LPARs), one VMWare admin (managing three VMWare clusters in three separate data centers with several hundred VMs between them), and one SAN guy (managing like a dozen Netapp SANs in the same three data centers), with no real redundancy (I can handle some of the server-level AIX stuff well enough, but haven't really touched the HMC, network, or storage stuff, and I know enough VMWare to set up VMs, but not much about managing a whole cluster or the storage and networking configs). Our boss was basically the only backup for AIX and SAN stuff, but he's a director now and doesn't really have time to be doing the day to day stuff anymore. Now as of this month we have two new AIX admins and a new SAN guy, and soon we'll have a new VMWare guy. We've also hired a new Oracle DBA or two to help our current DBAs with our 150 or so Oracle databases.

We do have two Linux guys already (including myself) though the other guy's primary duty is managing the Nagios monitoring system and that's nearly a full-time job all by itself, so I'm really the primary Linux admin. He can back me up when I'm not there, though, and so far I've been able to handle the workload well enough (though it does get pretty crazy some weeks when it seems like every department is doing some big project all at once), so we can't really justify hiring another admin yet. We're probably gonna hit 200 Linux servers pretty soon, though (if we haven't already; people keep building new Linux VMs without telling me :v: ), and we're always adding more, so one of these days we'll probably have to expand.

One thing that's a bit of a pain with all those systems is patching. I'm doing kernel updates every quarter and other package security patches every month, and right now I'm more or less doing patches manually (with the help of some hacked-together scripts and a little database to track stuff). The security updates aren't so bad, but the kernel patches are a pain; I have to contact the business owners of every server to work out a schedule for a reboot every quarter and then have to spend a few days (including a weekend or two) manually rebooting every server. I've thrown up a Spacewalk server and added a few systems to it as a test and it seems like it could make some things quite a bit easier (in addition to being a much nicer inventory system than my clunky little custom database), but I'm not sure if it'll get approved by the boss or not.

Oh yeah, and one of our upper-level directors has now decreed that *all* of our systems will have to be rebooted once a month no matter what. Luckily, our boss is trying to use that to push for setting up scheduled automated server reboots and application restarts for as many systems as possible, because he knows there's no way in hell our team is going to manually reboot hundreds of servers every month (many of which would have to be rebooted overnight and on weekends, which would really suck since we don't get any overtime or comp time for after hours work). At least if we actually could get automatic reboots in place and working reliably, it would make kernel patches a hell of a lot easier.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

DGK2000 posted:

Holy poo poo, I understand schools are cheap. But for the love of god, why did you connect a computer to a connector, to a punchdown block to a switch when all 28 computers are in the same room? Did whoever you select decide he wanted to make it as complicated as possible? On top of that, run another cord to a different switch across the room, and put 2 more switches behind the main switch all without labeling anything? Oh, and running wires through the ceiling without them being up to code. Jesus christ.

I remember way back in the day when my high school got a lab full of brand new first-gen PowerMacs. Since the school was old, there were like three outlets in every room, so someone came up with the bright idea to daisy-chain a dozen power strips together and plug about thirty PowerMacs into one wall socket. The first time the lab was used was when I was helping teach an elementary school class some multimedia stuff over the summer. Thirty little kids file into the room, sit down, and all push the power buttons at once. Five seconds later, as all the monitors come on, there's a terrible crackling sound and visible arcs of electricity start jumping out of the poor overloaded wall socket. :psypop: Thank goodness no one was close to the outlet at the time, though one assistant started to reach down and yank the power cord; he came within a few inches of getting fried before he reconsidered his plan.

Later that week, after messing with the (now properly powered) PowerMacs and their fancy CD-ROMs with built-in trays for a while, one kid ended up using an older Mac in one of the other labs, and managed to stuff about three or four bare CDs into the caddy CD-ROM drive before he figured out that something wasn't quite right. We had to take the whole system apart to get 'em out. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Volmarias posted:

T-Mobile is almost this bad. You need to tell the nice robot lady who you need to talk to today. She'll give you two tries, and if she doesn't understand you after the second try for any level, she'll say "Sorry, I'm afraid I couldn't understand you. Goodbye!" AND DISCONNECTS THE CALL.

I nearly threw my phone through the window the first time this happened.

I guess too many customers finally figured out that just screaming incoherently while mashing the 0 button repeatedly would usually get you to a live person without having to deal with their bullshit menus, which meant they still had to actually hire live people pay some company overseas to provide live people for customers to talk to.

The best calls are the ones where you spend five minutes wading through menus, then after sitting on hold for ten or fifteen minutes the system says "I'm sorry, no one is available to answer your call right now, please try back later!" and hangs up on you. :argh:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Lum posted:

People (hopefully) aren't working on Saturday or Sunday either, so by that definition 3 days would be Thursday.

I thought the whole point of the post that started this one was that the 3 day deadline was unrealistic. I was just trying to point out that it was even more unrealistic than first stated as the poster thought they had until tuesday.

Exempt employees in America work whenever their company wants 'em to. Hell, Labor Day is just the first fifteen hours of my 70 hours of active monitoring for next week, which is on top of my normal 8+ hours a day of regular work plus a few extra hours of rebooting servers at 4AM next Sunday. :toot:

As for the OP, I'd guess some nice gentlemen from ICE or DHS are going to be stopping by next week and the company doesn't want to get fined. (Does it count as a :yotj: when your company puts down today as your date of hire on the official government paperwork they actually forgot to fill out a year ago? :v: )

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse
Oh god, dev vs. prod stuff. At one place, we had an ailing data analysis system that was godawful slow on standard storage, so we deployed a proof-of-concept base model flash memory solution just to see what sort of performance gains it might provide. The improvement turned out to be a little too impressive, because the top-level execs saw the performance on the new device (which had barely enough storage space for the current data set and literally no data or hardware redundancy on any level) and immediately said "Wow! Cool! That's the production system now." :cripes:

On the other side of the coin was one of our Peoplesoft platforms, which had nearly 20 separate non-production environments at one point. :psyduck:

Edit: Add me to the list of people still using good ol' Winamp 2. I'm one of those people who hates change and doesn't update software unless it's absolutely necessary, though, which is why my Windows 7 UI looks like Windows 95 with a slightly different shade of battleship grey and I use Magus's Greasemonkey script on SA. :v:

dennyk fucked around with this message at 00:10 on Sep 6, 2013

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse
So far today I've:

- Watched a sever load test for an hour or so

- Rebooted several production servers for kernel patching

- Fixed a server whose NFS exports from our SAN got un-exported suddenly

- Did some poo poo that I don't even know to fix a corrupted Lotus Domino names.nsf file (I don't know a goddamn thing about Domino, but our Domino admin isn't a Domino admin and also doesn't know a thing about Unix, so :welp: )

- Cleaned up some Oracle archive logs before a filesystem filled up

- Actively watched our monitoring system for 12 hours

Only four more hours of monitoring and a dozen more reboots to go, then I can try to get a few hours of sleep before getting up at 4AM to do more server reboots, and then assuming that doesn't break anything, maybe a couple more hours of sleep before I start another 16-hour monitoring shift. Then it's just five work days left till the weekend (when I'll be rebooting yet more servers, hooray... :toot: )

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

MF_James posted:

All our notes in our ticketing system COULD be viewed by clients, but none of them ever log into the system, but why would you be putting stuff into your ticketing system that you don't want someone to look at? Seems like a Bad Choice.

Yeah, never ever ever write anything anywhere in a ticket that you wouldn't want a client to see. Even if you have an internal note field that remains internal, that's no protection from some outsourced tier 1 tech copying and pasting all your internal notes and sending them to the customer because they're too lazy to write their own response.

Farking Bastage posted:

They have a PE2800 with 2 failed disks out of a 6 disk RAID 5 array. Work with Dell to figure out if it is recoverable. There are no backups

Are the two drives actually dead? If so, then you're talking serious data forensics poo poo that isn't going to be worth it. In my experience, though, PERCs of that vintage love nothing more than kicking random working drives out of an array when one fails (or sometimes for no reason at all), so there's a chance one or both of those drives might not actually be dead, in which case you might be able to force one back online and get the system up long enough to recover data. This has a better chance of working if both drives dropped at the same time; if one actually died long before the other one and no one bothered to replace it, then I hope you know which one's been dead for ages and which just dropped. :v:

As for database chat, my time in the web hosting industry has taught me that the more critical a web site's database is to a business, the more likely it is to be Access. With a FrontPage front end. :cry:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Helushune posted:

I couldn't agree more. Funny enough, we were discussing how useful/useless file extensions at work the other day and you could easily point out the *nix/mac users and those who had used Windows all their life. The argument the Windows users kept making was "well, how would you know what program opens them if it didn't have a file extension?" which seems completely pointless if you can just change them and have them open in a different application. I'm still convinced they're horribly useless and Windows should make every effort to move away from them beyond just hiding them by default.

You can also point out the people who spend their days maintaining and administering servers and the people who spend their days cleaning up after hapless Windows desktop users, evidently. :v: File extensions are just fine; they're an OS-agnostic method to identify a file type within the filename itself, which not only makes it easy to tell what files are when working with them manually, it makes it easy to create scripts to manipulate files based on their types without having to delve into some proprietary meta-information to figure it out.

The real problem is that Windows associates files with applications, including system-executed files like .lnk shortcuts and even .exe files, based solely on the file extension, which makes it easier to accidentally (or maliciously) screw something up. Filename extensions don't need to go away, Windows just needs to use an alternate method like file metadata or magic pattern matching (or even a combination of methods) to determine file associations, not just the filename extension.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

TWBalls posted:

What? Where do you work that allows allowed alcohol consumption while working? I've never worked anywhere where that was allowed. I always thought it was some Hollywood cliché, where they show an exec that has scotch in those fancy decanters in their office.

The dedicated hosting company whose office my old company squatted in for a year or two (after selling our data center and all our dedicated customers to them) had a keg in the break room. Don't know if they had any rules about when it could be used, but I never noticed anyone abusing it. If people are literally getting drunk at work (as opposed to having a beer with lunch or while working late occasionally), though, a no-alcohol rule is a good idea.

Also, I'm pretty sure once you get to the executive level, silly little office policies like "no alcohol" don't apply to you unless you want them to.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse
Speaking of Windows file extension associations, as simple as that system is, Adobe has somehow found a way to gently caress it up. A while back when I updated from Reader 8 to Reader X, the uninstall for Reader 8 didn't actually remove anything, and PDFs continued to open in 8. Trying to change the file association to Reader X manually would do absolutely nothing. Now after my last reboot, PDF files suddenly aren't associated with any program, and can't be associated with anything via Windows Explorer. Open With... is available, but it won't save associations, and it's impossible to choose the Acrobat Reader executable anyway (when you select it with Browse, it doesn't add it to the list of available programs). If you try to associate Reader X with PDF files in the Reader preferences dialog, it runs an installer and then forces you to reboot, but doesn't actually fix anything. Uninstalling Reader X, like Reader 8, doesn't actually remove anything, so I can't reinstall. Seriously, gently caress Adobe. :argh:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

MJP posted:

In order to get the contact information for where to submit this job, answer the following question:

Two yachts are 100 miles apart and travelling toward each other at a constant speed of 10 mph. A helicopter is flying between the yachts at a constant speed at 100 mph. Once it reaches one yacht, it turns around and starts travelling toward the other yacht. It does not loose any time or speed when turning around. How much total distance would the helicopter have traveled once the yachts meet each other?

If the helicopter starts at one yacht and flies towards the other, the answer is a little under 91 miles plus whatever altitude it was flying at plus the depth of the ocean floor at the point it meets the second yacht, because no helicopter exists that could execute a maneuver that is basically equivalent to a 200MPH impact with a fixed object without breaking into tiny pieces. :colbert:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

notwithoutmyanus posted:

What? No. That's what working from home is for. Sick days/day off are when you're stuck on a really tough level and need another 18 hours to deal with it.

Also, guess what I got? The most important certification of all! ITIL Foundation! :v: I'm pretty sure that means...nothing.

I think it means something, but I don't quite recall what because I purged everything I learned in that class from my brain by the day after the test.

I do remember that there are like four more levels of certification in ITIL that you can earn with a dozen or so other courses, though. :stonk: I think if you pass them all you become some sort of ITIL Demigod who can crush entire cities with your process management buzzword Thu'um or something.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

MJP posted:

Not to pull a Corvettefisher but while I should have done that, chances are I'd be written up for being unavailable.

If they wouldn't bring up guys who were shomer shabbos to try to poke holes in my story I would start thinking about claiming that.

I'm just going to have to say nothing and give the Resumetointerviews guy some time and money to clean up my cover letter.

I could go on about our part of IT being the red-headed stepchild but we all know that this is not news. (Is it still hoping for pods if we dream of a sysadmin department that can say no?)

Hate to say it, man, but after-hours maintenance on production systems is part of being a sysadmin. You aren't going to escape it unless you find a shop that has actual 24/7 staffing (and somehow avoid being stuck on the graveyard/weekend shift yourself because you're the new guy) or get a job someplace with a small IT footprint where there just isn't much that has to be done after hours (in which case you will be the entire department and will always be on call for everything 24/7 and will never have an uninterrupted vacation again).

13 hours does seem a bit excessive for firmware updates, though, unless you have thousands of servers and only two of you doing the work. Hell, even then you can script that poo poo and do a bunch of systems at once (or at scheduled intervals if you have stuff that can't be down simultaneously); it's not like you have to be doing them manually one at a time. How many systems are you going to be updating? It's also pretty lovely if your boss doesn't at least let you show up late/leave early for a few days the next week, even if they don't do "official" comp time. At least they gave you some notice instead of dropping it on you the Friday afternoon before, I guess. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

teethgrinder posted:

I never understood how a computer could be 99% stable with bad RAM, but gently caress off trying to install Windows to it. Happened to me several times.

A lot of people really don't do poo poo on their computer but check their email and maybe surf the web with a single browser tab, which doesn't use much memory. Even if it does happen to encounter uncorrectable errors from a bad DIMM, it will probably just cause the current application to crash, which most non-technical people won't think twice about unless it happens really frequently. I'd guess a full Windows install uses quite a bit more memory and is therefore more likely to have an issue with a flaky DIMM, and of course when it has a problem, the whole install dies.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Agrikk posted:

It would be totally awesome if that big old son of a bitch was running NetWare. Just because.

Also those DL380 servers were great boxes. We had a poo poo load of them and they ran like champions until we did a hardware upgrade. I ended up snagging a bunch of them and shipping a few of those out to the Goonfleet guys where I think they ended up powering the forums there and other infrastructure for a while.

Linked for size.

You kids and your fancy new post-HP-buyout DL380s. :colbert:



The web hosting company I used to work for had tons of these old Compaqs. To this day, some of their critical internal infrastructure still lives on them. To be fair, they are goddamn tanks; I once dropped one (gently caress Compaq rails forever) and it landed hard enough to bend the front faceplace (which is like a quarter-inch thick slab of solid steel), and after picking it up and plugging it back in, it still worked just fine.

That same company also, to this day, still has customers paying $50 and up for shared hosting accounts which are hosted on these ancient HP LPrs, running BSDi (yes, BSDi) 3.5 or some poo poo:



That company had all sorts of crazy hardware. They were too cheap to pay for any new hardware unless there was no possible way around it, so our provisioning room was a museum of server hardware from days gone by. They'd buy other hosting companies left and right and literally haul their systems to our data center and stand them up as-is rather than converting the customers to a single platform (since that would require buying hardware, and damned if that was going to happen when the company they just bought out already has all this stuff sitting around).

While we did retire platforms now and then, there were times when we were maintaining well over a dozen separate hosting platforms at once, all of which were on completely unique hardware and software platforms. At one point in time, we were responsible for managing customer-facing hosting systems running on three generations of Windows, several flavors of Linux, multiple versions of SunOS and Solaris, FreeBSD, BSDi, and Irix all at once. When they bought a company, they'd boot most of the acquired company's employees (and the rest would usually see the writing on the wall and bail within weeks), so we'd have to take over managing their systems, usually with little or no documentation or training. There were times when we first learned that we owned yet another new hosting platform when a tier one tech support guy called us about an issue with it. Expertise among us sysadmins was determined by lottery; whoever got assigned the first incoming support ticket or call about a new system (or an old system whose previous "owner" left) became the expert on that system forevermore, regardless of whether they actually knew anything about the platform or even the OS.

From what I understand, despite some major consolidation projects in the later years I was there, the place is just as crazy as always thanks to several more acquisitions. Apparently, in addition to a ton of newly inherited hardware and hosting platforms, they now have about a dozen separate CRM systems (several of which directly execute hosting account provisioning tasks through completely different middleware systems), none of which can talk to each other at all. :psyduck:

Edit: I think my favorite system of the many we inherited was the customer email hosting platform which consisted of custom POP3 and SMTP server applications written in Java running on ancient Solaris servers, which stored emails (including attachments) as data in an Oracle database and never deleted them (it would just set a flag on the email message in the database when the customer deleted it). Oh yeah, and the webmail interface was a ColdFusion application running on a Windows 2000 server. The system was still in use as of about a year ago, and as far as I know it's still running today. :cripes:

dennyk fucked around with this message at 04:52 on Sep 29, 2013

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

GargleBlaster posted:

Nothing has moved yet. This was about 4 months ago.

A company I worked at once hired some outside company to make a WordPress blog. That's literally all we wanted them to do, set up Wordpress on a server and make a simple little blog. Hell, I could have done it in an afternoon. They proceeded to do god knows what for months, then finally sent our team a request demanding that we install a bunch of crazy software on the Linux server they were using, including some weird remote desktop server software I've never heard of (the system in question didn't even have a desktop environment installed, because Unix servers don't need a goddamn desktop). We told them "No, you don't need all this, you're just installing Wordpress. You have Apache, you have PHP, you have MySQL, you have an FTP account. Now get to it."

They sat on their asses for another few months and did nothing. Then we fired them. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse
Whenever I use Dell chat, I open with several paragraphs of detailed technical analysis about the issue, what I did to troubleshoot it, and what I need them to do to fix the issue (which is usually just "send us a replacement part via courier"). Usually works fine, but the last couple times, this has resulted in about ten minutes of baffled silence followed by "Sorry sir you will have to please call our engineering department at XXX-XXX-XXXX for assistance with this matter thank you sir and have a nice day sir" :argh:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Varkk posted:

You don't get rich by paying out more than you absolutely have to for something. Although usually the ones ranting about the failure of a bottom dollar item costing them millions in lost business opportunities are full of poo poo and looking to blame someone else for their failing business.

The less money a small business actually makes, the more they will be losing during every moment of downtime. Judged solely on the metric of dollars claimed to be lost per minute when calling technical support, the American banking industry's financial clout has nothing on some dude's grandma selling homemade candles on her $5/mo shared web hosting site. (No joke, the place I used to work at literally had a bank hosting their site on our shared servers, and when their site was down, they gave no fucks. Meanwhile, some dude with a crappy little ecommerce site that gets like ten visitors a week would be calling every twenty minutes screaming about all his lost revenue.)

quote:

The stupid thing is any decent staff are going to care enough about their systems to make themselves reasonably available anyway. I do 1 week in 5 of 24 hour on call, for critical systems only (and not directly with end users ever) for which I get a monthly payment, plus overtime for any actual work I do and I can take that same amount of time off the next day. While on call I have to respond to a call within 30 minutes (so I don't even have to actually answer the phone) and I'm expected to keep an eye on my email for monitoring alerts which is just glancing at it once an hour or so while I'm awake.

Man, I wish my current job would do something like that. Our team has a monitoring rotation during which we have to play NOC and actively watch our monitoring system until midnight all week (and 9AM to midnight on the weekend) and be available instantly at all times to respond to alerts. That person is called the "on-call" person, but in reality we are all on call 24/7/365 and expected to work after hours for all emergencies affecting the systems we're responsible for, and for all planned changes (patches, system reboots, migrations, etc.) that have to be done after hours, in addition to our normal 40-45 hours of work during normal business hours every week. We get no overtime pay or comp time from the company for any of that.

Luckily my boss is cool and doesn't mind if we leave a bit early sometimes or need to take a day off here or there for personal reasons, because he knows how much work we put in. Some sort of official comp time policy from the company would be nice, though; pretty much all of the technical departments put in tons of after-hours work all the time and don't get anything for it, and not all of them have managers who will give them some leeway. :(

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

EAT THE EGGS RICOLA posted:

I have firstinitiallastname@gmail.com, with my last name being pretty common, and I get probably 5-10 emails a week. Loads of sensitive medical/financial stuff.

I get other people's credit card statements at my personal email account, in addition to all kinds of random personal and business emails for various people who aren't me. That's what we get for being early adopters, I guess. :v:

Apparently the elderly and otherwise technically non-inclined just assume that if their name is X then X@gmail.com must be their email address. :downs: I wonder if these people ever wonder why they never get the emails they sign up for?

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

I believe I've found your problem here. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

KillHour posted:

It's really... odd actually. The cameras are pointed at digital signage. The company needs a way to prove that the displays are showing what they're supposed to when they're supposed to, so they are recording all 10,000 displays they have, and having the video compared against what SHOULD be displayed there. They want to store the data all at the same place so they can have people "forensically analyze" the feeds (their words, not mine) in their NOC to make sure everything's correct. It's basically one of the craziest requests I've ever had.

So did this come about because one of their execs saw one of those pictures of hacked electronic road signs on the Interweb and is now in a panic about the Chinese taking over their signage network to spread the evils of Communism, or because one of their disgruntled underpaid graphic designers drew Dickbutt on a client's electronic billboard ad? :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Agrikk posted:

My mind is boggled at the :stinkeyes: I get when I raise opposition to "make the intern do it" as a viable solution to patching and rebooting a hundred fifty production windows VMs.

Ain't that what WSUS is for, anyway? Or whatever they call WSUS these days.

Hell, I have to patch kernels and reboot a hundred and fifty Linux systems once every few months; every single one has to be manually coordinated with the department that owns the application on it and each of them has to be rebooted at a different after-hours time on a different day of the week for business reasons. :sigh:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

QuiteEasilyDone posted:

"So you're fine with this computer failing on its own terms and bringing your entire assembly line to a halt for however long it takes to to repair or inevitably replace when it catches fire one of these days. Gotcha"

Of course they are; if they decide to spend money and take downtime to replace it now, then the resulting loss of profit is their fault. If the computer dies, then it's the IT guy's fault, and management gets bonuses for successfully minimizing the impact of a difficult unforeseen emergency by yelling at the IT guy to fix the problem that he created. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse
That Penny Arcade job is pretty much the solo IT position at every small company with an online presence ever, so it's not really that unusual. They're just being more up-front about it instead of completely hiding it behind buzzwords. There are folks out there who do like that sort of intense work and don't mind the hours. That said, the experience they're asking for is pretty absurd; I doubt they're going to find anyone with that much experience in all those separate fields who would be willing to work four jobs for what is probably an entry-level salary. They really should be looking for another young guy without much experience who can learn quickly and doesn't mind spending a couple years being underpaid in exchange for what is probably some decent experience with various IT work, like their current admin when he started.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Antioch posted:

It's not Nagios, it's the barely literate dipshit overnight 'ops' guys that cause me grief.

Schedule recurring downtime or change the check period on that server so it doesn't check it when it's supposed to be rebooting, then it won't even show up on the Nagios screen and ops won't even know it's down.

I really wish we just had on-call weeks in our place. Instead, each person on the sysadmin team has to play NOC for a week and actively watch the Nagios monitors from 9AM until midnight for seven days and either fix stuff when it alerts or call whoever the server owner is if it's not us, while still working our normal hours as well. It kind of sucks to go seven days in a row without being able to stop thinking about work for more than a few minutes at a time. (Counting monitoring time I've worked 65 hours so far this week, only about forty to go... :toot: )

Also, I just found out I will be spending the Saturday night before my birthday shutting down and starting up servers for twelve hours instead of having a nice dinner like I'd been planning on. :welp: The constant weekend work here is honestly starting to annoy me a bit, since we don't get any comp time for it.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

crashdome posted:

Well, technically, he/she WAS asking for it.

Hope sfwarlock put in a change request to apply Hug to User and got all of the required management approvals, or he's screwed.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

blackswordca posted:

So an email came in.

"We are extending our hours from 8am - 5pm to 6am-6pm. This is starting tomorrow, shift changes will be posted shortly"

Our sysadmin team already has to babysit our monitoring system from 9AM to midnight seven days a week. Today our boss mentioned that management wants to have someone actively monitoring things 24/7. He's trying to convince them to just hire some drat NOC people already, but I'm not sure they're going to bite, which means it'll probably end up in our laps. There's about ten people on our team, but everyone has different roles and there isn't much overlap, so whatever monitoring hours they stick us with will be in addition to our normal ~45 hours a week of actual work just like they are now, I'm sure, except they'll have to have at least two people doing it every week, so we'll end up on monitoring and have no time off at all for a week almost every month, and have to shift our sleep schedules around all the drat time in the process. :(

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Migishu posted:

A vacation came in...

Time to do jack poo poo for a week and a half.

In our weekly meeting this week our boss told those of us on going on vacation to remember to check our email at least three times a day. Guess it's a shame I won't have a computer or Internet access and my phone doesn't have my work email on it... :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

tomapot posted:

A little late to the degree talk but wanted to chime in. Whenever I feel less-than for my HS diploma and tech school cert I remind myself that 1) I did not sink a ton of money into my education and still landed into a sweet career and 2) that my boss was a theater major and the guy I now manage was a stage magician so who the hell knows about the paths our lives take.

IT, especially the administration side of things, is one of those fields where a degree doesn't really matter compared to your actual experience. Hell, the top sysadmin at the first place I worked had an art degree from SCAD. I've only got a liberal studies BS myself, but I started my first job while I was still in college and stayed at the same place for 11 years, going from phone jockey to sysadmin.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

rob_squared posted:

Am I too late for the "working with people with famous names" thing? If not, I work with a Tom Cullen.

Every time I see an email from them I think "m-o-o-n, that spells Tom Cullen."

I work with a John Bunnell, which always makes me think of this guy:



However, while he isn't Sheriff John Bunnell, he is the John Bunnell who co-founded DragonCon, so I guess that counts for something. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

A c E posted:

I too can't type properly. I've tried a few times to retrain myself but it's a hard habit to shake. I don't ever use the CAPS key though, I will hold shift if I need something all in caps because caps lock is the devil. I usually disable the key entirely because I hit it accidentally while gaming all the time.

I only use my index fingers and thumbs while typing. Occasionally I throw in a random other finger to assist. Despite this I can still type at 80wpm and don't need to look at the keyboard at all while typing (except if I am stuck with one of those ergo keyboard with the split, those gently caress me up). I get comments about it all the time.

Heh, this is me, except I only use my middle fingers. :v: Never could learn to touch-type the right way, for some reason. I can't go quite as fast as some folks, but it's plenty fast for the work I do, so I don't care.

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Caged posted:

Who's job was it to put the working hours into your ticket system?

You're assuming that "working hours" isn't a redundant term at his job. :v:

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

myron cope posted:

We aren't in any kind of trouble over it (especially not me, since I just started in mid-November), it's just a policy change going forward.

Three months from now if it's still happening, I could see it being an issue for us. I'm just going to ticket everything no matter how dumb it is.

To be fair, this isn't as dumb as it might seem. Opening a ticket for every issue, even minor ones, is actually helpful for a number of reasons: it lets you track trends that could indicate a bigger root problem, it provides documentation of what was done to fix issues (something that seems "minor and stupid" to you might not to a new guy who hasn't seen it and fixed it a hundred times...or to you when you're trying to fix the same problem while half-asleep after being awakened at 3AM), it provides a record of who did what to which systems at what time if you don't have a proper change management tool that you use for every change no matter how minor, and, of course, it clearly shows your management team how much work you're doing and whether that workload is increasing or decreasing. It may seem annoying to have to spend a minute or two creating a ticket for a thirty-second fix, but it's usually worth the pain, unless your ticket system is literally so broken as to be useless for anything but basic metrics.

Adbot
ADBOT LOVES YOU

dennyk
Jan 2, 2005

Cheese-Buyer's Remorse

Belial42 posted:

The rumor floating around is that someone asked for EPO testing procedures shortly before the power was lost. Wonder if the cameras will pick anything up.

This happened in our old datacenter once; someone mistook the EPO button for the button that unlocks the door. Of course, this was over a weekend, so we all had to spend the entire weekend powering everything back on and fixing all the broken crap.

I guess the upside is that our inadvertent EPO test revealed that the button only cuts power to half the data center. :v:

  • Locked thread