|
skooma512 posted:My lead got put on heartbeat monitoring like 10 years ago. If something missed a ping or something it would alert him. https://en.wikipedia.org/wiki/Alarm_fatigue
|
#
?
May 6, 2016 02:03
|
|
|
|
| # ? May 14, 2024 20:46 |
|
|
Inspector_666 posted:Doesn't mono just make you sleepy and you lose a bunch of weight? Between that and everybody knowing you're out there kissing folks it seems pretty great! I wish it meant I was out there kissing people. It probably meant that some restaurant that I went to didn't clean a glass or fork properly and I got it that way. Or however else it works. Sadly, at 10 I was not yet kissing girls.
|
|
#
?
May 6, 2016 02:05
|
|
|
After 15 years at this job, we finally decided to direct all mail to root to /dev/null this month. Makes me a little nervous, but change is good and we have a really well implemented nagios install so it should work out okay. Not getting a couple dozen random insta-trash emails every day is a little disconcerting, a quiet inbox makes me wonder if email is down.
|
|
#
?
May 6, 2016 02:10
|
|
|
I think my iPhone is dying. A couple of weeks ago it stopped launching new apps (loading image would pop up and then it would crash back to the home screen), had to boot it to recovery and restore the OS because it wouldn't power back on after I shut it down to see if a reboot would fix things. Now it's doing it again  . Could do without having to replace it right now but can't really afford to be without it either.
|
|
#
?
May 6, 2016 02:53
|
|
|
Who the hell alerts on a single ping drop? That's the dumbest poo poo I've ever heard. At least setup the alert to only fire off if the condition is sustained for X amount of time/polls.
|
|
#
?
May 6, 2016 02:53
|
|
|
xzzy posted:Not getting a couple dozen random insta-trash emails every day is a little disconcerting, a quiet inbox makes me wonder if email is down. I get you, every now and then we talk about a better solution than having all cron jobs everywhere email their results to us. Services exist that can grab those and put it all into a nice Node.js page, but we would end up ignoring it, I just know it. The thing is, of the ~100 cron emails I get every day, every now and then we do spot a valid error that needs to be fixed (ex. a backup rsync, the host ssh key changed, backup job fails) so it IS useful to check them. But I usually do with my finger on the delete key, just watching for patterns different from previous days that my morning brain can catch.
|
|
#
?
May 6, 2016 04:07
|
|
|
Senso posted:I get you, every now and then we talk about a better solution than having all cron jobs everywhere email their results to us. Services exist that can grab those and put it all into a nice Node.js page, but we would end up ignoring it, I just know it. The thing is, of the ~100 cron emails I get every day, every now and then we do spot a valid error that needs to be fixed (ex. a backup rsync, the host ssh key changed, backup job fails) so it IS useful to check them. But I usually do with my finger on the delete key, just watching for patterns different from previous days that my morning brain can catch. Sounds like a good monitoring tool will pay for itself just in time saved.
|
|
#
?
May 6, 2016 04:40
|
|
|
Pipe a digest message to Slack via a web hook.
|
|
#
?
May 6, 2016 11:12
|
|
|
This sums up the homebrewed monitoring at the E911 vendor I used to work for. Our system would check each market for a "good" E911 call in the last 15 minutes, and if there was nothing it would throw an alarm. Guess what happens overnight in really small markets? Nothing, so that poo poo would be going off non-stop. From about 11PM until 7-8AM (for West Coast/Hawaii) 5-15 markets would be alarming every 15 minutes for each check... so it would be non-stop alarms going off every few minutes. Meanwhile, if you missed a single alarm that was an outage, the director of the department would have to go upstairs and justify not firing you to the board. Spazz fucked around with this message at 11:54 on May 6, 2016 |
|
#
?
May 6, 2016 11:49
|
|
|
What would your on-call guys do in those situations? Make a test call and verify it went through?
|
|
#
?
May 6, 2016 12:05
|
|
|
Hahaha on call guys? No you just sit there tailing log files and checking the system health with your butthole clenched firmly until a call comes in. My record for no calls with a perfectly functioning system (even after checking with the carrier) was 14 hours. Normally there were ways to tell if there were problems -- SS7/SIGTRAN links down or flapping. SS7 let you actually peek at the data and you could look at the signal and determine if they were sending anything or not, or if you were getting zero signal. I can't remember all the terminology since it's been so long. You could also look at the individual cell towers and whether or not they were up, and a certain number of towers in a geographic area counted as an outage. If you had a cluster of say, 15 towers go down all at once, but they were dispersed, it would be fine. If you had 15 all go down in the same area, that was an outage and would be FCC reportable. You started to learn the problem markets -- Hawaii being the biggest one because there's so little crime (or at least going unreported). When your big markets (NYC/LA/Chicago) start throwing alarms, you call the carrier and have them confirm with the MTSO that things are healthy. E911 is interesting because it's market to market. Some of the guys at the MTSO are friends with the local PSAP so they can do test calls, and in the middle of the night its slow so they are usually OK with a test call. Other ones, like near major metropolitan areas (but the carrier doesn't have strong coverage) they will send the police to the MTSO and write you a citation for improper use of emergency services if you don't get clearance during the daytime to do test calls. That only happened once, normally it would just be a nastygram though. Ahhh the days of working in an industry with government mandated uptime...
|
|
#
?
May 6, 2016 12:21
|
|
|
At that point they really should get a NOC guy for the overnight.
|
|
#
?
May 6, 2016 16:04
|
|
|
I was more getting at the process for confirming that these false positives are indeed false positives (presumably you do that rather than just ignoring them), and why can't whatever is doing the monitoring use that process instead. Throwing alerts because a thing hasn't happened for a while instead of monitoring the underlying links just seems really backwards.
|
|
#
?
May 6, 2016 16:08
|
|
|
I know why it's done, but I loving hate having to fill details in to "get a quote" for an idea of product pricing. You've got a standard set of costs for whatever you sell, just loving tell me so I know if it's wildly outside my budget before I even bother continuing. I often find I won't even bother to go further and just keep looking.
|
|
#
?
May 6, 2016 16:23
|
|
|
skooma512 posted:At that point they really should get a NOC guy for the overnight. It was a 24/7 department that managed and provided support for the equipment, so there was almost always someone in the office. I worked overnights for the better part of 3.5 years, either 7PM to 6AM Mon-Thu or 10:30PM to 7:30AM Sun-Thu. That also included 11-11 weekend shifts... which means sometimes you'd be doing 2-11PM Thursday and then have to do 11PM to 11AM Friday night. I do not miss this place. Thanks Ants posted:Throwing alerts because a thing hasn't happened for a while instead of monitoring the underlying links just seems really backwards. Preaching to the choir.
|
|
#
?
May 6, 2016 16:24
|
|
|
LordVorbis posted:I know why it's done, but I loving hate having to fill details in to "get a quote" for an idea of product pricing. You've got a standard set of costs for whatever you sell, just loving tell me so I know if it's wildly outside my budget before I even bother continuing. I often find I won't even bother to go further and just keep looking. I've never gotten an explanation for this that made sense: Why is it done?
|
|
#
?
May 6, 2016 16:24
|
|
|
It's because the product is really expensive and/or comes with massive initial setup fees, so instead of giving you a price the vendor wants you to sit through an hour long webinar where they can talk a load of poo poo and then act like they've gotten the deal at the end of it and send over a quotation. Then they will never stop calling you. Even if you're up-front with people they try this - I have told vendors before that I had a 20k budget for something and they turned around with a final price of like 90k and acted like the 60k discount that had already been applied somehow meant it was within budget. And then when you finally turn them down they get upset at you wasting their time.
|
|
#
?
May 6, 2016 16:28
|
|
|
Yeah, they just want their sales force to get their hooks in you. Can't do that if there's a price sheet on their website.
|
|
#
?
May 6, 2016 16:40
|
|
|
Thanks Ants posted:It's because the product is really expensive and/or comes with massive initial setup fees, so instead of giving you a price the vendor wants you to sit through an hour long webinar where they can talk a load of poo poo and then act like they've gotten the deal at the end of it and send over a quotation. Then they will never stop calling you. xzzy posted:Yeah, they just want their sales force to get their hooks in you. Can't do that if there's a price sheet on their website. That still doesn't make sense because I haven't met someone who went "A price request form? I'll just fill this bad boy out!" Even C-levels consider that sort of thing BS and just don't bother and go elsewhere to get pricing for whatever it is they are pricing. Although, I have gotten the "well, we have wasted all of this time putting together this quote...." thing, before. I laughed at a sales guy's face over that blatant attempt at emotional blackmail. Like I'll go "OMIGAW, you had to do your job?! That's terrible, man! I'll buy 140 seats!"
|
|
#
?
May 6, 2016 18:13
|
|
|
Arsten posted:That still doesn't make sense because I haven't met someone who went "A price request form? I'll just fill this bad boy out!" Even C-levels consider that sort of thing BS and just don't bother and go elsewhere to get pricing for whatever it is they are pricing. I don't get it either but I assume it's done by people who did a sales training course. It's sort of fading out as more and more stuff becomes commoditized partly thanks to  In news that doesn't piss me off: feels loving good to get my last Sonicwall out of production. Awful products, worse support.
|
|
#
?
May 6, 2016 18:24
|
|
|
Judge Schnoopy posted:Back in my MSP days we constantly theorized a device that could automate this inefficiency (because so many businesses refused to fix the problem the right way). I wonder how long you could get away with selling this device if it didn't actually print anything. Just make it a giant display to pacify idiots who think printing to a pdf is too hard or carries some sort of risk. I'd bet you could buy an island to live on before the fraud charges were finalized.
|
|
#
?
May 6, 2016 18:43
|
|
|
Promote it as a way of passing documents from a lower security network to a higher security one, have separate NICs for the printing and scanning part, charge installation fees and huge support costs, only support proprietary paper, make the shredding bin unopenable except to your own waste collection staff who are also expensive.
|
|
#
?
May 6, 2016 18:48
|
|
|
Thanks Ants posted:Promote it as a way of passing documents from a lower security network to a higher security one, have separate NICs for the printing and scanning part, charge installation fees and huge support costs, only support proprietary paper, make the shredding bin unopenable except to your own waste collection staff who are also expensive. You also need to make a display of burning it and separating the ashes into 2 separate bags to be disposed of in different cities.
|
|
#
?
May 6, 2016 18:49
|
|
|
That would cost extra and require a separate proprietary rack placed next to the original object.
|
|
#
?
May 6, 2016 18:50
|
|
|
Arsten posted:Although, I have gotten the "well, we have wasted all of this time putting together this quote...." thing, before. I laughed at a sales guy's face over that blatant attempt at emotional blackmail. Like I'll go "OMIGAW, you had to do your job?! That's terrible, man! I'll buy 140 seats!" "Yea, man. I feel ya. I hate when people waste my time, too. Y'know, if there'd been a price list on the website,I coulda saved both of us a lot of time and hassle. Oh well, gotta do what you gotta do."
|
|
#
?
May 6, 2016 19:24
|
|
|
Sorry you spergs have to talk to people to put together a quote. I'll point my EMC and IBM reps to this thread, I'm sure they'll post a price list ASAP.
|
|
#
?
May 6, 2016 20:52
|
|
|
Arsten posted:I've never gotten an explanation for this that made sense: Why is it done? To be none other than annoying assholes, especially when they call you every single day even when you point blank tell them money is going no where stop calling me. No price list is an instant ignore, if I don't know how much your poo poo costs you may as well not even exist.
|
|
#
?
May 6, 2016 21:03
|
|
|
Super Slash posted:To be none other than annoying assholes, especially when they call you every single day even when you point blank tell them money is going no where stop calling me. Wizard of the Deep posted:"Yea, man. I feel ya. I hate when people waste my time, too. Y'know, if there'd been a price list on the website,I coulda saved both of us a lot of time and hassle. Oh well, gotta do what you gotta do." Wake_N_Bake posted:Sorry you spergs have to talk to people to put together a quote.  And I don't mind talking to people to put together a quote when I need to, but when you do something almost insanely straight forward - like provide hardware - or an application that should have a relatively basic OOB setup, making me contact you to go "Yeah, it's $600 a seat, but I won't tell you until the end of an hour-long powerpoint put together by my kindergartener while I go 'uh....um' every three seconds while I pretend that I understand your business after reading your website for eight minutes three weeks ago." means that I don't want to even bother. Because the conversation invariably goes like this after the nails grind on a chalkboard for an hour: "$600 per hour? My budget tops me out at $150 per hour." "Oh, there's no way we can do that." "Well, have fun with life, then. Bye."
|
|
#
?
May 6, 2016 21:35
|
|
|
Arsten posted:
I was almost blown away by my interaction with Nimble Storage the other week - yeah, I had to send in an initial email and do a 15 minute phone call with the sales guy, but he sent over a quote an hour later, as he promised, with partner pricing no less, and sent me the spreadsheet of the other products since I'd asked about the other options. I mean don't get me wrong, in an objective sense it's still loving sad that I couldn't just get the information without interacting with them, but compared to every other SAN vendor I've dealt with it was piss easy. (Equallogic used to have their prices available on Dell Premier, but then they made it so it was only some "fast-ship" options and then they removed them entirely and required you to contact their sales, and it was like "idiots, the only reason we put up with your disadvantages over NetApp and EMC like your insane 16 MB snapshot blocksize was because you didn't give us too much poo poo purchasing the products, so how about you gently caress off"). Anyway yeah in general if it's simple poo poo, and a SAN should be simple poo poo because I'm paying a huge premium over a generic Dell server precisely so you can make things easy and simple and abstract away all the scary design issues around networking and storage, the purchasing should ALSO be simple. I'm constantly shocked at how many companies treat the customer from the get-go as an enemy who has to be forcibly separated from his money. All the hard-sell does is make me reconsider whether the product is worth the hassle of obtaining it, and guess what, in most cases there's a competitor's product that's almost as good, so if I can get that without the smarmy sales rear end in a top hat, that's what I'm doing. Yes Solarwinds, that means you.
|
|
#
?
May 6, 2016 23:15
|
|
|
I tried to buy some Dell storage a few weeks ago and it was insane just how few fucks they give about answering questions in a timely fashion/at all. I mean we ended up with NetApp and I feel we dodged a bullet but Dell almost overnight went from cheap(ish) storage that worked, didn't have too many features but was easy to buy and shipped fast to almost impossible to work with in the space of a few months.
|
|
#
?
May 6, 2016 23:18
|
|
|
Thanks Ants posted:I tried to buy some Dell storage a few weeks ago and it was insane just how few fucks they give about answering questions in a timely fashion/at all. I mean we ended up with NetApp and I feel we dodged a bullet but Dell almost overnight went from cheap(ish) storage that worked, didn't have too many features but was easy to buy and shipped fast to almost impossible to work with in the space of a few months. dell storage is poo poo. you dun good son.
|
|
#
?
May 6, 2016 23:21
|
|
|
Netapp can be pretty dodgy too unless you're content to drive a dumptruck of money to their door every few years. "Oh that filer we talked you into buying that does double the iops you said you needed? Well it's end of life so if you want support you're gonna have to buy a new one." But at least their solutions Just Work(tm).
|
|
#
?
May 6, 2016 23:29
|
|
|
DigitalMocking posted:dell storage is poo poo. you dun good son. What's so bad about Dell? We just installed 2 new setups in different datacenters, 3 vmhosts each (dell servers), a separate backup server (also dell I believe), and a SAN that (I believe) is also a Dell something or other. I didn't do purchasing or speccing for anything, and the SAN was setup by another guy, I've been too busy to really look at it and see what the deal is, but again, I believe it's a Dell of some kind....
|
|
#
?
May 6, 2016 23:34
|
|
|
Is there any storage vendor that doesn't do that? Hit them at quarter/year end for a trade-in deal on your old unit and you should do alright. Otherwise you can dangle the competitive upgrade carrot in front of a different vendor. Granted this is not the straightforward simple purchasing experience that we yearn for but I try and make things relatively easy for myself.
|
|
#
?
May 6, 2016 23:34
|
|
|
MF_James posted:What's so bad about Dell? We just installed 2 new setups in different datacenters, 3 vmhosts each (dell servers), a separate backup server (also dell I believe), and a SAN that (I believe) is also a Dell something or other. I didn't do purchasing or speccing for anything, and the SAN was setup by another guy, I've been too busy to really look at it and see what the deal is, but again, I believe it's a Dell of some kind.... I have no issues with their servers and the recent iDRAC firmware update to enable HTML5 remote console is awesome. Their storage range just seems to be totally without direction - they have the original Dell MD3 series which competes with HPs MSA, then they have EqualLogic and Compellent as well as FluidFS boxes to turn EqualLogic SAN into unified boxes. It's quite difficult to get to talk to anyone about the EqualLogic and Compellent products and their channel partners don't really seem too keen to push them either, and the MD3's USP was 'cheap and works and is easy to buy', but the price went a bit crazy after NetApp bought Engenio. I've not had a bad experience with it but the marketplace is full of people who make storage that isn't poo poo and Dell just don't really have a reason to look at them. Edit: gently caress, did not mean to double-post this.
|
|
#
?
May 6, 2016 23:48
|
|
|
Potato Alley posted:Yes Solarwinds, that means you. A while ago, I was involved in a bake off for monitoring 75,000+ nodes and managing about 10,000 those. Plus network health monitoring and basic analytic visualization. The bake-off was to select which suite was going to be used by both the NOC and SOC for monitoring, management, and event investigation (for the SOC). Turns out their scaling is poo poo and would require an insane number of servers (yeah VMs, whatever. I'm still tying up way more resources than I do with the big boy tools), and it's a bitch and a half to add any MiBs not in their "MiB DB" the program uses. The only way I could get some of poo poo to work was creating a manual translation table for the OIDs I was interested in, which is a HUGE pain in the rear end. I know the other network and security people working the project were fed up with the lack of flexibility. IBM and HP were able to come in and set up their POC instance in less than 5 days. THREE WEEKS later SolarWinds was still fighting us every step of the way. The sales guy kept insisting we try to find a way to make it work with their engineers. The thing is, both myself and the engineers knew we were wasting time. This guy was so pushy, I ended up telling him that I no longer want to be contacted in any way by his company. The sad thing is, while not suitable for our large project, their product would have fit some smaller stuff we have in the pipeline. But not anymore, I refuse to even entertain them as an option for now.
|
|
#
?
May 6, 2016 23:56
|
|
|
flosofl posted:A while ago, I was involved in a bake off for monitoring 75,000+ nodes and managing about 10,000 those. Plus network health monitoring and basic analytic visualization. The bake-off was to select which suite was going to be used by both the NOC and SOC for monitoring, management, and event investigation (for the SOC). This is actually really interesting - I haven't had occasion to monitor that size network but for whatever reason I assumed SolarWinds could handle large installs. I probably assumed that because their poo poo costs a hefty sum - sounds like in fact it's a small to midsize solution that they will push wherever they can because that's how their sales works. And yeah, the pushiness of their sales team is legendary at this point I think. "Don't contact me anymore". <month later> "Hey it's Mark from SolarWinds, just checking in to see if we can do something before end of quarter here!" <repeat for the next two years before finally giving up>
|
|
#
?
May 7, 2016 00:12
|
|
|
The way I deflect sales calls is to tell them I'm headed to a meeting and will call them back later. Then I never do. After three or four attempts the calls stop.
|
|
#
?
May 7, 2016 00:18
|
|
|
Potato Alley posted:This is actually really interesting - I haven't had occasion to monitor that size network but for whatever reason I assumed SolarWinds could handle large installs. I probably assumed that because their poo poo costs a hefty sum - sounds like in fact it's a small to midsize solution that they will push wherever they can because that's how their sales works. Well, they can but it's something like 10,000 nodes per instance. After that you have to start clustering. Which is really a bunch of instances reporting in to a "master" instance. Don't get me wrong. As long as you have a small to medium environment and use no niche or less popular equipment, I think it's a fantastic tool. We have a couple instances of their config management tool in a few isolated environments and it's a dream to push and revert config changes to 100s of devices at a time. And honestly, our needs were a little unique, since the vast majority of those are APs (this solution was for WLAN management and security). We actually manage them through controllers and the controller reports status, pushed the configs, but the infuriating thing is the APs *still* require a node license if we want to have them displayed in Orion. Even though we're technically doing config and monitoring through the controller, it still counts. I can't be too harsh about the licensing, since HP NNMi and IBM NetCool require the same. What sucks is the scaling. I think for Orion we'd need something on the order of 8 servers for Orion and another for the MS SQL DB, and yet another if we wanted to do any log aggregation. We ultimately ended up with NetCool, while SUPER expensive has stellar support, and I think we have 4 or 5 (I'm not the admin for the tools) 1 for the object server, 1 for NetCool "brains", 1 for the Bridge Server, and 1 for MS SQL. It scales well, and you can just throw storage, memory and cores at it to expand. You do need a NetCool admin, and people with that skill set are not cheap either. The other thing is I've never come across a tool that didn't allow you to straight up import a MiB it didn't have native to itself and with a couple tweaks be up and running. They had our stuff, but using a MiB from a major revision ago. And by major I mean the version that MiB is for is *nothing* like the current version of the device (plus it's well over 6 years obsolete). Yep, no dice with SolarWinds. They got engineers on the phone with me, but flat out said they would submit the MiB to the dev team, but they would not to commit to it being integrated into Orion. So it boiled down to trial and error OID mapping by creating a custom db for Orion. It felt very mid-90s tool for that. Fortunately, upper management was 100% behind this project and we ended up spending significant capital on NetCool. But I'll admit to being disappointed with SolarWinds. It was their bid to lose and they lost hard. In retrospect, I'm pretty glad we went the direction we did. I'm currently working on ways to integrate and expand our SOC using NetCool as more than just a SEIM, but a full featured analysis and investigation tool. NetCool makes it easy to silo SOC needs away from NOC and the poo poo this thing can do is making me revise my SOC schedule in a very good way. Proteus Jones fucked around with this message at 03:52 on May 7, 2016 |
|
#
?
May 7, 2016 03:47
|
|
|
|
| # ? May 14, 2024 20:46 |
|
|
MF_James posted:What's so bad about Dell? We just installed 2 new setups in different datacenters, 3 vmhosts each (dell servers), a separate backup server (also dell I believe), and a SAN that (I believe) is also a Dell something or other. I didn't do purchasing or speccing for anything, and the SAN was setup by another guy, I've been too busy to really look at it and see what the deal is, but again, I believe it's a Dell of some kind.... Just not a fan. I've been burned by Dell storage, especially after they bought equillogic and gutted their once great support and hosed up their next 3 major firmware releases. I find shopping their storage to be a huge annoyance since they've got poo poo all over the place in their offering without rhyme or reason.
|
|
#
?
May 7, 2016 06:30
|
|