|
the current *40 series track touch whatever thingy is honestly the worst loving interface i've used in my entire career
|
#
?
Feb 12, 2015 04:22
|
|
|
|
| # ? May 8, 2024 10:28 |
|
|
Just wanted to give a shoutout to Logstash (and ElasticSearch + Kibana) for being awesome. I've wanted to set it up forever, and the past couple weeks I finally had time to do a reasonable POC. Today alone I found and fixed two fairly major production issues just while loving around in my sample data. The amount of visibility it gives you and the freedom to ask arbitrary questions and get answers back in real time from logs scattered across 1000 servers is pretty amazing. As one example, I did a search for "number of requests per second to <service>, broken down by data center". All of the traffic was going to one data center. Whoops! Our CDN was misconfigured and only knew about one data center. Fixed that, ran the same query a few hours later and saw an even split as expected. So cool. If you don't have a central logging solution, and your volume exceeds Splunk's free license, definitely give Logstash a look. Only downside: I now spend all my time staring at it like the code from The Matrix. evol262 posted:Hey, no, it isn't. Getting a foreign work visa is all about talent and desire. Going to a "code bootcamp" is not more likely to get you offers abroad. And "just moving to London and applying for jobs there" is going to put you in a tricky visa situation. Get good at your job in the US and build some skills people may want abroad, then apply for those jobs or get sponsored by your current employer. Don't go about it backwards because you really want to leave the US. London is expensive. Would you "just move to New York and apply for jobs there"? Or SF? It's the same cost of living. This is a pipe dream. I know working overseas has come up before, so sorry if this is a rehash. My wife and I have talked about moving abroad for a few years. Either Europe/UK or Australia/New Zealand. If I were to work for a large company in the US that I know for a fact has people in identical jobs in other countries, how viable is it to work there for a time and then apply for a transfer to a similar role in another country? I assume it varies widely by company and I'd ask about it during the interview process. But wondering if anyone has past experience with that. Just curious. I have no immediate plans to do this, or leave my current job at all for that matter.
|
|
#
?
Feb 12, 2015 04:44
|
|
|
Yeah, having used other central logging solutions that store everything in DBs and take forever to return results, ElasticSearch+Logstash+Kibana is amazing! loving it.
|
|
#
?
Feb 12, 2015 05:26
|
|
|
The company I worked for previously moved to Dell laptops specifically because of that trackpad being so bad. v V
|
|
#
?
Feb 12, 2015 05:36
|
|
|
22 Eargesplitten posted:No, I have that exact same issue on my thinkpad. It's very minimal, but if, for example, I'm rapidly clicking on a small button, it will shift enough to make me miss it. Click and dragging is also pretty awful with it. Not unusable, but terrible because they had this poo poo figured out for so long that they really don't have an excuse for shifting to a worse design. 
|
|
#
?
Feb 12, 2015 05:39
|
|
|
I have been getting a poo poo-TON of web advertisements for Elasticsearch and ELK stacks. Can someone give me an idiot explanation for what they actually are because some Google algorithm has decided that I NEED to hear about them.
|
|
#
?
Feb 12, 2015 05:58
|
|
|
Dr. Arbitrary posted:I have been getting a poo poo-TON of web advertisements for Elasticsearch and ELK stacks. Can someone give me an idiot explanation for what they actually are because some Google algorithm has decided that I NEED to hear about them.
|
|
#
?
Feb 12, 2015 07:46
|
|
|
Docjowles posted:I know working overseas has come up before, so sorry if this is a rehash. My wife and I have talked about moving abroad for a few years. Either Europe/UK or Australia/New Zealand. If I were to work for a large company in the US that I know for a fact has people in identical jobs in other countries, how viable is it to work there for a time and then apply for a transfer to a similar role in another country? I assume it varies widely by company and I'd ask about it during the interview process. But wondering if anyone has past experience with that.
|
|
#
?
Feb 12, 2015 09:11
|
|
|
Docjowles posted:I know working overseas has come up before, so sorry if this is a rehash. My wife and I have talked about moving abroad for a few years. Either Europe/UK or Australia/New Zealand. If I were to work for a large company in the US that I know for a fact has people in identical jobs in other countries, how viable is it to work there for a time and then apply for a transfer to a similar role in another country? I assume it varies widely by company and I'd ask about it during the interview process. But wondering if anyone has past experience with that. I know it's certainly easier to do it in that direction (departing the USA) than migrating to the USA. We've had devs that have wanted to move to the US and continue to work for us there, but couldn't get the 1099 (Others were successful at earlier times). They ended up working in our UK office for the US team instead! I can't actually recall any existing staff we have making the move to NZ though, but maybe just none have applied, or it's happened and I'm just unaware of them. We do however have systems in place for new hires that happen to be overseas and want to work in NZ. It's typically a huge investment (from both employer and employee) to go to those lengths however, so it's only done if we believe it's worth it. With regards to your logstash setup, how much time and effort did you spend on setting it up? How much do you foresee in ongoing maintenance? We currently pay a fortune for splunk, which works really well for us, but doesn't mean we shouldn't consider other options from time to time. We do have splunk doing a lot of good things for us though, such as sending emails to our team for certain log events as they happen (the kind of stuff nagios may not be able to pick up on), and simplified dashboards that our customer support team can use to determine errors particular customers may have been getting. It's fantastic, but we pay an appropriate price for it. I always envisioned my splunk use as more Minority Report though, scrubbing out irrelevant background noise to find the killer 
|
|
#
?
Feb 12, 2015 10:13
|
|
|
NZAmoeba posted:I can't actually recall any existing staff we have making the move to NZ though, but maybe just none have applied, or it's happened and I'm just unaware of them. We do however have systems in place for new hires that happen to be overseas and want to work in NZ. It's typically a huge investment (from both employer and employee) to go to those lengths however, so it's only done if we believe it's worth it. I always thought NZ would be one of the easiest places to go, since they want developers and engineers so badly. When I visited, they gave me a packet on immigrating in case I was ever interested, because NZ neeeeddds computer people.
|
|
#
?
Feb 12, 2015 10:32
|
|
|
evol262 posted:Hey, no, it isn't. Getting a foreign work visa is all about talent and desire. Going to a "code bootcamp" is not more likely to get you offers abroad. And "just moving to London and applying for jobs there" is going to put you in a tricky visa situation. Get good at your job in the US and build some skills people may want abroad, then apply for those jobs or get sponsored by your current employer. Don't go about it backwards because you really want to leave the US. London is expensive. Would you "just move to New York and apply for jobs there"? Or SF? It's the same cost of living. This is a pipe dream.
|
|
#
?
Feb 12, 2015 11:03
|
|
|
evol262 posted:I always thought NZ would be one of the easiest places to go, since they want developers and engineers so badly. When I visited, they gave me a packet on immigrating in case I was ever interested, because NZ neeeeddds computer people. It is, I'm just not sure if any of our existing staff have decided to make that move after working in one of our overseas offices for a bit, but I'm well aware of NZ staff now working in one of our overseas offices.
|
|
#
?
Feb 12, 2015 12:29
|
|
|
I feel like I should buy something from this guy, he tries pretty hard. I've only bought one switch from him and it worked, so I guess for a typical reseller he's alright. He always sends quirkly sales emails with memes and stuff though.
|
|
#
?
Feb 12, 2015 15:27
|
|
|
Spudalicious posted:I feel like I should buy something from this guy, he tries pretty hard. I've only bought one switch from him and it worked, so I guess for a typical reseller he's alright. He always sends quirkly sales emails with memes and stuff though. That's actually kinda interesting. If anyone other than a salesperson emailed me like that, I'd be deeply unimpressed, but because it's a salesperson his corny schtick actually makes him more approachable.
|
|
#
?
Feb 12, 2015 15:32
|
|
|
I don't understand why Lenovo swaps the Fn and Ctrl keys. I know there is a BIOS setting to swap them back but why even do it in the first place Did IBM do this and Lenovo hasn't changed it?
|
|
#
?
Feb 12, 2015 15:46
|
|
|
Docjowles posted:Logstash love Well if you did all that you should go the last mile and get Logstalgia set up on a flatscreen in your ops area. https://www.youtube.com/watch?v=HeWfkPeDQbY
|
|
#
?
Feb 12, 2015 15:47
|
|
|
myron cope posted:I don't understand why Lenovo swaps the Fn and Ctrl keys. I know there is a BIOS setting to swap them back but why even do it in the first place Apple does the same thing...
|
|
#
?
Feb 12, 2015 16:06
|
|
|
Bob Morales posted:Apple does the same thing... I know. Well, now I know. I never use one, but I had to do some scripting on a macbook yesterday and I thought I was going crazy.
|
|
#
?
Feb 12, 2015 16:28
|
|
|
MagnumOpus posted:Well if you did all that you should go the last mile and get Logstalgia set up on a flatscreen in your ops area. The last mile is not ricer setups that are useless for actual metrics or indexing
|
|
#
?
Feb 12, 2015 16:39
|
|
|
Has anyone come across a good conversion tool that goes from a ansible inventory -> ~/.ssh/config <-> iterm2 host profiles? I find keeping 3 files is really, really stupid and I can't be the only one who wants to just keep my server info in once place. I'd be nice if there was any sort of centralized management, but you work within the system you are in... I'd write one myself, but I can't imagine I'm the only one who wants to do this.
|
|
#
?
Feb 12, 2015 16:52
|
|
|
evol262 posted:The last mile is not ricer setups that are useless for actual metrics or indexing Really? We get all of our mission critical stuff out of the way in the first miles and do the fun stuff last. Works pretty well!
|
|
#
?
Feb 12, 2015 16:52
|
|
|
Also, never underestimate C-level support for projects that can present pretty pictures to tours.
|
|
#
?
Feb 12, 2015 16:54
|
|
|
evol262 posted:The last mile is not ricer setups that are useless for actual metrics or indexing Sure, but they come in handy when you're conducting Dog and Pony shows for potential customers. Even if you're not really USING it for anything, having something like that on one of the NOC big screen wall monitors can be impressive. It's a lot better looking than a bunch of CLI script output on a 105" wall monitor. Plus, we tend to remove anything that could identify actual customer data when giving a tour.
|
|
#
?
Feb 12, 2015 17:26
|
|
|
NZAmoeba posted:With regards to your logstash setup, how much time and effort did you spend on setting it up? How much do you foresee in ongoing maintenance? We currently pay a fortune for splunk, which works really well for us, but doesn't mean we shouldn't consider other options from time to time. We do have splunk doing a lot of good things for us though, such as sending emails to our team for certain log events as they happen (the kind of stuff nagios may not be able to pick up on), and simplified dashboards that our customer support team can use to determine errors particular customers may have been getting. It's fantastic, but we pay an appropriate price for it. If you're dealing with structured log data -- log formats like JSON/GELF/RELP -- Logstash is a great fit. You can configure which fields get indexed by Elasticsearch (default is all fields) and work some really great visualizations off of that data. On the other hand, where Splunk really excels is inferring information from logs that it's never even seen before. You can feed it something with an IP address and it will make a pretty good effort to figure out what looks like a source or destination IP address and it will automagically correlate all that stuff together. With Logstash, you have to parse those log messages yourself (some built-in patterns make this easier than writing pure regex, but not my that much), make sure your field names match, and all that good stuff. It's much harder to extend coherently as part of a large team. Another thing to remember is that if you have to deal with compliance like HIPAA, SOX or PCI-DSS, Splunk has the technical requirements to be considered a legal system of record. Elasticsearch does not. Regarding scale: if you have enough cheap hardware to throw at it, Elasticsearch will perform exceedingly well. Throw a few out-of-warranty servers from the scrap pile at it -- if one breaks, replace it and let the data replicate from the other cluster nodes -- and give it a few terabytes of storage, and you're good on logs for a long time. Kibana can create some really great dashboards, and Logstash itself has an email output that you can use (in addition to PagerDuty, Nagios NSCA, and dozens of others) on messages matching certain patterns.
|
|
#
?
Feb 12, 2015 17:56
|
|
|
NZAmoeba posted:With regards to your logstash setup, how much time and effort did you spend on setting it up? How much do you foresee in ongoing maintenance? We currently pay a fortune for splunk, which works really well for us, but doesn't mean we shouldn't consider other options from time to time. We do have splunk doing a lot of good things for us though, such as sending emails to our team for certain log events as they happen (the kind of stuff nagios may not be able to pick up on), and simplified dashboards that our customer support team can use to determine errors particular customers may have been getting. It's fantastic, but we pay an appropriate price for it. I'd say setup took about a week where I spent 80% of my time on it, with ongoing fine-tuning. With the disclaimer that I was starting from zero centralized logging and only minimal understanding of how to configure ELK. So I spent a LOT of time reading up on best practices to make sure I built it to scale correctly the first time (I'm looking to store upwards of a billion events per day with a retention of 1 week, this was not a tiny deployment). I now have it deployed to about half of our infrastructure, but have hit a wall in terms of available hardware so that is stalled until I get approval to buy some SSD's and memory to finish the job  I'm sure I'll eat my words but I don't see it as a huge ongoing maintenance burden. Adding/removing dashboard items and log parsing patterns as we deploy or decomm applications, mostly. Cron jobs automatically drop old ElasticSearch indices so it doesn't grow indefinitely. I built everything in config management so it's a snap to deploy and scale. Logstash definitely has output plugins for email, Nagios, and most anything else so you should be able to keep that workflow. Kibana has a bunch of different visualizations, and there's a version 4.0 coming out later this year that looks really nice. and lol at that video game visualization thing 
|
|
#
?
Feb 12, 2015 18:28
|
|
|
How do you guys handle the 'only one person in IT should be able to change email passwords and potentially access others email'? It just seems like it would cripple us if someone locked themselves out or forgot their password (we have hosted Exchange btw)
|
|
#
?
Feb 12, 2015 21:13
|
|
|
mewse posted:That's actually kinda interesting. If anyone other than a salesperson emailed me like that, I'd be deeply unimpressed, but because it's a salesperson his corny schtick actually makes him more approachable. I wish my tech suppliers were like that, our stationary supplier however is completely bonkers; but she gives us free packs of biscuits and sweets, even sent three us who are quite buddy buddy with her some bottles of wine for Christmas! (Even our ISP sent us a bottle of wine as well, but I never saw the delivery...) Bob Morales posted:How do you guys handle the 'only one person in IT should be able to change email passwords and potentially access others email'? It just seems like it would cripple us if someone locked themselves out or forgot their password (we have hosted Exchange btw) Sounds like a fun situation without external support, if I'm not in then at least our provider can step in and have things authorised by a set list of people. Not having access to each others inbox sounds a bit nuts though, a lot of people here are in each others pockets to pickup/cover for people who are unavailable.
|
|
#
?
Feb 12, 2015 21:32
|
|
|
Bob Morales posted:How do you guys handle the 'only one person in IT should be able to change email passwords and potentially access others email'? It just seems like it would cripple us if someone locked themselves out or forgot their password (we have hosted Exchange btw) Simple... ADFS. Integration/Federate authentication so it is effectively done by AD and use all of the normal "user must change password on next Logon" stuff to keep techs from knowing anything. The other option is to build a tool that a tech can use to reset somebody's password and have the tool send them a text message or something clever like that. Most places understand that the administrators of things will have privileged access. It's part and parcel of the way things work. Dr. Arbitrary posted:Maybe have one extra account that has admin privileges and keep the password of random letters and numbers on a slip of paper that you keep in a safe. Zaepho fucked around with this message at 21:53 on Feb 12, 2015 |
|
#
?
Feb 12, 2015 21:50
|
|
|
Maybe have one extra account that has admin privileges and keep the password of random letters and numbers on a slip of paper that you keep in a safe.
|
|
#
?
Feb 12, 2015 21:50
|
|
|
Zaepho posted:Firecall account are definitely an option as well. I have seen this go so far as an A part and B part that separate groups of people have access to. (usually reserved for things like generating encryption keys for ATMs and the like) This is exactly how we managed our crypto boxes for inter-bank routing, ATM keys, and transaction log integrity. We had a Keeper of the Safe. Each team member would check out the appropriate physical keys to open locked compartments in the safe which had one half of the access passphrase. After use the team members would generate a new half, do a password change, seal up their half in a tamper resistant bag and put it back. All while being observed by a third party to ensure no shenanigans. The safe was also rated for an insane amount of time for direct application of fire, and the compartments themselves had additional fire resistant ratings. We also had different compartments containing access details to mission critical systems as a method to avoid a "hit by bus" scenario. It was very cumbersome, but fortunately that never had to be used very often. Of course the Auditors loved us for having that system in place. Those systems were by far the most critically sensitive pieces of the bank's infrastructure.
|
|
#
?
Feb 12, 2015 22:51
|
|
|
flosofl posted:Of course the Auditors loved us for having that system in place.
|
|
#
?
Feb 12, 2015 23:48
|
|
|
Anyone else in an environment with no technical peers? I'm starting to go a bit whacky from lack of people to bounce ideas off of. Is it batshit nuts to have a single member of the staff responsible for all storage/vmware/wan and datacenter network architecture and configuration for a 2000 person company and the next most senior technical person doesn't even know what vmware or a subnet mask is? I'm rolling out a few new disk shelves and re-designing our DR plans and waking up at night thinking about best practices and what I could be overlooking, because there is literally no one to talk tech with or to check my work. I'm confident in my ability and spend a lot of time on IRC and forums, but I'm just really burning for some professional feedback 
|
|
#
?
Feb 13, 2015 02:10
|
|
|
syg posted:Anyone else in an environment with no technical peers? That's what we are here for. Go ahead buddy. Let it all out. We will tell you if you hosed up something.
|
|
#
?
Feb 13, 2015 02:12
|
|
|
syg posted:Anyone else in an environment with no technical peers? Yes
|
|
#
?
Feb 13, 2015 02:18
|
|
|
jaegerx posted:That's what we are here for. Go ahead buddy. Let it all out. We will tell you if you hosed up something. Thanks. I don't have any particular technical questions, I know I'm on track at the moment any way. I've done all this before - every few years we refresh vmware/storage and I become an expert on that again until I forget everything after the dust settles. But I really wish I could chat to another architect or engineer level person when these ideas are bouncing around in real time. At our company money is tight for staffing right now and I understand that, but the amount of autonomy I have is frankly disturbing. I've been left to design and choose every infrastructure system we use without anyone ever checking my work or second guessing my choices. I could literally be creating a multi-million dollar bot net here and I don't think anyone would notice. Granted this has been the case for years and I feel very appreciated, but this just seems like bad business practice giving one person this much leeway.
|
|
#
?
Feb 13, 2015 02:28
|
|
|
syg posted:Anyone else in an environment with no technical peers? Sort of. I think if it were ten years ago then a lot of the knowledge my colleagues have would be relevant, but stuff like VLANs and routing shouldn't be voodoo magic. Unfortunately they seem to be in a position where sounding like they know what they are talking about is good enough, and actually bringing a solution together isn't a required skill. Since they can just walk away from a half-done job and nobody seems to call them on it. I would understand if they were in a management role and delegated the technical stuff, but the first response to being asked to look into providing a new service for our customers is to start pricing up servers and co-lo space because the #1 priority is to play with new toys and who cares if it never generates a profit? I think it's going to be  o'clock soon, I'm terrified of getting stuck.
Thanks Ants fucked around with this message at 02:35 on Feb 13, 2015 |
|
#
?
Feb 13, 2015 02:32
|
|
|
.
Chickenwalker fucked around with this message at 03:00 on Mar 1, 2019 |
|
#
?
Feb 13, 2015 02:50
|
|
|
syg posted:Thanks. I don't have any particular technical questions, I know I'm on track at the moment any way. I've done all this before - every few years we refresh vmware/storage and I become an expert on that again until I forget everything after the dust settles. But I really wish I could chat to another architect or engineer level person when these ideas are bouncing around in real time. At our company money is tight for staffing right now and I understand that, but the amount of autonomy I have is frankly disturbing. I've been left to design and choose every infrastructure system we use without anyone ever checking my work or second guessing my choices. I could literally be creating a multi-million dollar bot net here and I don't think anyone would notice. Granted this has been the case for years and I feel very appreciated, but this just seems like bad business practice giving one person this much leeway. Have you thought of engaging a proffesional services firm for a day to sanity check your work. It is how I dealt with similar situations.
|
|
#
?
Feb 13, 2015 02:56
|
|
|
syg posted:Anyone else in an environment with no technical peers? I have one. He's a contractor and only comes in when he wants/needs to, which comes out to about two days a week. He used to take the other desk in my office but his part time status and our lack of space meant he got booted to the data center when we hired another full-timer. She was hired to process user account requests. I absolutely do not blame her for losing what was a great office-mate, and we do get along, but goddammit this is a terrible office pairing in terms of... well, everything.
|
|
#
?
Feb 13, 2015 03:45
|
|
|
|
| # ? May 8, 2024 10:28 |
|
|
KennyTheFish posted:Have you thought of engaging a proffesional services firm for a day to sanity check your work. It is how I dealt with similar situations. This is probably the best idea because frankly no matter how diligent you are you aren't truly becoming an expert in all (or even any) of those things if you're only dealing with these issues extensively once every few years. Operations and Engineering/Architecture are pretty different and it's unlikely that you've got enough time to keep up with the engineering stuff you'd need to know if you're running operations all day every day.
|
|
#
?
Feb 13, 2015 04:39
|
|
