Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
luminalflux
May 27, 2005



I live and work downtown and don't have a car, mostly because I don't want to pay $250/mo for parking. Which is great since I've barely ever driven, get stressed out driving in the city and don't have to worry about driving drunk this way. Public transit, the occasional Lyft and very infrequent Zipcar/rentals cover 99% of my needs.

(the other 1% is going down to San Jose on weekends but seriously gently caress that place)

Adbot
ADBOT LOVES YOU

luminalflux
May 27, 2005



Virigoth posted:

A good go to is make them explain what they'd do in a scenario with a whiteboard. Maybe it's a task you've been thinking about automating that is just busy work for a new tech.

Ask them to solve a current problem you're working on or just solved and ding them because they didn't come up with exactly the same solution you did with the same tradeoffs.

luminalflux
May 27, 2005



I mean my interviews I have to basically play dumb about poo poo i know a lot about to figure out what they know about. If they don't know stuff I just start talking about video games until my 45 minute slot is up.

Hint: if you're interviewing for my company and we start talking about video games after 10 minutes it's probably not going good.

luminalflux
May 27, 2005



Working at the HPC center at KTH seemed interesting. Basically most research projects in the Stockholm area colocate their supercomputers there, so the center isn't actually purchasing the hardware, the projects are. Unfortunately this means that it's been tricky to source money to expand infrastructure like power and cooling when someone happens to drop a new cluster in the lap of the people running it.

Also, you have to deal with academia politics in environments like that.

luminalflux
May 27, 2005



Vulture Culture posted:

My current hate trend is companies that want 10-20 hours worth of code sample homework from candidates with absolutely no compensation for their time

Or even worse, after that a "work trial" period.

No i'm not going to quit my job to audition with you, either you hire me or not.

luminalflux
May 27, 2005



Tab8715 posted:

18 to 6-month contracts aren't bad but holy hell I don't think I'd ever pick up a gig that had some 90/30-day probation exception. What the hell are these people thinking?

This isn't that long, it's a few weeks or something.

In sweden it's normal with 3-6mo probation, but you have full benefits et c during that time. During probation, you can quit or be fired with 2 week notice. After that, your notice period is like, 1-3 months.

luminalflux
May 27, 2005



KS posted:

We have Cisco UC phones and I could buy nice video endpoints, but maybe there's a simple point to point solution I'm missing?

We use Chromeboxes at work with google hangouts.

luminalflux
May 27, 2005



We have a shitton of bare metal but we contract all that stuff out to Racklive. They build the servers to our spec, rack and cable and ship them to wherever they go in the world.

luminalflux
May 27, 2005



H110Hawk posted:

I'm curious, how accurate have your orders been? We recently added Racklive to our vendor mix and it seems they talk a lot more game than they actually have on the "cookie cutter rack" approach.

Accurate enough that they're the only vendor we use and we're one of their featured clients. I'm not in DCOps though so I just get a pile of servers' IPMI addresses handed to me after DCOps and Network has done their stuff, I don't really deal with any snafus with Racklive if they arise.

quote:

Your avatar actually made me think you work with me, you don't work in adtech do you?

gently caress no

luminalflux
May 27, 2005



Docjowles posted:

Yeah I think you'd do well in NC. Good opportunities, and you can cruise out west toward Asheville and pretend you're in Lyons or something when you need a break from the "big city" life of RTP.

If I needed to settle down and for whatever reason needed to be on the east coast, I'd do NC in a heartbeat. Good food and beer, climate isn't to horrible except for a couple months in the summer.

luminalflux
May 27, 2005



BaseballPCHiker posted:

At what point do you say enough is enough when it comes to interviewing for a job? The most I've ever had was a phone screening, 1st interview that was informal, and then a second more technical interview and overview of future projects and IT needs. If I ever have to past that I think I'll just say no unless I'm really desperate.

Facebook (2013): short phone screen with recruiter. 3 45 min long phone interviews with coderpad where they ask you to solve stuff, after that, on-site (i didn't get the onsite)

2 silicon valley unicorns (2014): Short sync with recruiter, 45 min phone screen with technical person, after that on-site with 6 hours of interviews.

I got the gig at one of the unicorns, hasn't changed remarkably in the last year or so. Seems to be standard for this area.

luminalflux
May 27, 2005



An email came in

beer@work.com posted:

On tap:

Bourbon County Stout
Dogfish 90min IPA
Tahoe Mountain Festivus (Dark Dry Saison w/ spices)
Scrimshaw
Tieton Cherry Cider
Schneider Aventinus


Also i woke up at 11.45 (had no meetings early today so why not) and saw my boss had sent an email at 8am asking me to be in a meeting at 11. Woops.
(actually he was OK with it since he's been asking me to take it easy the last few days)

luminalflux
May 27, 2005



Sirotan posted:

BCBS on tap, in your office?? I hate you.

So glorious. Schneider Aventinus was OK but the Dark Saison with spices is the vilest thing this side of pumpkin beer.

luminalflux
May 27, 2005



H110Hawk posted:

Step N is hire someone else to do it, which is why these look as good as they do.

I'm never cabling a rack again if I can help it.

luminalflux
May 27, 2005



role-instanceid.pop. InstanceID is either from AWS or from the last 3 octets of eth0's MAC address. POP is derived from the IATA airport code with a sequence number, or AWS region.

Before it was just role#.pop, where number was assigned sequentially when bringing up services. Changing to use instance id was apparently very traumatic, to the point where people were actually yelling about how bad it was that they can't remember smtp-af09c3 when they could have a hallway conversation about mail7.lhr01 (on several occasions).

It was however very necessary, since before this our main service machines were named foo1,foo2,foo3,foo4 et c. There are 2 different subroles inside this service, and blades could either be a foo-blah or foo-ugh, which have implications in IP address space, kernel options et c. We need one foo-blah per chassis, so it was tribal knowledge that foo1,foo9,foo17 did foo-blah and others were foo-ugh. Oh yeah also the AMS pop had 4-blade chassis so there foo5 and foo13 were foo-blah servers most of the time. Well until we had enough foo-blah in that rack so then it went to 1 foo-blah per 7 foo-ugh.

:smithicide:

luminalflux
May 27, 2005



This is very traumatic thinking to a group of engineers who have been hand-feeding and hand-tuning everything for the last 6-7 years. "Well just shoot it in the head, provision a new server in it's place and we'll open a datacenter ops ticket to get it looked out" got me looked at funny and i was shouted at that since my team was so slow at provisioning new systems there could never possibly be any improvement over what they currently had.

luminalflux
May 27, 2005



It's laid out in the blog post.

quote:

1. Neti contacts Zookeeper1-proxy, and, using its instance ID, inquires if it has ever been registered. If found, it gets the same overlay IP as before. If not, it randomly chooses an available overlay IP and locks it to this instance ID.
2. Neti sends up the IP information and network location to Zookeeper to complete registration.
3. Neti downloads the current list of running instances from Zookeeper, including all of their public, private, and overlay IPs, as well as the network they live in.
4. The list is parsed, and iptables filter and DNAT rules are generated for each of the entries.
5. Neti sets a watch on the Zookeeper instance list.

Concurrently, as soon as step 2 finishes, all the rest of the registered instances get their Zookeeper watches triggered with the new set of instance data, and their iptables configs get updated automatically.

Once this dance is complete, all of the instances have full access to each other, and are successfully blocking any unauthorized traffic. If another instance spins up, this process starts again; if any instance dies, Zookeeper notifies all of the Neti daemons of the change and rules are updated within seconds across the entire fleet.

luminalflux
May 27, 2005



Turtlicious posted:

Do you guys think you could help me with my resume? Or are you all convinced I made everything up, and there's no more help on the table? I'd go to the resume thread, but since I have a specific field in mind, I feel like here would be better.

If getting on BART scares you I can't help you with your career

luminalflux
May 27, 2005



Turtlicious posted:

That was a half joke, and I have a car, but doesn't the fact the thing goes under the bay in an earth quake prone area, even a little disconcerting?

Less disconcerting than surviving on peanuts in the bay area.

(Earthquake is probably more welcome than whatever gross poo poo grows in the BART seats)

luminalflux
May 27, 2005



Super Soaker Party! posted:

I mean all that said, there's also something to be said for the outdoors feeling just like the indoors in terms of climate. Generally not having to do any preparation to walk outdoors no matter what time of year is a luxury you don't even realize you needed until you wander outside in your bathrobe and slippers at 2 AM to get the mail and you don't freeze to death, or you go out in the middle of the day and don't burn to death. And for me personally, the sun is the topper - the Pacific Northwest has a similar climate for a lot of the year, but I could not live with the loving grayness.

Things I love about SF:
The fact that "long pants and a hoodie" will be perfect clothing 95% of the time
That majority of other tech companies are here so at some point "hey look we can just swing by instead of hashing this out over support tickets", esp for early-stage companies. When we were evaluating Vault, Hashicorp came to our office since it was literally 3 blocks away.
Wonderfully varied nature.

Things I hate about SF:
Hearing people talking about their loving bitcoin startups at the bar.

luminalflux
May 27, 2005



uhhhhahhhhohahhh posted:

I scp the file to my windows laptop, edit it in vscode, then scp the file back, because I'm a disgusting noob

Vscode can edit files over scp so you don’t even have to copy back and forth

luminalflux
May 27, 2005



BeastOfExmoor posted:

Honestly, the best experience I've had with wireless screen sharing in meetings is using whatever conferencing software you're using (Teams, Zoom, Webex, etc.) and just joining the meeting from your laptop and sharing the screen. Obviously this requires that your rooms already have the hardware present to join the conference room to your conferencing software directly, but I think that's becoming more and more common.

Zoom Rooms is pretty much the best experience in this regard. It Just loving Works - you can open your laptop, hit "Share Screen" in the zoom client and it will automatically figure out which room you're in and cast to the right room without having to enter any meeting details. Also supports Airplay if your guests are used to that.

luminalflux
May 27, 2005



12 rats tied together posted:

SQL Server is a fine database except it is also expensive as poo poo -- another problem you aren't likely to run into in AWS since you guys pass the per-core licensing model onto the customer.

I mean there’s RDS SQL server but ok

luminalflux
May 27, 2005



eonwe posted:

2. Their online resources are much better than the online resources of more traditional universities.

Holy poo poo regulate universities must suuuuck then. I’m doing a BS in computer science at WGU (after dropping out of a good uni 15 years ago) and the math books at WGU are godawful

I’m doing it mostly as a checkbox to affirm that yes I know wtf I’m doing after 20 years in the industry. Mentors are hit or miss, mine is a nice grandmotherly lady who knows I’m trying to finish it in 1 year and helps me schedule accordingly. My buddy’s mentor for network and security was not as good and he had to fight with their boss to let him complete at a fast pace.

luminalflux
May 27, 2005



TheParadigm posted:

Ugn. Isn't charter godamn awful?

They (Spectrum) are pretty bad where I am in LA. My internet went out at like 5pm on Monday. Their twitter support claimed "there's no outage in your area" despite their UI saying there was a TV outage. Apparently the TV outage was "only affecting on-demand TV" so having no RF at my modem was unrelated??? They scheduled a tech to come out, earliest was for Sunday. So I went ahead and ordered AT&T fiber and will use Spectrum as a backup.

Later coming back from getting tacos, I see a spectrum cherrypicker doing cable guy things and 12 hours later we had internet again.

luminalflux
May 27, 2005



Spring Heeled Jack posted:

Anyone here working as an SRE? I’ve been a system admin working alongside a dev team for a little while now and I feel like this is the next step for me. There’s no real place to go up aside from management at my current place unfortunately.

If anyone can provide some insight, what does your day-to-day look like?

SRE at a Series D SaaS company, 4 SREs supporting 50 engineers and my day to day is Different.

I've spent the last couple days:

* Trying to switch to a blue/green deploy system
* Realizing that blue/green deploys hammer the gently caress out the database since we hit the DB with 2x the number of queries
* Support the SRE that really knows mysql to help bring up a new instance with different configs and sizing
* Digging through Datadog metrics and traces to realize why we're suddenly hammering the database
* Digging through SQLAlchemy weirdness to figure out why one specific query is now happening way too often
* Poking at Cloudflare bot mitigation rules
* Poking at Cloudflare caching

Tomorrow maybe I can dig into getting the instance bake and start times down, along with rolling out a new consul and vault cluster.

Happy to answer questions over PMs.

luminalflux
May 27, 2005



I used to work at a really toxic """unicorn""" that got subsumed into Amazon (it's a service you're probably familiar with!). While it was great getting the relocation from europe and they pay certainly was good, being on-call for stuff I couldn't fix, working long hours and basically being shat on by every part of the organisation because SREs aren't real engineers took an immense toll on me psychologically. Like, multiple people yelled at me and I had one engineer scream at me for changing a host name convention.

After I left I went to a place that was strictly 9-5, no on-call and very relaxed for a slight pay cut as an SWE. I lost a ton of weight and felt emotionally a lot better, but after a while I realized i wasn't really doing anything creative or pushing any boundary, just shoveling more Go into the TDD machine. 9 months and an IPO later I went to something with a bit of a pay bump and on-call, but with 50 engineers instead of 500 I have a lot more control and impact over my life as an SRE

luminalflux
May 27, 2005



Zorak of Michigan posted:

What was the SRE role at this place? I usually hear them described as pretty senior folks, since they're dealing with problems much bigger than ordinary computer-toucher stuff.

This was in 2015-2017. Everything from building inventory systems, building imaging systems for bare metal, secrets management to maintaining a really bad mess of puppet, running DNS servers and getting caught in departmental politics

luminalflux
May 27, 2005



Wibla posted:

What kind of dystopian hell do you live in where they remove the (free, of course) coffee from the office?

Seriously.

I worked nights as a temp at a NOC for a telecoms company. They had a wonderful WMF machine that would grind on demand, Illy beans, milk foamer, the works. Of course they would shut that off at the end of the normal working hours and run the cleaning cycle and then start it up at 8am for 9-5ers, leaving us with lovely instant coffee machines for the night shift.

luminalflux
May 27, 2005



jaegerx posted:

i just meant their interviewing practices are awful, not this, not this...

Glassdoor that poo poo. Recruiters care extremely about glassdoor. (At least the ones I work with)

luminalflux
May 27, 2005



jaegerx posted:

I’m actually liking consul right now.

It’s become a lot better since we first used it in beta at $unicorn. Stuff like consul-template restarting varnish 300 times in a minute since it couldn’t coalesce updates was fun, and no authentication on anything

luminalflux
May 27, 2005



12 rats tied together posted:

Rolling new amis for every change sucks rear end to actually do in prod, even if it is the standard answer that a lot of people reach for.

I have Spinnaker do this for me, works great.

luminalflux
May 27, 2005



12 rats tied together posted:

- no way to define spinnaker jobs in a DSL or markup language

https://github.com/armory/dinghy is what we use for pipelines as code.

quote:

- auto-configuring login users, permissions, etc, was kind of a nightmare (I wanna say it had groovy scripts or something?)

We use Okta for auth, haven't had to deal with fiat for setting permissions.

quote:

- hard to integrate with other tools, in our case, ansible

What's your ansible integration case? In our case, in the bake case packer kicks off an ansible playbook on the instance that configures the instance (pulls down apt packages, pulls code, configures crap). When the instances launch in the ASG, they run an ansible playbook that does a quick reconfigure of datadog / filebeat / application and starts the correct services based on tags.

We've integrated Spinnaker with our various chatbots and deploy services with the API and it's been pretty OK - not the best api, not the worst.

luminalflux
May 27, 2005



12 rats tied together posted:

This was a couple jobs ago so I don't recall the exact case but we wanted "the full playbook experience" -- slack messages, grafana annotations, cloudformation update orchestration, etc. I'm interested in the instance launch scenario, are those nodes using ansible-pull? It's a really cool use case either way.

No idea what ansible-pull is - we do a git clone of the ansible repo, switch it to the branch specified in tags (for testing ansible branches) and run ansible-playbook locally. We don't run ansible from a controller or anything, each instance runs it locally.

luminalflux
May 27, 2005



Methanar posted:

We have a bunch of jsonnet that is instantiated for each service to render out standard pipelines for all of our environments. We just get a big json blob out that gets posted to the spinnaker API.

pre-prod vs prod registry to pull from, graphite annotations, canarys, optional manual approval steps, optional canary analysis

It works okay if you can tolerate jsonnet

I wrote something like that for templating out pipelines until we get dinghy fully up and running, and at one point I wrote a Terraform provider for expressing pipelines. It's not hard to do, just tedious.

luminalflux
May 27, 2005



12 rats tied together posted:

ansible pull is a utility that turns ansible into chef, you probably already have it installed even! it's part of the standard library

the idea is that you cron it, basically, and it otherwise does what you describe

Looking at it, it's similar to what we do but not entirely for branches. Their behaviour seems to be to check out a branch. We check out the main branch and then merge the branch we're testing on top of it. Basically it makes it a lot easier for testing your branch since you don't need to constantly rebase on top of the main branch (so if someone's changed app config and merged it, you get those changes).

quote:

I actually can't stand jsonnet/ksonnet/proliferation of go templates/etc. IMO tools should focus on the end documents and be engine agnostic, let me bring my own templating basically.

If JSON is your end result, you can template Jinja in YAML and just convert that to JSON - that's what I do for my pipeline templates.

luminalflux
May 27, 2005



12 rats tied together posted:

My initial reaction to this was very negative -- I want to test my branch, right? Not my branch after I had done some other thing to it. If I wanted to test that other thing, I would make my branch into that other thing, and then test that instead. But, thinking about it for a sec, about 99% of the time I do actually just want to test my branch plus latest master. I always rebase anyway because it's muscle memory, but, this is a valid workflow too and would be helpful for people who aren't super into git.

It's a bit of a mindfuck at first but it's also how we test app branches in production too when someone wants to canary a change before merging, since devs aren't great at git and we have a fast-moving monolith. 99% of the time this is what you want - you care about your changes, and if someone else changed stuff in the other part of the app that's fine, and merging it on top of the main branch will most likely be correct and good.

The logic is basically: if there's no canary branch specified, use the sha that is baked into the AMI. Else:
code:
cd /app
git fetch
git reset --hard origin/release
git merge --no-commit origin/${CANARY_BRANCH}

luminalflux
May 27, 2005



WFH + cloud means i can have AC, but the noise levels are vastly lower than in the datacenter. Plus Telecity always kept their cold aisles at like 23-24 degrees celsius at 50% humidity.

luminalflux
May 27, 2005



Dick Trauma posted:

The CEO insisted on moving us last winter, into a space that was undergoing renovation. There is basically no upside other than he can piss away a ton of money on a worthless and ugly redesign. It has a been a clusterfuck, and the renovation is still going. The place is a disaster zone, concrete and drywall dust coating everything (including my servers.)

It would be funny if he winds up shitcanning the whole plan to go with some sort of WFH arrangement.

Pinterest paid $89 million to get out of a long lease for a new buildout that was presumably already underway. Instead of moving everyone under one roof, they'll keep their 4 offices and have a lot of people remote. So it's not like your CEO would be the first.

Adbot
ADBOT LOVES YOU

luminalflux
May 27, 2005



The Fool posted:

It's a 100% remote cloud/automation engineer role. I should never have to think about on-prem servers again if I take this job.

I love being in this position. I just manage the AWS fleet that runs the app. IT has a couple physical servers for god knows what reasons but that's their problem, not the SRE team's.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply