|
StumblyWumbly posted:It sounds like the relationship between product and engineering has fallen apart. You can fight for a maintenance budget that is under the control of engineering, which can make it easier to get changes through that improve your life but are invisible to the customer. I agree with most of this. I just want to say that "brilliant" code that only one person understands is usually not "brilliant." The more years I've spent as a developer, the more I've come to prioritize readability and maintainability. I'd almost rather see wrong code that's readable in a PR than correct but inscrutable code. At least I can mentor the dev who wrote wrong code towards writing correct code. I don't know what to do with inscrutable code aside from asking for in-code comments to explain what the code is doing until I can understand it.
|
#
?
Jun 24, 2023 15:31
|
|
|
|
| # ? May 26, 2024 02:51 |
|
|
LLSix posted:I agree with most of this. I just want to say that "brilliant" code that only one person understands is usually not "brilliant." The more years I've spent as a developer, the more I've come to prioritize readability and maintainability. This I saw some quote I liked that boiled down to “avoid writing code that could be called clever”. Clever in this case meaning taking advantage of something non standard or doing some cool one liner trick. It’s neat for as long as the initial PR for it is up and then it becomes a headache for future you and anyone else coming in and seeing it for the first time lol
|
|
#
?
Jun 24, 2023 15:39
|
|
|
LLSix posted:I agree with most of this. I just want to say that "brilliant" code that only one person understands is usually not "brilliant." The more years I've spent as a developer, the more I've come to prioritize readability and maintainability. I'd almost rather see wrong code that's readable in a PR than correct but inscrutable code. At least I can mentor the dev who wrote wrong code towards writing correct code. I don't know what to do with inscrutable code aside from asking for in-code comments to explain what the code is doing until I can understand it. Yeah, I run into this a lot with Python list comprehensions, where the folks who've completely internalized some big stuff find it beautiful and self-explanatory, but its completely inscrutable and impossible to debug for everyone else. An important thing is to have standards for "readable". When I interview folks, especially senior folks, if they don't have a standard for readable I worry it means they just rewrite things, because everyone can read what they wrote themselves.
|
|
#
?
Jun 24, 2023 15:46
|
|
|
Thank you for the guidance!StumblyWumbly posted:And you should understand that going from garbage code everyone is familiar with to brilliant code one person understands is not attractive to some people. Jabor posted:The second question I think is really key. Why are you considering refactoring this 1k LOC function? Is it causing a significant number of defects and cleaning it up will result in fewer bugs shipped to production? Is it a common bottleneck in making changes and refactoring it will improve developer velocity? Is it just because it offends your technical sensibilities but doesn't actually impact your day-to-day at all? Product's priorities always beat out engineering priorities, and so even if I make that case then it won't change anything. I know this because people far more experienced than I have come and gone. Maybe it's just a matter of telling an even more compelling story than Product can, but I wonder if that would also be a waste of time. Maybe I'm looking in the wrong places, but I don't see any success stories from other Large Organizations about how Engineering priorities beat out Product priorities in that way. Maybe I just need something from which to draw inspiration. leper khan posted:You make a case for how the changes you want to make will positively impact the metrics your organization cares about. Make a clear connection to increasing revenue, reducing costs, or affecting something else with a clear line to those. Make the thing you're targeting measurable. "this code feels bad" isn't measurable. If you give your boss a new metric that lets you do the work you want to do and can get them bought in to how that metric affects the bottom line metrics you'll get to do the work. Love Stole the Day fucked around with this message at 16:44 on Jun 24, 2023 |
|
#
?
Jun 24, 2023 16:23
|
|
|
Love Stole the Day posted:Isn't this way of thinking is the entire reason why knowledge silos and poor confidence in Production deploys occur in the first place? Reducing the risk of a bad deploy affecting revenue or reputation is not as compelling a motivation as the chance of a new Product feature's soft launch going well. Unfortunately that has to be born out in the sales cycle or the development process for it to affect change. If you can identify areas of a codebase whose improvement will lead to faster feature delivery that’s a compelling argument to make to product to ensure it’s addressed. Similarly if there’s a defect or quality issues in a codebase that’s affecting sales or renewal then product is incentivized to hopefully want to address those problems. But, at the end of the day, in an organization of sufficient size, you don’t hold the decision making power of whether to address these concerns. The best you can do is make the argument. Stealth edit: it’s worth noting that engineer attrition is a lever you can pull in terms of incentives for product to want to do something about quality as well.
|
|
#
?
Jun 24, 2023 16:42
|
|
|
oliveoil posted:When I started, $200k was top 5% of households and the opportunities were so great that micromanaging was unnecessary to hit profit margins. Now $250k is only top 7% of households and every task of two or more hours needs to fall under an entry on a sprint board. We're definitely heading in that direction. At our level of TC, we should really realize that it’s not the most important part - especially after a year or two. We need to focus on what it takes to be a good, focused, happy, productive and proud human who just so happens to do software engineering. No amount of figgies can help us if we’re jaded, burned out, unreliable, and disengaged. Our hierarchy of needs is malleable. I, at least, am not fulfilled if my TC looks good on paper but I’m doing a poor job as an engineer to the point where my manager’s like “drat you’re kinda useless huh”. Also, keep in mind that our compensation is not permanent or simply a function of us as individuals. It is far more informed by the market, the company, the people, the strategy, and the hype. We can’t let ourselves just be numbers in a bank or trading account. Pollyanna fucked around with this message at 16:49 on Jun 24, 2023 |
|
#
?
Jun 24, 2023 16:47
|
|
|
Love Stole the Day posted:Lack of functional coverage (or even a standard way to measure it) makes teams scared of merging PRs that refactor old, load-bearing code (e.g. a 1k LOC load-bearing function written and maintained by contractors). Tangential but this reminds me that imo coverage stats seem like a security blanket for people who don't actually understand the codebases they're "managing". If you understand the components of your codebase and what they're supposed to do then it should be very clear from looking at the tests whether you've covered the classes of potential inputs. I guess if they're hiring people who wouldn't even do the bare minimum of diligence to make sure their code works then having the equivalent of a smoke test for every unit is infinitely better than nothing but still seems like a security blanket. Half the time the one test is functionally a change detector anyway. Cyclomatic complexity measures might be different, never seen one actually implemented but cyclomatic seems like the only way to understand code coverage that didn't seem like an oversimplification made to soothe non-coding types who can't look at the code but have a mandate to make sure it's tested. So you get code paths covered with just one of maybe half a dozen classes of valid inputs and maybe one class of error but probably not as it's tempting to let that fall under the 10-20% of uncovered code a lot of places allow. I don't know if this is really how things have to be but it seems like we have bad abstractions that only exist to justify box-checking. It's like satisfying the letter of the law but not the spirit and I hate it so much. Rant over. Pollyanna posted:Also, keep in mind that our compensation is not permanent or simply a function of us as individuals. It is far more informed by the market, the company, the people, the strategy, and the hype. We can’t let ourselves just be numbers in a bank or trading account. Yeah it's the lack of permanence that alarms me but I don't worry about it too much because if AI takes our jobs then it should be able to take over management too. oliveoil fucked around with this message at 17:41 on Jun 24, 2023 |
|
#
?
Jun 24, 2023 17:35
|
|
|
I have no idea what will come of AI or how exactly it's relevant to my continued employment and financial prospects, so I just don't think about it. I know at least one person was curious to hear about how things went wrong on my team, but I haven't written it up yet. If people are still interested, I can share whatever's appropriate.
|
|
#
?
Jun 24, 2023 20:47
|
|
|
I'm still interested.
|
|
#
?
Jun 24, 2023 20:50
|
|
|
Post it up
|
|
#
?
Jun 24, 2023 20:52
|
|
|
oliveoil posted:Yeah it's the lack of permanence that alarms me but I don't worry about it too much because if AI takes our jobs then it should be able to take over management too. Lol. Neither the "make computers do things" or the "manage people that make computers do things" or the "correctly transcribe the vague conflicting desires i have into a list of things computers need to do to satisfy me" jobs will be subsumed by AI.
|
|
#
?
Jun 24, 2023 20:56
|
|
|
Gotcha! This will take a while to write up.Lyesh posted:I don't really see who's going to replace them as ad-tech kings tbqh, so they may be fine once the AI bubble pops and it stops looking like they're going to get their lunch eaten by openai/microsoft. There's nobody else sizeable in that space besides Microsoft and Amazon (arguably), is there? I have no fuckin idea what the business side is planning or whether they will survive the downturn and remain top of ad-tech, but I do know that I'm not confident in at least one part of Alphabet's future, so if it does survive it's not going to be the same. Its future ultimately doesn't matter to me, I'm a mercenary in the end and if a business agreement goes bad that's the breaks sorry. (hope you're doing well btw ❤️) Pollyanna fucked around with this message at 21:19 on Jun 24, 2023 |
|
#
?
Jun 24, 2023 20:56
|
|
|
Pollyanna posted:I have no idea what will come of AI or how exactly it's relevant to my continued employment and financial prospects, so I just don't think about it. I'm not a real AI expert but I can reasonably claim to be the local expert in most groups, and I'm not convinced that current approaches are ever going to do much more than they can do right now. In brief, the issue is that text is generated by guessing the next word based on what's been written so far. That's fine for writing short snippets of code or a five paragraph essay, but it can't scale to the length of a novel, or a thesis or a large program because those documents have a structure that can't be described by just looking at the words they contain. We're going to need a fundamentally different approach to get to that point, and that probably involves solving some hard problems regarding the way our brains store and generate information. I think we probably will see some tasks successfully automated but I don't think any of us are going to be replaced by machines.
|
|
#
?
Jun 24, 2023 20:56
|
|
|
previous putative methods at putting small and large structures together have proven utterly worthless compared to shoving more data at it. like the last 9 times. current actual bottleneck in my view is an incredibly prosaic algorithmic one: attentional context scales O(n^2) compute, so you can't feasibly jack n above like 10^4 or 5. there are like 6 putative O(n log n) dealios or whereabouts and they all, i must note in the strongest words, fuckin suck poo poo out of a fuckin butt. every one sucks dick. and not in the happy fun way. if they find an O(n log n) dealio that doesnt suck, coherence at novel length will then be trivial 0 help from that front on the bullshitting bob dobbs is dead fucked around with this message at 21:12 on Jun 24, 2023 |
|
#
?
Jun 24, 2023 21:10
|
|
|
Others have mentioned it, but adding my support to the whole 'tie to something people give a poo poo about', but also...it does help if you have leadership that at least tries to give a poo poo. The biggest problem with quality is trying to turn it into a measurement in terms of 'If we add tests to this codebase that has none, we'll accomplish XYZ savings', which is often very difficult to guess at, even if reasonable people will all agree it does improve it. Blinkz0rz posted:Stealth edit: it’s worth noting that engineer attrition is a lever you can pull in terms of incentives for product to want to do something about quality as well. Adding onto this: the people who are going to leave due to stuff like this are also often the ones you want to keep, not the ones you quietly hope quit on their own to help you lower headcount. Finally: re: AI - My company is also going balls deep on AI and I find it just baffling. I haven't seen any actual AI Engineers who are backing up the insane claims that PR people are making, and while I certainly think there's room for it to do cool improvements (ML in general I think does have some very interesting ideas for finding patterns in large datasets, for example), some internal video just called it one of the most important developments of the decade and I'm just like 'uhhhhhhhh'. I'm in the same boat as Pollyanna; I'm happy to continue not thinking about it.
|
|
#
?
Jun 25, 2023 01:15
|
|
|
There is exactly one funny and entertaining thing AI has accomplished and it smells like a gymbag.Rocko Bonaparte posted:Pollyanna, did your team and you just wind up being support and janitorial services for everybody else? Like, any SRE team for software engineering? I'm curious about how it all's gone to poo poo. tl;dr: Yes. My team is historically the dumping ground of our org, and has had so much difficulty and taken so much blame that I am left burned out and disappointed. Now the blame is getting personal, and now I am getting pissed. --- Alright, here's how we got here. The Basics I work in Google's networking organization. This organization handles its network and networking needs. Simple as. My team's job is to configure switches and routers:
My team also has a bad reputation: We are slow. Let's take one of my first owned features for the big new non-production project thing as an example: ACLs. It took us over a year to implement ACLs as asked. I first started work on them shortly after I joined in October 2021. Whole thing's stop and go - "stop" because the ACLs team doesn't like the solution and keeps rounding back for concerns and changes, "go" because how loving DARE we block the project? At one point I misinterpret the requirements and start emitting the wrong ACLs, leading to the feature getting kicked back. No harm done in this instance other than some failed tests and annoyed stakeholders, though. Problem is, it keeps happening, keeps getting delayed, keeps trying to deploy and getting kicked back because something's hosed. Eventually I get sick of the back and forth and hand off the project - it was such a bad experience that I don't want to touch ACLs ever again. The current lead dev on the team took up the feature and has been working on it since. To this very day the ACLs are still breaking poo poo and blocking project delivery. Because we are passing those ACLs along, we break poo poo and block delivery. You can repeat that story for every other kind of configuration that goes on a networking device, and there are a lot. We are dumb. In reality, we are software engineers with responsibilities that require a non-trivial amount of network engineering experience. When I first joined, I was assured that I didn't need to be a network engineer - but it turns out that whoever's in charge of the software that configures a network should probably have network engineering experience! "What do you mean you don't know the default BGP configuration for <insert random hardware here>??? Why are you annoyed that we're submitting bugs with a chunk of vendor-specific switch config and nothing else???" We've regularly asked for assistance from actual network engineers, to which the response has been "no and gently caress you for asking". I am no longer interested in network engineering, I'd rather go back to fart apps. We are unreliable. See ACLs. We have a track record of submitting changes and fixes that have to get rolled back because something was misunderstood, some requirement was not properly communicated, something else broke on someone else's end, or some rando team isn't ready for XYZ feature. Stakeholders get mad, tickets get kicked back, sprint progress gets reverted, I lose confidence in my ability to deliver and understand the domain. We keep breaking things. The reasons why are complicated and incredibly stupid and thinking about them hurts my head, so I'll just summarize it with: code:We are the bearer of bad news. What our team does happens to be the nexus of many different stakeholders and services that I don't feel comfortable elaborating on. If anything goes wrong along this entire pipeline, it shows up in our service first. It's especially bad when something breaks further up this pipeline and we get pinged for a breakage that isn't our own! For example, we originally called out to an internal P4RT library as part of generating switch/router configuration. At some point, someone screamed at us because a change in this library broke a device (took it offline, caused it to fail a test, whatever), and we switched to a hardcoded version of the pre-change data. Then recently, the internal library team got mad because they assumed we were depending directly on that library and would be able to handle some new feature that needed it - turns out we couldn't cause the hardcode was too old! When we pinned ourselves back to the internal library, the tests broke again and we were interrogated about it - back to the goddamn hardcoded version.  So what the gently caress. A Rough Timeline 2021 Q4: Aside from my (first) manager, there's one person on the team. Dude's a total yes-man and just says yeah sure to whatever is asked of him, even if it's a bad idea, if some other team objects to it, or if it's something we shouldn't even own or be responsible for at all. This sets the baseline expectations and working relationships between our team and everyone else. I join, and now there's two people on the team. Then the other guy leaves, and now there's one person on the team.  Then, our team gets merged with another, similar team. Like four people total now or something. Okay, a reorg - I'm new, so I don't think much of it. Plus, I'm happy to get team members again! A member of the merged team leaves. Then our manager steps down and takes up the team lead position. Then our manager-turned-team-lead leaves and someone else is made team lead. I start getting nervous and wonder if I joined the wrong team. (We eventually get a new manager, he's cool and I get along well enough.) 2022 H1: Outside of our team, a brain drain as network engineering experts and project/product leaders announce their departures from our organization - including the original writers of the codebase I now have to speak for. Then our team lead announces he's leaving oh no wait, uh oh!!!!!!! Stock tanks, hiring and transfer freeze. Can't hire anyone and can't change teams. Current TL is un-transferring and starts showing up to our meetings again (he did not seem happy about this). I start experiencing the poor working relationship between our team and every other team as we get yelled at for breaking test devices during our efforts to prove our service works as expected. 2022 H2: Like three or four more reorgs. Our org changes names, leadership, reorganize a bunch. None of it really matters to me in terms of day to day work, so I don't care for the details. Our complaints for headcount and assistance result in a couple hires! I start communicating my experiences and frustrations with my team's manager, and he agrees with me. For once, I feel heard and have hope that this all might work out. Otherwise I don't remember this era very well because I don't fuckin know. Blame long COVID. 2023 Q1: In January 2023, we were assured that Google is perfectly capable of weathering the storm of recent economic troubles. Not long after, they fire a bunch of managers and directors in our org - including the manager I got along with. Then they lay off a bunch of people across the company. You might remember this from the news! The office's mood changes. Everyone on our floor looks sad and depressed, they still do today. The newbie engineers we hired in 2022 H2 look disappointed. I recall being kinda miffed about the move corporate pulled, at the time. In retrospect, I should have actually been ripshit pissed. Thennn I have a bit of a moment. See, one of the less-popular decisions management made for our team was for us to become co-owners of some old codebase with a vaguely similar role to our own and start reviewing PRs for it. There's very little overlap between the two codebases and none of our core knowledge transfers to it. I heavily disagreed with the decision, but kept my mouth shut. One morning, I get DMed by an owner from the old team in an absolute conniption because of some change to the codebase that he really really really disliked for some architectural reason and starts whining to me about how THIS IS REAAAAAALLY UNFORTUNAAAAATE. I get genuinely upset and start ranting about giving my two weeks because I'm sick of being blamed for everything, I'm sick of having to take responsibility for something I never wanted, etc. He and I quickly calm down, I apologize for my frustration and outburst, and we get to talking about how badly this handoff is going. He and I agree that things are Not Good, and I agree to pass word up the chain. I tell my new manager about my experience and how upset it made me to be personally fingered as a representative of our team's failure, and warn him about how badly we feel this handoff is going. He gives some handwavy response, and nothing comes of it. Cool beans. 2023 Q2: Burnout era. At this point, I've lost confidence in my team's higher-level management and the organization's competence at large. I now have a mental block on even thinking about the code or our product's design - basic requirements and features just roll off my brain, and doing my day-to-day work is like pulling teeth. The Current Situation Last Thursday, I had a one-on-one with my manager. It went like this: quote:i am concerned that you are not performing at an l4 level Look, let's not beat around the bush: there's no way Google isn't signaling further layoffs. This sounds like an early PIP warning. It's upsetting, but not that much of a surprise - managers probably got the order to start reducing headcount. My problem is that this is clearly implying that poo poo's bad because I'm bad, not that I'm bad because poo poo's bad. How personally, exactly, should I be taking this? There's two possibilities here: 1. My time on the team dealing with its bullshit burned me out, in which case my performance is a symptom and not a cause, orrrrrrrrrrrr 2. I am in fact such a terrible loving engineer that I singlehandedly brought this team down, in which case goddamn, how crap can our org be that one person can tank poo poo this badly? I AM ALL POWERFUL!!!!! TRANSFER ME TO ADS OR SEARCH SO I MAY SLAY THIS GIANT!!!!!!! VALHALLA AWAITS Either way, I'm tired. Get me out of this wack-rear end crystal prison. --- There's a lot more that I just don't have the energy to talk about right now, but if anyone has questions about exactly what the gently caress, I can share any details that wouldn't get me murked by Google lawyers.
|
|
#
?
Jun 25, 2023 01:50
|
|
”
|
jesus christ, im sorry pollyanna. i don't have any advice, that just sucks. feel free to hmu over video or corpchat or whatever if you want to vent about google-specific stuff that can't go on non-google networks. iirc you work with prod networking stuff that is way divorced from the network acl stuff my team works on, but also lmk if there's anything i can do that would help smooth inter-team friction stuff. if nothing else, i can talk to a manager on our side if there's stuff we should be doing to make your life better. but i think our teams don't actually have any contact with one another
|
|
#
?
Jun 25, 2023 02:07
|
|
|
That sounds lovely, best of luck. Sounds like you're in need of a new home that isn't a dumpster fire.
|
|
#
?
Jun 25, 2023 02:12
|
|
|
That sounds awful. I wish there was a way to measure “team chaos”, because I know that it would be inversely correlated with productivity, and it would be a way to show “shits not my fault.”
|
|
#
?
Jun 25, 2023 02:35
|
|
|
Jesus Christ, how can anyone work under those conditions. How does a company even *function* with that bullshit going on?? I’m real sorry to hear all that Pollyanna. Thanks for sharing.
|
|
#
?
Jun 25, 2023 03:01
|
|
|
Pollyanna posted:So what the gently caress. Did you go through three managers in a single quarter? 1) First manager 2) Reorg Manager 3) Whoever replaced Reorg Manager when they became team lead If so, anytime any asked why nothing works, just that fact alone was a good answer. Everything about that sounds awful. Nobody would be productive in that environment. Productivity nosedived across my current team when our new hire got fired and he'd only been on the team for less than a month. He wasn't as load bearing as a manager or a team lead. Edit: in case it's not clear, it's obviously answer 1. LLSix fucked around with this message at 03:11 on Jun 25, 2023 |
|
#
?
Jun 25, 2023 03:04
|
|
|
csammis posted:Jesus Christ, how can anyone work under those conditions. How does a company even *function* with that bullshit going on?? Sounds like half the jobs I've worked The only way to solve the cycle of suck is to change the process and tooling. Sounds like you guys are short a real network engineer and improperly staffed, good luck
|
|
#
?
Jun 25, 2023 03:24
|
|
|
It sounds similar to some tightened belt dysfunction I'm seeing. Over the past year, we keep inheriting codebases and systems that we don't know. Their stewards left or were laid off. We can't learn these areas by working in them, because everything needs to be tied to growth, and we don't get work that requires working in or improving these systems. We've lost half our team and cannot backfill. There's a lot of fire fighting that's happening, and everyone's burned out. There's been 4 reorgs in the past year. The industry seems like a more miserable place lately.
|
|
#
?
Jun 25, 2023 03:26
|
|
|
There's always the chance that I have misunderstood and therefore misrepresented the situation. Maybe I'm genuinely so dumb that I don't understand what's going on, poo poo, I'm fine with that. All I can claim is what I think I've seen and how I've felt these past 20 months. And I also resent being told, to my ears, that I'm to blame. All this is mostly just whining, to be honest. The core of it is that I'm burnt out and have had trouble delivering for a variety of reasons that include, but are not limited to, just losing focus and interest. I could talk about the codebase being difficult to understand, the blockages from vendors, having to rework my PRs for constant conflicts for a churning codebase, scoped work turning out to have weird timing and dependencies - but I don't think it matters in the end. Not enough delivery = drop 'em. LLSix posted:Did you go through three managers in a single quarter? Nah, sorry - to clarify: October 2021~March 2022ish?: first manager, dude who stepped down and went team lead then bounced later March 2022~like June 2022: I don't remember who was official manager at the time June 2022~January 2023: reorg manager (laid off) January 2023~now: successor to reorg manager (current manager) There was a hilarious amount of churn between October and March on the team. I am the only remaining member of the team I was originally on, whatever it is now. quote:Everything about that sounds awful. Nobody would be productive in that environment. Productivity nosedived across my current team when our new hire got fired and he'd only been on the team for less than a month. He wasn't as load bearing as a manager or a team lead. It was pretty awful and I've only gotten less enthused as time goes on. Frankly I'm considering just taking a break from tech altogether and riding out the bubble burst on my savings. Hadlock posted:Sounds like half the jobs I've worked Haha, no. They're short a real network engineer and improperly staffed. I don't have to go down with that ship. biceps crimes posted:It sounds similar to some tightened belt dysfunction I'm seeing. Over the past year, we keep inheriting codebases and systems that we don't know. Their stewards left or were laid off. We can't learn these areas by working in them, because everything needs to be tied to growth, and we don't get work that requires working in or improving these systems. We've lost half our team and cannot backfill. There's a lot of fire fighting that's happening, and everyone's burned out. There's been 4 reorgs in the past year. The industry seems like a more miserable place lately. That really loving sucks to hear. Honestly, there's not much an individual can do - they can only do what makes the most sense for them. Personally, I'd bail and let the decision makers deal with the fallout. Pollyanna fucked around with this message at 05:48 on Jun 25, 2023 |
|
#
?
Jun 25, 2023 03:30
|
|
|
the very notion of whining being a thing that grownups dont do is bullshit. if we didnt whine we wouldnt improve anything and you have a lotta valid casus whineii
|
|
#
?
Jun 25, 2023 03:37
|
|
|
You are not responsible for you being burned out, nor are you responsible for your inability to deliver while being burned out. Blame the system. Of course the system will blame you, gently caress it, it's not a person and you don't have to care what its mouthpieces think.
|
|
#
?
Jun 25, 2023 03:42
|
|
|
BIGPOST ALERT: My background is in large-scale live services crisis management, so I have a lot of respect for networking teams (and a lot of first-hand insight into how terribly they tend to be treated, too, unfortunately). I did some schooling in networking as well, so I learned enough (long enough ago) to understand how complex networking is. First, this sounds like a hilarious application of 'more devs is always better'; which I don't FULLY disagree with (In my time there, Microsoft was WAY too willing to fling contractors at problems that really should have had a small team of devs assigned instead for example), where you value development experience over domain knowledge. Pollyanna posted:My team also has a bad reputation: This just sounds like classic requirements failures, and any network engineering team worth it's salt should be the ones that should be screaming the loudest about it, not necessarily y'all. Networking is a complex, state-rich system with the most demanding uptime in your entire company, and thus requires real investment to work at scale. Not only should you have access to real networking engineers to work with, you should probably have some fulltime on your team at the bare minimum (or at least a handful of your devs having real concrete networking background). Pollyanna posted:We are dumb. In reality, we are software engineers with responsibilities that require a non-trivial amount of network engineering experience. When I first joined, I was assured that I didn't need to be a network engineer - but it turns out that whoever's in charge of the software that configures a network should probably have network engineering experience! "What do you mean you don't know the default BGP configuration for <insert random hardware here>??? Why are you annoyed that we're submitting bugs with a chunk of vendor-specific switch config and nothing else???" We've regularly asked for assistance from actual network engineers, to which the response has been "no and gently caress you for asking". I am no longer interested in network engineering, I'd rather go back to fart apps. Yup, this ties directly into my original point about 'just throw devs at it, how hard can it be?'. poo poo's complicated. A lot of networking is not really understandable to folks without domain knowledge at first blush and there's a lot of concepts that for good or bad reasons, are not clear. To put this another way: I have regularly had arguments with senior loving developers about how them getting a 500 error doesn't mean I need to page the entire networking org "just in case". (Before someone mentions it: No, we didn't use any layer 7 networking equipment; nothing in our network stack was emitting HTTP signals of any kind). The idea that just any dev should be able to quickly pick up much more complex networking concepts such as ACLs or WAN connectivity is loving baffling to me, given the real risk associated with failure to your company's bottom line. Pollyanna posted:We are unreliable. See ACLs. We have a track record of submitting changes and fixes that have to get rolled back because something was misunderstood, some requirement was not properly communicated, something else broke on someone else's end, or some rando team isn't ready for XYZ feature. Stakeholders get mad, tickets get kicked back, sprint progress gets reverted, I lose confidence in my ability to deliver and understand the domain. In a perfect world, this should be leading to actual changes in your process. Either better requirements development, better testing, etc. The management churn you're mentioning is probably contributing to this, but this is yet another example where this isn't really your fault for loving up; your team needs to be seriously sitting down and examining why these occur and producing action items to close that gap. (Again, not you, but your team.) Part of what really shocks me about this is that Google has produced a fair amount of work in this space, such as the O'Reilly Site Reliability Engineering book, although having worked at companies that size before, I am not entirely shocked that they aren't a hivemind and that some areas are worse than others. Pollyanna posted:We keep breaking things. The reasons why are complicated and incredibly stupid and thinking about them hurts my head, so I'll just summarize it with: Without knowing the details, this definitely sounds like your team doesn't have the backing from a senior/principal engineer who should be putting their foot down and going to bat for you on stuff exactly like this, because if someone asked me what the outcome of switching away from a shared library to hardcoded settings, I would have predicted the rest of your story because...well anyone should be able to. Another point: Being the bearer of bad news really requires that your org has the respect necessary to deliver that bad news, and that means both being able to deliver quality output (of whatever output is your requirement) as well as the backing of leadership to pull rank sometimes and say 'no you have to fix this or we're going upstairs'. Crisis management / safety engineering is the exact same boat there; I only talk to people when poo poo goes wrong, and I've been in orgs before where leadership only paid lip service to us, and so every discussion about 'hey that big outage is gonna happen again unless you do x' turned into 'welllll we really don't want to make time in the schedule to do x, so instead we're just not going to'. Pollyanna posted:So what the gently caress. Mostly emptyquoting this because it covers a lot. Pollyanna posted:<Timeline stuff goes here.> So, I don't know you from adam, and I can only take your story at face value and compare it to what I've seen in my time at other companies (never there specifically), buuuut: I really don't think this is your fault in any meaningful way. This is what can only be described as ongoing systemic failures of management, and since management is, as an entity*, almost entirely immune to being self-critical in any meaningful way, means that they're looking at the 'well we have to reduce cost and headcount' and instead of having hard talks about what has to be done to keep networking running, are pulling a Principle Skinner and have decided that "No, it's the engineers who are wrong." You can either try and ride it out with a sort of zen equanimity or you can polish up your resume and look elsewhere. In my advice? b]Get the gently caress out of there.[/b] * Individual managers can be, but in my experience any company big enough to get to Google's size, will never actually say "whoops we hosed up, sorry folks, we're going to try and make things better." It happens sometimes, (in fact my current job is in the recovery phase of one such massive fuckup that management is actually big on trying to avoid again) but it really requires poo poo crashing and burning hard enough that nobody wants to survive it. Edly posted:Sorry Pollyanna, what a shitshow. Sounds like your managers failed you; you're not supposed to have to figure all this out on your own. Oh yeah that's a good final note: If you were at some tiny rear end company and running into these issues...well, that might be the breaks. There might literally be nobody there who knows any better. But at scale, one of the biggest challenges any large company has to address is sharing best practices across the company. You don't want to be Google's size and having each individual org relearn how to loving manage. Falcon2001 fucked around with this message at 04:15 on Jun 25, 2023 |
|
#
?
Jun 25, 2023 04:12
|
|
|
Sorry Pollyanna, what a shitshow. Sounds like your managers failed you; you're not supposed to have to figure all this out on your own.
|
|
#
?
Jun 25, 2023 04:12
|
|
|
So, pollyanna, you guys are basically making docker for switches? aside from all the awful organizational stuff you mention, it sounds a lot like automation for automation's sake. none of which is your fault at all! Also, hello! <3
|
|
#
?
Jun 25, 2023 04:24
|
|
|
Falcon2001 posted:this sounds like a hilarious application of 'more devs is always better'; yes to all of this feel free to punch up on these points, be loud about it. You deserve better management, but at the same time you're responsible for your own happiness and if they're not doing their job you need to identify this for them, and grade them appropriately on it during your weekly 1:1
|
|
#
?
Jun 25, 2023 05:48
|
|
|
Pollyanna posted:2023 Q2: Burnout era. At this point, I've lost confidence in my team's higher-level management and the organization's competence at large. I now have a mental block on even thinking about the code or our product's design - basic requirements and features just roll off my brain, and doing my day-to-day work is like pulling teeth. Burnout is actually a medical condition. You might want to consult Google's policies on FMLA/disability. I had similar issues and talked to a therapist and they pulled me out for stress leave, and I got two months off with pay, and by the time I got back to work, my boss had already left the company.
|
|
#
?
Jun 25, 2023 06:27
|
|
|
Pollyanna posted:2022 H2: Like three or four more reorgs. reorg'ing every 2 months is just a thing that normal, healthy, deeply serious companies do
|
|
#
?
Jun 25, 2023 06:51
|
|
|
They should call the company Poogle!! Because it smells like farts!!!
|
|
#
?
Jun 25, 2023 07:56
|
|
|
This sounds very similar to the old fintech company I was just laid off from. Network automation is constantly breaking, new corp acquisitions every couple months, nobody knows how their own programs work, senior engineers who think http errors come from the network hardware or that someone should "check the database" instead of reading app logs. I've wanted a new job for a while, but the job had diminished my functioning to the point that I couldn't even get started on the search. That's not even touching on how unavailable I had become at home. Your situation is pure corporate dysfunction that's preventing you from using your skills. It sounds like it's putting you in a full-time dorsal vagal state, leaving you cognitively frozen all the time. That's a normal nervous system response, not a personal failing.
|
|
#
?
Jun 25, 2023 10:29
|
|
|
Falcon2001 posted:Part of what really shocks me about this is that Google has produced a fair amount of work in this space, such as the O'Reilly Site Reliability Engineering book, although having worked at companies that size before, I am not entirely shocked that they aren't a hivemind and that some areas are worse than others. Old Google: Hire generalists, understand they need time to learn the domain New Google: Hire generalists, discourage them from spending time on anything other than the direct production of artifacts, gotta keep that productivity evidence up We productivity over impact now. cum jabbar posted:senior engineers who think http errors come from the network hardware or that someone should "check the database" instead of reading app logs. Holy gently caress. Were these real seniors or Leetcode "seniors" who studied system design and passed the senior interview with like 3yoe? oliveoil fucked around with this message at 11:50 on Jun 25, 2023 |
|
#
?
Jun 25, 2023 11:45
|
|
|
Pollyanna posted:
This part stands out to me as a real risk depending on your manager. Some are hesitant to make waves to address disfunction, and likely more so if they’re expecting layoffs soon. It sucks obviously, but doesn’t mean you won’t be successful elsewhere
|
|
#
?
Jun 25, 2023 12:37
|
|
|
oliveoil posted:Holy gently caress. Were these real seniors or Leetcode "seniors" who studied system design and passed the senior interview with like 3yoe? People who had spent many years stagnating, for the most part
|
|
#
?
Jun 25, 2023 13:44
|
|
|
This response is…not quite what I expected! It really is that bad, huh?  Next steps are pretty clear - bounce. Either to a different team, or just hit da bricks.cum jabbar posted:People who had spent many years stagnating, for the most part This is what I’m afraid of happening to me if I remain on my team. Assuming I don’t just get let go. I genuinely think I’ve become a worse engineer during my time here, not a better one.
|
|
#
?
Jun 25, 2023 14:47
|
|
|
Pollyanna posted:This response is…not quite what I expected! It really is that bad, huh? http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-dead-sea-effect/ quote:Instead, what happens is that the more talented and effective IT engineers are the ones most likely to leave — to evaporate, if you will. They are the ones least likely to put up with the frequent stupidities and workplace problems that plague large organizations; they are also the ones most likely to have other opportunities that they can readily move to.
|
|
#
?
Jun 25, 2023 15:15
|
|
|
|
| # ? May 26, 2024 02:51 |
|
|
…yeah that’s exactly what happened, isn’t it.  Better late than never, I guess 
|
|
#
?
Jun 25, 2023 15:25
|
|