|
My understanding from talking to AWS employees in the past is 1) AWS gives a LOT of their comp via stock, backloaded into like year 3 and 4 at the company. If you stay on you will get refresher grants but it may not hold up to that initial windfall. And the stock is down a lot (by their standards) this year 2) Once you get in with one of the big tech companies, you're a hot commodity and can start seeing ludicrous offers to jump ship. Also the current tech job market is just very favorable in general so you can probably get a raise by moving although I'd be interested to hear more from current Amazon goons if they're able to share.
|
#
¿
May 16, 2022 18:54
|
|
|
|
| # ¿ May 22, 2024 10:51 |
|
|
You will also come to learn that there is no consistency between the API various AWS services present. One might have a get operation where another has describe. It may use a name as the attribute to identity a resource, or an arn. It might let you update an existing resource or force you to delete and recreate. Makes working with each new service an adventure!
|
|
#
¿
May 18, 2022 12:34
|
|
|
22 Eargesplitten posted:I asked if we had a contractual RPO or RTO for our customers and haven't heard back yet. I'm also not sure how far back we might be contracted to be able to retrieve something. I'm going to need to get answers to that for sure. It's also worth mentioning that depending on the nature of your company and where it does business, there may be various regulations you need to follow related to backups. Who can access them, whether they're encrypted at all times (and who can decrypt them), who has access to restore to production and how, retention period, where they're stored, whether they contain PII or credit card data, auditing of success/failure and documentation of what is to be done when a job fails, etc etc. This is a conversation to have with your manager, or with legal/finance if they glaze over when you ask about it. Getting kinda offtopic for the AWS thread, though.
|
|
#
¿
May 18, 2022 15:47
|
|
|
Hughmoris posted:Am I correct in stating that to use a GUI with Amazon RDS (e.g. SSMS, MySQL Workbench), I either need to make the RDS Instance publicly accessible OR connect via AWS VPN or Direct Connect? You’re right that putting a database on a public IP isn’t a great idea. You could set up the security group to only allow your home IP or something but mistakes happen, and it would be annoying to update it every time your IP changes. Setting up a very locked down “bastion host” like you did to ssh tunnel through to your private resources is a decent option. Edit: lol I see this was already answered multiple times as I wrote this up
|
|
#
¿
May 30, 2022 20:13
|
|
|
Yeah the python boto3 library is excellent. I don’t think you’re hamstringing yourself at all using python to interface with AWS.
|
|
#
¿
Jun 6, 2022 00:10
|
|
|
It's also cheaper to run bare EC2 instances than run RDS. But this argument falls apart pretty quickly if you put any value at all on your own time spent managing those instances, because operating databases loving sucks. And while it's true that certain things work differently or are disallowed in RDS, the vast majority of apps will be just fine. As you branch out from standard mysql/postgres/mssql in RDS to Aurora, or Aurora serverless, etc. then the number of caveats grow. Docjowles fucked around with this message at 21:31 on Jun 14, 2022 |
|
#
¿
Jun 14, 2022 21:27
|
|
|
BaseballPCHiker posted:Oh god this is my company to a T. I cant wait to hit a year and bounce so I dont have to pay back my sign on bonus. Management: it’s ok we just need to get the move done asap so we can terminate our data center lease and avoid paying for the next hardware refresh. We’ll optimize later Also management: what the gently caress do you mean you want another 9 months to rewrite the app to use s3 and containers vs oversized EC2 instances with a petabyte of high iops EFS volumes. We just spent a year moving to the cloud it’s fine
|
|
#
¿
Jun 20, 2022 14:45
|
|
|
For my money it’s some very basic info on how networking and/or dns work. I’ve had some absolutely  conversations with senior devs where you suddenly realize that what they’re trying to explain only makes sense if their world view about how two computers communicate is totally and fundamentally broken.
|
|
#
¿
Jul 12, 2022 01:11
|
|
|
Falcon2001 posted:I've worked at huge tech companies and I can confirm that the number of senior devs who believe that network is basically magic is concerningly high.  My passive-aggressive, not-real-because-I-want-to-remain-employed answer would be "how to read error messages". Because Jesus Christ the number of tickets and questions I've dealt with in my career that were resolved by simply reading and comprehending the text that was blindly spammed into the ticket at the first sign of trouble is astronomical. Sometimes it requires breaking a sweat, like pasting the error message into Google or in extreme cases going to the software's Github readme or issue tracker. But often it's just literally reading the text of the error and fixing what it says is wrong.This is fine when it's a non-technical user hitting up the help desk. That's why a help desk exists. I feel like it's less fine when it's an engineer making deep into six figures grinding work to a halt and escalating to peers with Ops expertise because "I am trying to call the password reset API endpoint /pussword and I get a 404 error please advise @ALL_MANAGERS"
|
|
#
¿
Jul 12, 2022 03:37
|
|
|
necrobobsledder posted:Having to tell frontend devs how to read network traces in developer tools and the difference between timing out, active denial, and DNS lookup failure seems embarrassingly condescending and patronizing in most circumstances but having had to do it at almost every company I’ve been at reluctantly I think developers that don’t understand networking when working on uh… web applications is a handicap for day to day work. I don’t expect software devs to know what the three way handshake or what the OSI model is, but I do want to know that they won’t be asking me to help them figure out why their application on their laptop can’t be accessed from the public Internet (and for that I just point people to ngrok anyway). Yeah this is the level of detail I am talking about. I am not asking devs to pass the CCNA. I would be elated if they understood the significance of an IP address that starts with 10.x and one that doesn't. What a private vs public subnet means in AWS. And, yes, the significance of a DNS lookup failure. Not even how to troubleshoot it, just that it's a distinct failure mode. I don't feel like this is "embarrassingly condescending and patronizing" at all, devs I've worked with across several companies struggle with this.
|
|
#
¿
Jul 12, 2022 04:28
|
|
|
Internet Explorer posted:By all means, do what you want, get paid, play with fun stuff, add resume bullet points, but I'll just say this - Tbh this is every business. Everyone thinks they are a snowflake. Never worked in law but I’ve dealt with plenty of cases of “well I know there is a free open source solution that does 98% of what we need. But it lacks this one specific feature so gently caress it we’re building our own from scratch!!!” And you get the one feature but it’s significantly worse in every other way. Plus now you’re stuck maintaining some bespoke piece of poo poo forever that you can’t just Google answers for when the guy who wrote it leaves. The company where I’ve been for a while now had a huge bias for build over buy that is finally changing over after some changes in engineering leadership. It’s so refreshing.
|
|
#
¿
Jul 14, 2022 15:05
|
|
|
If it’s any consolation networking in the cloud seems much simpler than traditional enterprise networking. There’s so much stuff you don’t have to care about. Like layer 2 just kind of isn’t a thing. Which is great, because later 2 sucks  I’m sure it feels like a lot if you’re coming from zero networking background, but it’s not that bad. The clouds really do insulate you from a lot of the esoteric poo poo.And once you get it set up it mostly just runs as others said. The biggest hassle is if you are forced to run a hybrid network connecting back to data centers with direct connect. Cause then you suddenly do have to care about the baggage of the last 40 years of networking again. For me the worst part is optimizing your bandwidth costs, or even understanding them. AWS at least does not make this easy or cheap out of the box. Everyone eventually has the “wait I have been paying these assholes how much to get my data in and out of S3 when I could have been paying $0 to use a VPC endpoint?” moment.
|
|
#
¿
Jul 16, 2022 02:37
|
|
|
Jeoh posted:Brown bag lunch sounds like you're downing a bottle of vodka at the office Just lol if you aren’t drinking a 40 of King Cobra out of a bag at morning standup I didn’t realize brown bag had a racial history, so thanks for the comment
|
|
#
¿
Jul 17, 2022 01:58
|
|
|
Internet Explorer posted:Brown bag implies you are bringing your own lunch. Lunch and learn implies lunch will be provided, at least to me. I mean, lunch should always be provided for lunch meetings, but yeah. I guess maybe that's changing with remote work being the norm in our industry? My first thought was exactly the same as yours but agrikk's post drove me to Google and apparently this was a thing https://en.wikipedia.org/wiki/Brown_Paper_Bag_Test. Which, as with most things in American history, was way more horrible than I expected. As a white dude I personally don't give one single gently caress if it's called a brown bag or a lunch and learn so if there are people actually being harmed by the brown bag term I'm cool with changing for their sake. Docjowles fucked around with this message at 06:17 on Jul 17, 2022 |
|
#
¿
Jul 17, 2022 06:14
|
|
|
Internet Explorer posted:I'm very familiar with the brown bag test. But we still call them brown bags. I dunno. I'm glad to be having this discussion. My feeling is it costs nothing to swap to a term that has no hurtful connotations for anybody. So even if you feel like the number of people who would notice or care is low, why not. The industry is swapping from blacklist to denylist etc, this is part of that
|
|
#
¿
Jul 17, 2022 06:30
|
|
|
I can't just react to a post on this old dumb forum so
|
|
#
¿
Jul 17, 2022 06:41
|
|
|
I feel like every few months some smart person I follow on Twitter starts working at Honeycomb. Starting, obviously, with Charity. Certainly seems like a good place to be associated with.
|
|
#
¿
Jul 19, 2022 23:33
|
|
|
The siren song of s3fs is loving irresistible to people working with AWS early on. Especially ops people. It’s like the first stop on the “No seriously the cloud is not the place to just rebuild all your lovely patterns from the data center” journey. I’ve had to push back on that a lot as a component of a design. If you’re looking for a quick replacement would AWS Transfer Families work for you? It’s basically a managed service with SFTP on the front end and S3 on the backend. It’s kind of expensive but you also aren’t supporting an abandoned rube goldberg machine anymore which has value. https://docs.aws.amazon.com/transfer/latest/userguide/create-server-sftp.html
|
|
#
¿
Jul 26, 2022 04:51
|
|
|
I am falling out of love with terraform more all the time. It was a revelation when it came out and I was a die hard Terraform fan for quite a while. But even after years of improvement and passing 1.0, MAN is HCL still kind of awkward and lovely to work with. Any kind of non trivial loops or conditionals are just the most tortured crap. You can “fix” it by wrapping it with something else but some point, I would just rather be using something else entirely than papering over shortcomings. I haven’t played with it much yet but I know a number of CDK true believers. Haven’t met any Pulumi users in the wild.
|
|
#
¿
Aug 19, 2022 04:17
|
|
|
12 rats tied together posted:- S3 storage, there's usually at least one s3 bucket named after the org that has an order of magnitude more data than it should. I just had to pull this out for appreciation. There is always a bucket called $companyname and it is always like the first thing some rando dev ever did in AWS years before the rest of the org thought about using the cloud. It will be a giant dumping ground of poo poo with no lifecycle policy and probably leaking PII. There will also be some horrible reason you can't just easily fix it.
|
|
#
¿
Aug 23, 2022 16:18
|
|
|
Since we were talking about managing AWS costs recently, today in "lol. lmao.": Senior leadership has a goal to port the vast majority of our stuff out of the data centers into AWS. Naturally, after getting started on this, finance poo poo a brick when they saw the monthly bill. So they forced us to engage a third party consultant to analyze our spending and suggest improvements. It turned out that a handful of our accounts, for various bad reasons, had multiple CloudTrails set up. This was absolutely a lurking mistake, but had not incurred any meaningful cost before. And (I dunno if this is a bug or what) certain resources like SNS topics cause the tool this vendor uses to make roughly 500 ho-jillion API calls per day. Which we were now paying to log because of the aforementioned duplicate CloudTrails that go above and beyond the free tier. And of course, it was the couple accounts that had SNS topics with a zillion subscribers that also had extra CloudTrails. I caught it fairly quickly but still this "free POC" ended up costing us five figures in loving CloudTrail charges of all things. They did not mention this 1000% increase in CloudTrail spending at all in their final report. Or really provide anything of value, it was basically "you could pay us to micromanage RIs and spending plans for you if you want". lol. lmao. The only silver lining is that I feel pretty good about all the cost optimizing work I've been leading. We can do better of course but they couldn't make a compelling case for themselves above and beyond what I've been doing.
|
|
#
¿
Aug 30, 2022 23:36
|
|
|
Anecdotally from a couple people I know at AWS, Amazon has been at least doing some stuff lately to make people whole if they get turbo hosed on RSU value (mostly granting even more stock, lol). Because lord knows if those golden handcuffs get loose, the other FAANG companies will happily poach talent. Historically this has not been a problem because AMZN stock only went up. You’d be insane to leave when you had a large vesting coming up that had doubled in value since your start date. But in the face of a downturn or recession, being so heavy on stock comp isn’t the guaranteed jackpot it has been. Not to make this sound like all doom and gloom. You’re still probably making vastly more at Amazon than most non-FAANG places.
|
|
#
¿
Sep 4, 2022 19:58
|
|
|
I don’t think I’ve ever seen more divisive opinions on what it’s like to work at a place than Amazon. There are plenty of takes like yours, that it’s a low paying burnout factory. Then I talk to people who have been there going on 5-10 years (so way after the initial churn and burn phase and grant cliff) and think it is the greatest job going. From the outside it’s pretty interesting.
|
|
#
¿
Sep 4, 2022 20:48
|
|
|
lazerwolf posted:I have an architecture question. I have a Django REST api running on ECS. I feel there is something off with the configuration because the instances start crashing when 1000 concurrent POST requests to the api occur. I do have autoscaling enabled but I’m not sure if the instances scale fast enough or there is something else going on. I think you're on the right track. If the work can be changed to a model where requests are posted to a queue and consumers pull items off to process, that is going to be infinitely more scalable than trying to handle them synchronously. SQS and SNS are your friends here, if you're fine being AWS native. If that is not possible, we are going to need more specifics about the failures. Error messages, what resource is being exhausted. Docjowles fucked around with this message at 05:07 on Sep 5, 2022 |
|
#
¿
Sep 5, 2022 05:04
|
|
|
Is the lambda in a different AWS account than whatever writes the objects to the bucket? You might need to mess with object ownership settings, such as turning on “bucket owner enforced”.
|
|
#
¿
Sep 20, 2022 20:23
|
|
|
ledge posted:Anyone else's day being ruined by the us-west-2 API gateway outage? 3 hours and counting now. Nope because all my stuff is in us-east-1  Do not ask me how many other days have been ruined by that fact
|
|
#
¿
Sep 29, 2022 00:40
|
|
|
+1 if you don't actually need a traditional database, don't use one. DynamoDB or S3 + Athena could end up costing pennies compared to Aurora.
|
|
#
¿
Oct 3, 2022 05:17
|
|
|
Pile Of Garbage posted:Just want to say that this makes a massive difference. Earlier this year I was in a lovely role with an even shittier manager that just stressed me to no end. Then finally I moved into my current role where I've got a competent and chill manager plus a great director which has honestly made a world of difference. This is universally true, not just Amazon. At my last job I was actively looking to get out because my manager was absentee most of the time and incompetent to actively harmful when he did bother trying to steer the team. He eventually left and was replaced by one of the best managers I've ever worked with. I ended up staying several more years, getting promoted twice (previous manager had never even indicated to me that there were job titles above my level lol), and moving into management myself. It's unreal the difference a good manager makes in your experience at a company. Yet there's so little effort put into making sure managers are adequately trained, or even want to be in that role at all vs just taking it cause it's the only path to higher pay.
|
|
#
¿
Nov 3, 2022 22:10
|
|
|
Yeah AWS SSO is quite cool and good these days. Really my only gripe (and this applies to regular old IAM too) is that you still can’t be logged into more than one console at once. So if I want to look at things in more than one account simultaneously I have to open incognito windows or multiple browsers or some poo poo. Which is very awkward. I don’t know how they would reasonably make that work, though.
|
|
#
¿
Nov 28, 2022 22:13
|
|
|
ledge posted:I have multiple chrome profiles, one for each AWS account I have to work on. Slightly painful, but it keeps me sane and means I only have to login once per day for each account I'm working on that day. What is it like having an IT group that can correctly configure MFA to not reauthenticate you 69,000 times per day? I feel like I spend half my life responding to Duo prompts
|
|
#
¿
Nov 29, 2022 08:48
|
|
|
LtDan posted:Are there any situations where a reserved instance will also reserve capacity or is that entirely done by capacity reservations? I believe if you buy an RI for a specific AZ within a region it reserves capacity. If you buy a more generic one for the region, it does not. https://aws.amazon.com/premiumsupport/knowledge-center/ri-reserved-capacity/
|
|
#
¿
Dec 2, 2022 22:16
|
|
|
jiffypop45 posted:Bots required manual approval via a team that had like 5 people and there was no process in place for that or many other things. There was no way to join private channels without being added by hand. FC/contract employees were unable to use slack. Emojis gave people with epilepsy seizures at least twice due to bad internal actors. Amazon tried to make retention two weeks likely to cover something up only to brown out slack globally when they rolled it back after hiding whatever they sought out to hide. 
|
|
#
¿
Dec 17, 2022 00:31
|
|
|
Yeah the Graviton chips are absolutely not “raspberry pi in the cloud” lol. They are great price and performance options if you don’t mind doing the work to get your workload on there. And you can use them as nodes in an EKS cluster. I feel like you may not be up to speed with what Amazon is doing with ARM. Agree that abandoning containers to return to fleets of bare EC2 nodes is probably a terrible idea though.
|
|
#
¿
Dec 21, 2022 06:42
|
|
|
Also worth noting that you can run a lot of AWS managed services on graviton like RDS, elasticsearch, elasticache. And there’s no reason not to use them really since they are cheaper and Amazon handles all the compatibility work.
|
|
#
¿
Dec 21, 2022 16:28
|
|
|
Hughmoris posted:I heard that QuickSight was the neglected child of AWS services and your discovery certainly doesn't dispute that... I've only used QuickSight to deploy the CUDOS cost dashboards but it sucks enough to make me wonder why it exists. Are there real people who evaluate it next to any other BI product and go "gently caress yeah we gotta have this"? It's fairly cheap I guess if you have minimal requirements. beuges posted:Wouldn’t you use an api to access the db rather than access the db directly from the app? Regardless of whether the db was on a private subnet or available directly over the internet Yes, please in the name of all that is good, do not write an app that opens direct, synchronous connections from a phone to a database. DB's are not designed for high latency, slow, unreliable links or huge numbers of simultaneous connections. It's also an unnecessary security risk putting it directly on the internet. You want a a backend application that your mobile app talks to over HTTPS, which in turn queries the DB and returns the results. Speaking very broadly you want to write a lightweight frontend that sends requests and renders/reacts to the results, and put as much business logic (I'll include "dealing with behind the scenes infrastructure bullshit like databases" here) as possible on the remote backend. This also gives you more freedom to change stuff about your app without waiting and praying for users to update it on their devices.
|
|
#
¿
Jan 12, 2023 07:41
|
|
|
luminalflux posted:Yes, we run tflint as part of pre-commit-hooks for the repo. We also run pre-commit in CI on all our repos to ensure that someone doesn’t miss running it. Branch protection in GitHub means that you need a PR to merge and checks must be green to merge. Yeah we actually do all of this stuff too. https://pre-commit.com makes sure various linters run before poo poo even gets committed, let alone to the code review stage. Pedantic assholes arguing about the number of spaces or whatever is such an obnoxious waste of time, just make a tool enforce it. And I enjoy Atlantis too. Docjowles fucked around with this message at 21:03 on Jan 28, 2023 |
|
#
¿
Jan 28, 2023 21:00
|
|
|
IIRC your account hangs around for like 90 days just in case you change your mind. Bunch of fine print here https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/close-account.html
|
|
#
¿
Apr 8, 2023 00:15
|
|
|
fletcher posted:Anybody using AWS Controllers for Kubernetes ? Disclaimer: I have not personally used ACK But reading the page, I think you're mistaken as to the purpose of ACK. It is not for building and managing k8s clusters. It's a way to let you manage AWS resources (RDS databases, Lambda function, S3 buckets, etc) using k8s YAML files. It defines a bunch of custom resource types. You write configs using those types and upload them into your cluster. Then the controller will go out and manage those things on your behalf the same way it works for pods and services and ingresses. I do not know who in god's name is asking for this. But if you simply must manage your entire infrastructure via k8s YAML files and nothing else then ACK's got you covered? quote:Kubernetes applications often require a number of supporting resources like databases, message queues, and object stores. AWS provides a set of managed services that you can use to provide these resources for your apps, but provisioning and integrating them with Kubernetes was complex and time consuming. ACK lets you define and consume AWS services and resources directly from a Kubernetes cluster. It gives you a unified way to manage your application and its dependencies.
|
|
#
¿
Apr 13, 2023 20:32
|
|
|
CDK still ends up emitting cloudformation. So if you really hate CFN and insist on using Terraform to do all of your provisioning, that's where CDKTF comes in. There could be good-ish reasons for that. Like an ops group was previously managing all infrastructure via Terraform and had invested a lot in training and tooling and processes. Now management wants application devs to start writing their own infrastructure code, but they only know JavaScript and can't be assed to learn HCL. CDKTF bridges the gap and nobody has to also learn CFN. This seems pretty niche, and if you google cdktf basically all the content is from HashiCorp themselves ( ), but I can at least understand the use case.It is also kinda cool that it lets you write CDK style code targeting Azure and GCP. I agree with you that nobody is using Terraform to effortlessly swing workloads around between clouds on a whim. That was always moronic marketing hype. But at a big enough company you will absolutely end up running something on at least 2 if not more cloud providers because some group absolutely HAS to use Google BigQuery or Azure OpenAI or whatever for $reasons. Even if AWS is your primary provider. And when that happens it's nice not to have to learn some totally new thing to manage it. fluppet posted:Anyone using zesty to manage reserved instances or is there a better alternative? no but my kids have some nice stuffed animals with their logo on them from reinvent lol Right now our management of RI's is very manual and bad so I would love to get a vendor involved but my boss feels like it will erode too much of the savings Docjowles fucked around with this message at 23:12 on Apr 19, 2023 |
|
#
¿
Apr 19, 2023 23:10
|
|
|
|
| # ¿ May 22, 2024 10:51 |
|
|
It looks like they basically copy/pasted the service and pricing from AWS, lol. It's almost identical. All the cloud vendors shamelessly rip each other off constantly so this is not surprising. The cost is an oof although at the scale our bill is these days it's not even a rounding error so whatever. I prefer it to what we had to do in the Before Times where a fragile script would periodically dump every zone/record in our local BIND servers and mirror them to Route53.
|
|
#
¿
Apr 27, 2023 21:38
|
|