|
CarForumPoster posted:TIL I can get a desktop/GUI out from an AWS Lambda based docker container image. This makes diagnosing why some webscrapers are having issues much easier and maybe someone in this thread needs to know this. Interesting. Are you RDPing or shoving X down the pipe (lol that latency would be funnn)?
|
#
?
May 12, 2022 01:24
|
|
|
|
| # ? May 21, 2024 12:22 |
|
|
Hed posted:Interesting. Are you RDPing or shoving X down the pipe (lol that latency would be funnn)? I access it through a browser w/vncserver and noVNC. I’m only doing this locally. I can post the dockerfile bits tomorrow if anyone’s interested. One annoying thing though I have to use an intermediate site in a browser to copy paste.
|
|
#
?
May 12, 2022 03:48
|
|
|
Does anyone use AWS Step Functions for anything, and if so, can you speak a little to your thoughts on it? On the surface it sounds cool but I'm trying to think of a fun little personal project to solve with it.
|
|
#
?
May 12, 2022 05:19
|
|
|
Hughmoris posted:Does anyone use AWS Step Functions for anything, and if so, can you speak a little to your thoughts on it? I do a lot of ETL work with Step Functions. I think of it like serverless batch jobs, where each command in a batch job is replaced with an atomic Lambda function (or AWS Batch call). A log file lands in an S3 bucket which triggers a lambda function to strip special characters (or whatever) which triggers an SNS topic for job success/failure and triggers another Lambda to convert the log file into CSV (or JSON or whatever) which triggers an SNS topic for job success/failure and triggers a Glue Job which triggers an SNS topic for job success/failure and triggers a DMS job which triggers an SNS topic for job success/failure or something like that.
|
|
#
?
May 12, 2022 05:54
|
|
|
Hughmoris posted:Does anyone use AWS Step Functions for anything, and if so, can you speak a little to your thoughts on it? It’s pretty needs-suiting for native services workload automation. The state machine language can get a little complex if you try to do some of the more advanced options but it’s nice to be able configure/monitor disparate services.
|
|
#
?
May 12, 2022 12:04
|
|
|
Hughmoris posted:Does anyone use AWS Step Functions for anything, and if so, can you speak a little to your thoughts on it? I use them to limit the parallelization of my web scrapers and to do some of the business logic for the steps Eg scrape website->are there leads?->enrich leads->assign sales person->add to crm
|
|
#
?
May 12, 2022 12:17
|
|
|
To get selenium + chrome going in a lambda based docker container, check out: https://github.com/umihico/docker-selenium-lambda I use the shell scripts from gui-docker here: https://github.com/bandi13/gui-docker to get everything running. Then to get a desktop GUI out from the local docker container + install some helpful things like a text editor you can: code:Obvs this is a thing to be run locally only.
|
|
#
?
May 12, 2022 19:09
|
|
|
Thanks for the Step Functions ideas/info. My background is data stuff so I might try a small ETL project utilizing SF.
|
|
#
?
May 13, 2022 04:15
|
|
|
Internally at AWS we often use Step Functions state machines to implement asynchronous API operations.
|
|
#
?
May 14, 2022 23:32
|
|
|
Adhemar posted:Internally at AWS we often use Step Functions state machines to implement asynchronous API operations. Huh I figured the folks doing it would bias towards simple workflow service instead. Also sad I just thought to read this thread as my last day at AWS was Friday.
|
|
#
?
May 15, 2022 04:06
|
|
|
Tulenian posted:Huh I figured the folks doing it would bias towards simple workflow service instead. SWF is also used a lot in more old school teams. I’m all about that serverless life though. I’m probably leaving soon too…
|
|
#
?
May 15, 2022 07:16
|
|
|
Why are you guys leaving? Image always heard turnover there is stupid high
|
|
#
?
May 15, 2022 12:25
|
|
|
For me it was disagreements with things going on with the non-AWS side of the house.
|
|
#
?
May 16, 2022 07:09
|
|
|
Money.
|
|
#
?
May 16, 2022 07:34
|
|
|
Adhemar posted:Money. Not emptyquoting
|
|
#
?
May 16, 2022 15:01
|
|
|
Adhemar posted:Money. Does AWS not pay well or just typical have to move to a different company to get a big bump in $$?
|
|
#
?
May 16, 2022 18:35
|
|
|
Scrapez posted:Does AWS not pay well or just typical have to move to a different company to get a big bump in $$? Yea same ? AWS took my 3rd year CS student intern with a good GPA from a no name school at $33/hr. That's top of the pay scale and then some around here.
|
|
#
?
May 16, 2022 18:49
|
|
|
My understanding from talking to AWS employees in the past is 1) AWS gives a LOT of their comp via stock, backloaded into like year 3 and 4 at the company. If you stay on you will get refresher grants but it may not hold up to that initial windfall. And the stock is down a lot (by their standards) this year 2) Once you get in with one of the big tech companies, you're a hot commodity and can start seeing ludicrous offers to jump ship. Also the current tech job market is just very favorable in general so you can probably get a raise by moving although I'd be interested to hear more from current Amazon goons if they're able to share.
|
|
#
?
May 16, 2022 18:54
|
|
|
I’ve been with AWS for almost a year now and my comp is well above anything I had hoped to ever earn in my career. That being said, I know folks at my level who have been with AWS for some time make a lot less. AWS seems a lot better about making good initial offers than keeping existing employees happy. It’s possible in two or three years I might be below market and need to jump ship but I’m happy for now. It’s also true RSUs are heavily backloaded which leads people to leave after their initial grant finishes vesting in year four.
|
|
#
?
May 16, 2022 19:16
|
|
|
Amazon's stock compensation is weighted as follows: 5% granted on the 1y anniversary 15% granted on the 2y anniversary 20% every six months after that. So for someone hired 2 years ago just entering their 3rd year of tenure, their stock-based compensation (which could be 50% or more of their total comp) just took a 25% dive. Add to that the fact that 1) compensation changes happen once a year, 2) are frequently heavily weighted 2 years out, 3) Amazon is one of the lowest compensating of the FAANGs (gotta love that frugality leadership principle) and 4) new hires get paid current market rate but tenured peoples' comp lags quite a bit... Well, job hopping starts to look pretty dang attractive
|
|
#
?
May 16, 2022 19:46
|
|
|
I’ve been with AWS almost 8 years and the initial comp is good/great if the stock price goes up. However you can pretty much always jump to Microsoft/Google for another $50K/yr in normal times for senior technical roles and come back to Amazon if you like it for another big new hire set of shares. Now that the stock price has fallen sharply just after yearly refreshers that number is closer to $100K so a lot more people are looking.
|
|
#
?
May 16, 2022 20:02
|
|
|
I'm lazily studying for the SAA cert and poking about different services looking for fun little projects to build and learn on. The one thing that keeps hitting me in the face is that I really need to just hunker down and first get a better understanding of how IAM and AWS Networking work. Those topics are a bit of a snoozefest for me (and I'm guessing lots of people) but they seem to be crucial to accomplish anything meaningful with AWS services. I know that's likely obvious to anyone with cloud experience but it's definitely a challenge for me, coming from a data background. Also, holy poo poo at this AWS post highlighting the fact that AWS IAM was hitting 400 million API calls per second last year. Hughmoris fucked around with this message at 03:05 on May 17, 2022 |
|
#
?
May 17, 2022 03:03
|
|
|
Scrapez posted:Does AWS not pay well or just typical have to move to a different company to get a big bump in $$? I was going to effort post but it’s already been explained pretty well in the last few posts. I joined in 2016 so my cliff was 2020. My compensation for 2022 has dropped 40% compared to 2020. I’m way below the range for my role and the correction for that is way too slow. So I don’t really have a choice. I like my job but can’t afford to stay.
|
|
#
?
May 17, 2022 04:21
|
|
|
You might be able to get a proactive dive and save (without another offer in hand) depending on your manager and org, but it might be just as easy to take another job and boomerang if you want. #pay-equity and #pay-equity-discussion are good resources if you're not aware of them.
|
|
#
?
May 17, 2022 15:31
|
|
|
How often do you use the AWS CLI versus CloudFormation versus Console? My goal is to build good practices while I'm self-learning, in eventual hopes of employment using AWS. I.e. I'm walking thru a tutorial creating a VPC, subnets, IGW, SGs etc... I'm clicking thru the console but I know that isn't ideal. Should I be using the AWS CLI for these small spinups and save CF for multi-day projects? Or should I get in the habit of using CF and learning how to build templates for anything and everything? And please forgive me for peppering this thread with low-level questions, I don't know anyone that uses cloud tech.
|
|
#
?
May 17, 2022 20:47
|
|
|
Happiness Commando posted:You might be able to get a proactive dive and save (without another offer in hand) depending on your manager and org, but it might be just as easy to take another job and boomerang if you want. #pay-equity and #pay-equity-discussion are good resources if you're not aware of them.
|
|
#
?
May 17, 2022 20:51
|
|
|
Hughmoris posted:How often do you use the AWS CLI versus CloudFormation versus Console? My goal is to build good practices while I'm self-learning, in eventual hopes of employment using AWS. Generalization: Infrastructure as code (CFN, tf, etc): orchestration and managing resources you’re going to depend on over time and/or recreate ever. Console: Exploration, sometimes quick breadcrumbing connecting resources to view relationships; trying new services out before coding them AWS CLI: Mostly ad-hoc info gathering - “gimme a list of untagged ec2 instances”; sometimes bootstrapping or actions in a pipeline somewhere SDK: Building tools or automation to glue poo poo together Spoken from the perspective of platform engineering / SRE and slightly over-generalized but it’s how I tend to think about them
|
|
#
?
May 17, 2022 21:05
|
|
|
Hughmoris posted:How often do you use the AWS CLI versus CloudFormation versus Console? My goal is to build good practices while I'm self-learning, in eventual hopes of employment using AWS. I use CFN for everything, the CLI for sts:AssumeRole and s3 actions, and the web interface only for cloudtrail, cloudwatch, and examining CFN changesets. I would recommend building templates for everything.
|
|
#
?
May 17, 2022 21:20
|
|
|
That all makes sense. I'm halfway comfortable with basic CLI and Console use so I'll focus on better developing my IaC skills.
|
|
#
?
May 17, 2022 22:20
|
|
|
Where can I read about best practices for database backup lifecycle management? Because I’m pretty sure that two to three years of uncompressed full monthly backups, a few months of uncompressed full weekly backups, along with the same amount of incrementals isn’t it. My assignment is figure out where we’re bleeding money on storage and this is staring at me like a 200tb gorilla (literally half of our S3 storage, although at least some of it is thankfully in Glacier)
|
|
#
?
May 17, 2022 23:03
|
|
|
22 Eargesplitten posted:Where can I read about best practices for database backup lifecycle management? Because I’m pretty sure that two to three years of uncompressed full monthly backups, a few months of uncompressed full weekly backups, along with the same amount of incrementals isn’t it. My assignment is figure out where we’re bleeding money on storage and this is staring at me like a 200tb gorilla (literally half of our S3 storage, although at least some of it is thankfully in Glacier) You'll want to start by googling database RTO and database RPO. Primary questions to ask management about backups: How fast do you wish to recover from an outage? How much data loss is acceptable from an outage? Is disaster recovery the only time a database gets restored or are there other reasons to keep backups? If so, what are they? If you want a fast RTO, then snapshots and/or full backups are the way to go. If you can't handle much data loss, snapshots every (quick) time period and/or frequent incrementals should be used. Other reasons to restore data could be to create a dev or load test environment that mimics prod, or forked for another reason. This would have different requirements than straight RTO/RPO.
|
|
#
?
May 18, 2022 06:31
|
|
|
Hughmoris posted:That all makes sense. I'm halfway comfortable with basic CLI and Console use so I'll focus on better developing my IaC skills. You may know this already, but I'll point it out anyway. The CLI is automatically generated from the API specification, and the CFN schema closely wraps the underlying create/modify APIs (to the point where CFN documentation sometimes describes parameters in ways which only make sense if you consider the underlying API call). So figuring out how to do something in CLI or API will be very easy to translate to CFN. The console also usually reflects the API fairly well, and it almost always issues the same API calls so you could sniff traffic in your browser's development tools to figure out what it's doing.
|
|
#
?
May 18, 2022 06:50
|
|
|
You will also come to learn that there is no consistency between the API various AWS services present. One might have a get operation where another has describe. It may use a name as the attribute to identity a resource, or an arn. It might let you update an existing resource or force you to delete and recreate. Makes working with each new service an adventure!
|
|
#
?
May 18, 2022 12:34
|
|
|
Remembering that each service is an independent team running as a little fiefdom, it’s own tiny company in a sea of tiny companies goes a long way to help maintain sanity. Only then do the irregularities and inconsistencies between services make sense. AWS has tried to standardize, but yeah.
|
|
#
?
May 18, 2022 12:56
|
|
|
Docjowles posted:It may use a name as the attribute to identity a resource, or an arn. I’ve never had to use a name before and ran into this the other day for the first time and was like wtf is this now
|
|
#
?
May 18, 2022 13:05
|
|
|
Just-In-Timeberlake posted:I’ve never had to use a name before and ran into this the other day for the first time and was like wtf is this now Laughs in queue url
|
|
#
?
May 18, 2022 13:23
|
|
|
Quebec Bagnet posted:You may know this already, but I'll point it out anyway. The CLI is automatically generated from the API specification, and the CFN schema closely wraps the underlying create/modify APIs (to the point where CFN documentation sometimes describes parameters in ways which only make sense if you consider the underlying API call). So figuring out how to do something in CLI or API will be very easy to translate to CFN. I did not know that. Thanks! Question: is there a simple way to translate 'clicks in the console' to a CFN template? E.g. I click through the console and set up a simple EC2 with specific settings. Can AWS somehow spit out what the CFN template or stack would look like for that EC2 deployment? Or do I have to muddle through documentation to see how to recreate that from scratch in CFN? I'm thinking along the lines of how Azure automatically generates an ARM template for export.
|
|
#
?
May 18, 2022 13:26
|
|
|
Agrikk posted:Remembering that each service is an independent team running as a little fiefdom, it’s own tiny company in a sea of tiny companies goes a long way to help maintain sanity. Only then do the irregularities and inconsistencies between services make sense. AWS has tried to standardize, but yeah. They don’t “make sense” but they are understandable given that situation.
|
|
#
?
May 18, 2022 13:30
|
|
|
Quebec Bagnet posted:You may know this already, but I'll point it out anyway. The CLI is automatically generated from the API specification, and the CFN schema closely wraps the underlying create/modify APIs (to the point where CFN documentation sometimes describes parameters in ways which only make sense if you consider the underlying API call). So figuring out how to do something in CLI or API will be very easy to translate to CFN. This is usually true, but it's also the case that some things simply aren't implemented in CFn yet that you can do either through the CLI or the console
|
|
#
?
May 18, 2022 14:55
|
|
|
|
| # ? May 21, 2024 12:22 |
|
|
Agrikk posted:You'll want to start by googling database RTO and database RPO. I asked if we had a contractual RPO or RTO for our customers and haven't heard back yet. I'm also not sure how far back we might be contracted to be able to retrieve something. I'm going to need to get answers to that for sure.
|
|
#
?
May 18, 2022 15:28
|
|