Amazon Web Services - Cloud Giant Hits Hard - The Something Awful Forums

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Amazon Web Services - Cloud Giant Hits Hard

xpander: Sep 2, 2004

This is the thread for all things AWS! If you've got a question about a particular service, post it here and hopefully someone will answer it and you aren't just shouting into the void!! Possible topics include:

-how to use a particular service
-architecture questions/best practices
-exams/certifications
-cool stuff you're working on

I posted this in CoC because I want the focus to be on "devops"/scripting/coding because you should be automating your infrastructure, as part of the larger umbrella of "everything". If not - feel free to cruise on outta here on your dinosaur!

Further reading:
AWS Security Whitepaper
Architecting for the Cloud: AWS Best Practices Whitepaper

I'll try to keep the OP updated with must-have knowledge as appropriate, just holla if you have something worthy of this honour(<-- yes I'm Canadian).

Who The Hell Is This Guy?
I just started working at Flux7, who are primarily an AWS consultancy and devops shop. My title is amorphous, but "Cloud Infrastructure Engineer" has many impressive wizard-like words, as well as the added bonus of making real engineers mad. I wrote and passed(just barely!) my AWS Developer Associate exam yesterday, and am currently studying for Solutions Architect. I plan on going the distance all the way to at least one Professional cert, but given how brutal the Developer exam was compared to the practice questions/quizzes I did, I plan on accumulating some hefty experience before potentially throwing away $150 on that. I'm pretty much all-in on AWS's poo poo - the stuff I've seen and done in a bit over a month here has been extremely cool and good, and Amazon's rate of development on their services means it's only going to get better as time goes on. Admittedly I'm hardly a master of this stuff, but I'm learning a ton!

Why Should I Care About The Cloud?
An excellent question. The short answer: because it will probably save you money, time and aggravation. The less snarky answer: it depends! It feels shortsighted to say that everything belongs "in the cloud" - almost certainly there exists a use case or twenty where it's just not a great answer. The reality is that there's a good chance you can re-architect some portion of your system to make better use of on-demand resources so that you aren't paying fulltime usage on i.e. EC2 instances. Things like API Gateway and Lambda might obviate the need for certain fulltime-provisioned resources entirely! And that is all to say nothing about the fantastic failover/disaster recovery possibilities baked into the AWS infrastructure. It's certainly worth investigating to see what might be a good fit. Incidentally, this is exactly the kind of thing that we do at Flux7, and I'm happy to chat about it as a learning exercise for myself.

Are You Certifiable?
I think it's worth talking about becoming AWS Certified, and what that entails. I don't have much data on how valuable this is, but the exams themselves are comprehensive enough that I believe them to accurately demonstrate the knowledge required(and thus prove your skillz).

First up, getting some training - I really like A Cloud Guru(also it's the only training I've done, other than practical/on-the-job):

A Cloud Guru

They've got courses on pretty much everything - I bought the Associate package while on sale at Slashdot for I think $5, which was a steal. I think it's absolutely worth the full price, and will be getting the Professional pairing once I feel it's worth my time. I also bought their Lambda course this week, and while the quality wasn't quite the same(different instructor), it's a great hands-on demo of what's possible with the serverless approach(feels buzzwordy but man can you do some cool poo poo in this space!). Definitely post any feedback you have on other courses they offer, as I'd like to vet training materials so that people aren't wasting their time and money.

Speaking of which - DO NOT do the Webassessor practice exams. From all accounts(not my own) they are garbage, and I'll say that the Developer exam was WAY harder than the practice questions I got from A Cloud Guru. If that's the case for the "official" practice exams, then my assessment is that they lure you into a false sense of security and aren't even worth the paltry $15.

The exams themselves will run you $150 USD apiece, with the pass mark being approximately 65%(can confirm this to be try as of 09/21/2016, uuhh...from a friend). Reportedly it's on a bell curve, and they shift the pass mark based on real-world results, but who knows. Plan to need at least 36/55 correct answers for victory.

# ¿ Sep 23, 2016 21:41

Adbot: ADBOT LOVES YOU

# ¿ May 2, 2024 09:11

xpander: Sep 2, 2004

xtal posted:

I highly recommend installing this addon before reading this thread: https://chrome.google.com/webstore/detail/cloud-to-butt-plus/apmlngnhgbnjpajelfkmabhkfapgnoai?hl=en

How could I have missed such low-hanging fruit?? Thanks!

# ¿ Sep 24, 2016 17:10

xpander: Sep 2, 2004

Lutha Mahtin posted:

I signed up for AWS a few days ago because I want to write a little Internet-using server program for a spare-time project, and the free tiers of various Amazon services look like they will be more than enough for it (I don't really have much of a budget for this). I don't really know where to get started, though. I've been reading through some of Amazon's documentation and ~~marketing~~ "get started" videos, but I'm curious if there are third-party articles or documentation that explain things better for someone (like me) who isn't super familiar with all of this stuff.

What I want to do is query some data over the web and store it somewhere for me to download later. The most important queries would be two things, one is a single query run once and hour, and another would be up to a couple dozen queries that would be run once or twice a day. Neither of these are very high bandwidth or super complex, just a little parsing of the results to toss the data I don't need and store the rest.

There is another API I want to query that updates every minute, and while I don't know anything about cloud stuff, my intuition is to think that the tighter time constraint might produce more headaches. However, it's not critical at this point to hit this API every single minute, because I'm not sure if the data from it will be useful, and I'm pretty sure I can determine this usefulness just by having a few hours or days of data from it to combine with other sources, so I may end up not caring about it.

Right now I think what I need is an EC2 instance to load my program onto. Beyond that, I don't know what I'm doing. I need to figure out first of all how to set things up so that I get an instance and put my program onto it, of course. But I don't really understand how the lifecycle of my virtual server works, like if I need to watch for signals that my instance is going to be like, respawned or moved, or if these are even things that happen. I also don't know if I need to or should be making use of any other stuff, like the data store services. I'm going to keep reading, but any suggestions and resources are welcome!

You're on the right track regarding EC2 - this is your basic server virtualization service. If you know anything about how this works outside of the cloud(i.e. traditional server hosting), it's much the same. At the end of the day, you're going to fire up an instance(server) and log in remotely just like a physical machine. I'm not sure what your background is, so if you need further explication on anything then just point it out. You'll definitely want some sort of monitoring on a running instance, for exactly the reasons you described. Typically it will only get moved if the underlying hardware fails - this happens, though it's fairly rare. If the instance does get terminated from their end, note that they will not recreate it for you - that's your job. Because of that, you'll certainly want to keep in mind how you might save program state(if any), configuration details, and data. On that last note, there's a free tier offering of RDS, Amazon's relational database service. This is basically a managed EC2 instance running some flavour of SQL, where you don't have to worry about OS patching, software optimizations, etc - just the architecture of your actual tables. There's plenty of MySQL tutorials out there, and it's useful enough to know how to put together some basic queries. If you don't actually need a database, and instead can make use of file storage, S3 will be your best friend. For monitoring, CloudWatch has you covered. You can set up alerts(i.e. emails) that get sent whenever a given metric crosses a certain threshold. Basic instance health checks don't really need configuration beyond where to send those emails. Do be sure to note the exact parameters of free tier coverage - it's pretty obvious with RDS, from what I recall, but I think any custom metrics with CloudWatch aren't free. Still, I guarantee you can do what you want without paying a cent.

On that note, the first thing I do when setting up a new AWS account is set up a billing alarm called "Cheapskate" where it emails me at >= $0.01 so I instantly know if I'm being charged for something. Check out this page for how to do that. Keep in mind that once it's triggered at that point in a given month, that alarm is now useless for additional charges. It will still show as being in the ALARM state, but won't send out additional notifications. So maybe set up 2-3 at different thresholds just in case. I'm on account #3 because I didn't make much use of those first two years!

If you're querying an API and just storing the data, you might want to take a look at Lambda. It will let you run code without actually having to think about a server at all. The only runtimes it has available are Python, Node.js and Java, so if you're writing in another language then forget about it. But if not, it might be worth looking into. It too has a free tier(that you'll never exceed if you're at all minding how often you run functions). If it seems daunting, just go with whatever you feel you understand the best. I will say that it's nice to not have to think about what's executing my code, or do much setup to get it to that point.

I hope this was informative and not terribly dense - I didn't want to go off the rails too badly for a high-level overview. Again, if you want more specific info regarding a certain topic, Just Post!

# ¿ Oct 7, 2016 23:52

xpander: Sep 2, 2004

StabbinHobo posted:

I'm curious what other peoples realworld workflows are like.

For instance, do you work from a text editor editing cloudformation templates and then run an aws cli command on your laptop? Do you have an ssh bounce host? Do you use the web interface on a day to day basis? How do you really deeply browse/survey/crawl an account to make sure you haven't accidentally left some rds instances running in brazil for a month?

I use the web interface the most because I work in devops consulting, and frequently run client sessions where I have to show them how to do stuff. So I want to be sure that I know where most things are. There are still some things you can only do on the console though - I couldn't find a way to pull the ARN of an SSL certificate managed via IAM through the web portal. I use PyCharm because I end up working with Python a lot in addition to YAML.

My workflow is: Make changes in PyCharm -> deploy using web console -> verify changes working correctly -> commit and push. For CFN-related work I think I'm going to start using the CLI more. The architectures we deploy frequently have a jump box, but it depends on the client's needs.

As for cost-related things, I'd recommend making some billing alarms. I have one called "cheapskate" in personal accounts still under the free tier, so that I know if I'm hitting any paid usage breakpoint(the threshold is set at $0.01). For your specific example of resources being used in other regions, you can filter by region in the Cost Explorer. That will let you chase down errant EC2/RDS instances or whatever. I don't have a better solution for that, but I don't really have to deal with that facet of the work.

# ¿ Apr 3, 2017 00:32

xpander: Sep 2, 2004

oliveoil posted:

Any thoughts on CodeStar? https://aws.amazon.com/blogs/aws/new-aws-codestar/

Seems like price and the fact that everything is confusing and I feel like I need to know how all the different products work in order to know which products I need and how to configure and deploy to each one to make a basic CRUD app are all that make me from tinkering with AWS. Seems like this would help with the non-procrastinators related stuff? I write code and push buttons and and magically have all the pieces needed for a web app set up for me?

It looks pretty cool - you could manage a lot of those pieces with CloudFormation templates, or Terraform, or any number of other solutions. But for a lone developer or small shop/department, this sounds like a great first step into bringing all of that stuff under one roof. At the very least, it creates most of the infrastructure you'd use in a modern dev stack and gives you a nice dashboard for a bird's eye view. And it will likely continuing being developed, so it should get better. I'm going to try it out for my next project!

# ¿ Apr 19, 2017 23:23

xpander: Sep 2, 2004

SnatchRabbit posted:

Quick question here. I'm trying to use EBS to set up a Moodle platform. I've set up the environment, uploaded the moodle zip package which deploys correctly. I run through the web installation, connect to the db, but I get stuck at the pre-requisite checks with the following error:
code:
The Zip PHP extension is now required by Moodle, info-ZIP binaries or PclZip library are not used anymore.
I'm not quite certain how to 1) log into the EC2 instance that the EBS environment created and 2) how to install the php extension once I'm in there. I've looked through all the moodle docs but they all have you ssh into an EC2 instance. I also though that maybe the config settings for EBS - Zlib output compression: On/True would do the trick but that hasn't worked. Anything else I could try?

When you spun up the EC2 instance, it should have asked you to either generate or specify a key, depending on if you've done so before. Assuming everything was on default and you chose to give it a public IP, you should be able to ssh into it with the aforementioned key. Also check the security group(s) associated with the instance to make sure they are allowing port 22 either to the world, or your IP address.

I'm not sure where EBS comes in here - were you given a disk image to use somehow? EBS is Amazon's "hard drive in the cloud" offering, so it shouldn't have much to do with Moodle. But I don't know Moodle at all.

# ¿ Apr 20, 2017 22:42

xpander: Sep 2, 2004

SnatchRabbit posted:

Sorry, I'm referring to Elastic Beanstalk. Essentially, Moodle is just a php application that I can download in zip form and elastic beanstalk will accept it. The trouble is getting the environment I set up to play nice with moodle vis a vis php extensions.

My bad, I don't encounter Beanstalk that much so I don't equate it to the acronym. Looks like it's possible to install PHP extensions with an EB command:

http://stackoverflow.com/questions/38730483/how-to-install-a-php-extension-witn-amazon-aws-elastic-beanstalk

More general info on using EB configuration:

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_PHP.container.html

# ¿ Apr 20, 2017 23:35

xpander: Sep 2, 2004

Seventh Arrow posted:

I booked my Solutions Architect - Associate exam for Feb 12 so I'm going to try and do as many labs and practice exams as I can until then. I've heard that there are a lot of scenario questions, so it seems best to have a well-rounded knowledge of the material instead of just mastering AWS trivia questions. Looking at the A Cloud Guru forums, however, it seems that the exams take a keen interest in subjects that one would never think to focus on initially - like bastion hosts, SWF use cases, and so on.

I'm also studying data engineering at a local place and the teacher who runs it says he has employers requesting AWS-certified people all the time - to the degree that he's thinking of starting an AWS course just to fill the demand. But I wonder if SAA alone will help me get my foot in the door. I mean, it's kind of the "paper trainee hat" level of AWS certs.

Echoing that doing the practice exam on ACG will set up you fairly well for the exam. SA Associate is what my company generally recommends for our engineers, and I imagine many others are similar. It's definitely good for getting your foot in the door and/or being at the top of the list.

# ¿ Feb 2, 2018 03:11

xpander: Sep 2, 2004

Docjowles posted:

holy poo poo I'm starting to see why you tout TAM as a fun and cool job so much.

For anyone struggling with 2FA, I strongly recommend ditching individual IAM accounts and just using your corporate SSO solution. Because yeah, dealing with 2FA loving sucks. If you are at a company of any size you hopefully already have some sort of SSO backed by 2FA and you can just reuse that instead of making every AWS user set up a second solution. And not hate your life twice as much every time someone drops their phone in the toilet.

This has the added benefit that engineers do not have permanent access keys. Can't upload your god-mode key to GitHub if you don't have a key You can request temporary keys once you authenticate via SSO, and we make users do this. I wrote a lovely script that makes it very easy to authenticate to our SSO, pick which AWS account you want to work in (filtered to the set this user can access based on their Active Directory groups), and then dump the temp creds to their local environment. Some of the SSO vendors even provide this out of the box. Doing this has already paid un(?)expected dividends like devs coming to us saying "hey I run this production critical job from my laptop every day under my user, and now that's not possible, what gives?" and we can gently repoint them toward not loving running critical jobs from their laptops with admin access.

Apps running on EC2 instances should use IAM instance profiles to assume a role that can do what they need. There will always be service accounts that need an actual IAM user with a long-lived key. But that should be the last resort choice, IMO.

Actual human using AWS? Access via SSO with 2FA, get temp API keys if needed
Application running in AWS? Use IAM roles
App running elsewhere that needs to access AWS resources? OK fine, you get a key but it's restricted to the minimal set of features said app requires. And it's expiring on a set schedule.

That's also what we do at the MSO I work at now, and it's pretty much the One True Way if you have the resources to set it up. Long-running credentials, especially for your admin users, are a ticking time bomb.

# ¿ Aug 19, 2019 16:56

xpander: Sep 2, 2004

deedee megadoodoo posted:

new thread title please Amazon Web Services - It does not help that is this not documented anywhere

mods

# ¿ Aug 19, 2019 17:17

xpander: Sep 2, 2004

Schneider Heim posted:

For learning purposes, I'm creating a Twitter bot that hourly tweets random lyrics/phrases and I need to decide whether I should implement it on Lambda or EC2 (free tier).

The basic gist is:
code:
1 Read list A containing items to be tweeted
2 Read list B containing items that have already been tweeted for the day
3 Pick an item from list A at random
4 If that item is not in list B,
5   Tweet it
6   Add tweeted item to list B
7 Else,
8   Go to 3 (assume we have at least 24 items to cover an entire day)
9 If list B already has 24 items, clear it
Based on my knowledge, for Lambda you have to get the data from somewhere (as it's serverless), so it would be either in simple JSON files in S3 (list B could be simpler even), or DynamoDB. Is simple file I/O in Lambda permissible and well within the free tier? To implement on EC2 would be more straightforward, but I've done that before (I have a Twitter bot running on Heroku that does the same thing) and I prefer to go down the road not taken.

What do you think? At most the bot will only run 24 times a day.

This is easily doable in Lambda, reading files from S3 is a very straightforward operation. If you haven't used FaaS before, that sounds like a great first project. Just be sure to do your file i/o in /tmp. For bonus points, figure out how to keep the function warm and check /tmp first to see if your files are still there before downloading from S3.

# ¿ Nov 7, 2019 17:42

xpander: Sep 2, 2004

Hughlander posted:

With google's change to google apps for domains where you need to pay $$$ I have some 15 year old domains that I have gmail accounts for routing to other gmail accounts that I now need to get rid of. So I plan on following this blog post https://aws.amazon.com/blogs/messaging-and-targeting/forward-incoming-email-to-an-external-destination/ about being able to:

- Set up route 53 mx records
- Use SES to save incoming mail to an S3 bucket
- Use a lambda function to trigger on file writing to S3
- To resend outgoing mail via SES to the permenant email address

And set it up for about 5 domains. (IE anything sent to *@hughlander.com goes to hughlander@gmail.com)

Since there's a reasonable number of domains, I figure also to go do that with some infrastructure as code and make it repeatable, maybe get my own blog post or at least a github link out of it. So my question is, what's the appropriate infrastructure as code system for this? I've used puppet and ansible in the past and neither seem appropriate. Since the tech is all AWS specific Cloud Formation sounds like a possibility, though I have some interest in learning terraform but not sure how terraform would work with R53, SES, S3, Lambda.

Anything I'm missing / Any thoughts?

I'm a big fan of the CDK - https://aws.amazon.com/cdk/. This lets you use a high-level programming language of choice to generate the necessary CloudFormation templates. You could also check out CDKTF if you want to use terraform under the hood: https://www.terraform.io/cdktf.

# ¿ Jun 5, 2022 19:14

xpander: Sep 2, 2004

What's the best way to get AWS credits for a startup? I've tried this before when starting my consulting company, but don't know what magic incantation will result in getting anything at all, let alone the maximum possible. Appreciate any sage advice from Amazon folk or other knowledgeable types!

# ¿ Mar 22, 2023 18:26

Adbot: ADBOT LOVES YOU

# ¿ May 2, 2024 09:11

xpander: Sep 2, 2004

Thanks, that's really helpful! Right now we're pre-launch, and serverless dev environments don't exactly rack up the spend so we won't have an account manager any time soon. Very interesting re: APN as we're bootstrapping via consulting, so this might be applicable. I was kind of under the impression that these were "try out/launch your startup" credits, I didn't realize this was meant for organizations who already had significant operations.

# ¿ Mar 22, 2023 23:39

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Amazon Web Services - Cloud Giant Hits Hard