|
I passed architect and developer associate without studying but I use AWS every day. I also took practice tests first to see where I was in relation to what was expected.
|
#
?
Mar 14, 2019 03:54
|
|
|
|
| # ? May 17, 2024 14:29 |
|
|
Cloudformation drift detection...Does it just tell you that objects have changed since you launched your template or is there a way for it to produce an edited Cloudformation template that includes the changes? Or a separate template that only includes the additions/changes?
|
|
#
?
Mar 14, 2019 14:30
|
|
|
Scrapez posted:Cloudformation drift detection...Does it just tell you that objects have changed since you launched your template or is there a way for it to produce an edited Cloudformation template that includes the changes? Or a separate template that only includes the additions/changes? No. Cloud formation launches itself and then is done. Any subsequent changes to the environment has to be monitored by other means.
|
|
#
?
Mar 14, 2019 18:07
|
|
|
Agrikk posted:No. Gotcha. It would be neat if they could sync up drift detection with cloudformer to have it automatically generate a replacement template. As it is, it isn't possible to use cloudformer to create a cloudformation template of say your objects in us-east-1 and restore said template to us-east-2 without manually building some objects.
|
|
#
?
Mar 14, 2019 19:18
|
|
|
I am having a horrible time trying to get websockets to work on elastic beanstalk. My setup is cloudfronted s3 bucket with static react page ---> nodejs websocket server ---- > java backend No matter how I try I can't seem to get websockets working behind a application loadbalancer. I want to have the nodejs backend available from the web, but connected to the java backend which is on a vpc. I managed to get it working before, but that was without a working healthcheck and socket.io stuck in polling mode. Now I can't even recreate that it seems. Has anyone used websockets with elb before and got it to work? Edit: The closest I have managed to get is the frontend giving a 502 Bad Gateway, with no logs in the node elb. Sistergodiva fucked around with this message at 20:06 on Mar 16, 2019 |
|
#
?
Mar 16, 2019 19:54
|
|
|
Sistergodiva posted:I am having a horrible time trying to get websockets to work on elastic beanstalk. How flexible are you with the implementation details? Have you looked at the new API Gateway support for WebSocket?
|
|
#
?
Mar 16, 2019 23:32
|
|
|
Adhemar posted:How flexible are you with the implementation details? Have you looked at the new API Gateway support for WebSocket? I haven't really looked into API gateway. I already have the backend built. Could I use the api gateway with a already built nodejs app?
|
|
#
?
Mar 16, 2019 23:57
|
|
|
API gateway for default type setups but after that you probably fall back to nginx instances when you up complexity.
|
|
#
?
Mar 17, 2019 02:33
|
|
|
We're looking to potentially transition our EFT server to an AWS service. What are people's experiences with the SFTP offerings? I know AWS has AWS Transfer for SFTP now, but there are other things out there and I'm not familiar enough with the market to know what's popular and, more importantly, good.
|
|
#
?
Mar 20, 2019 18:36
|
|
|
I like Thorntech's SFTP Gateway. I don't think there's any decent alternatives, honestly.
|
|
#
?
Mar 20, 2019 18:44
|
|
|
Not AWS specific, but I use the atmoz/sftp docker container for that kind of stuff. specify volumes and keys in docker-compose or as run arguments
|
|
#
?
Mar 20, 2019 19:00
|
|
|
Jeoh posted:I like Thorntech's SFTP Gateway. I don't think there's any decent alternatives, honestly. I'll have to look into the cost breakdown on things, since if we're running a go-between service on an EC2 it's not that huge of a change from our current setup. A lot of what we're running in house is still server-based and accesses data via network shares, so it's not a great benefit until we make larger moves to serverless workflows.
|
|
#
?
Mar 20, 2019 19:02
|
|
|
I have some users in AWS Workspaces that are accessing a resource in Azure and I want to lock it down a bit more. Is there any documentation about what IP addresses the Workspaces traffic will be coming from?
|
|
#
?
Mar 20, 2019 23:05
|
|
|
The Fool posted:Is there any documentation about what IP addresses the Workspaces traffic will be coming from? Yes and no. AWS lists its IP ranges by region and some services (but not workspaces). I assume Workspaces will fall into the "service": "EC2" category so you can restrict it those huge spaces. https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html Alternatively put your Workspaces in a VPC and give them internet access via a NAT instance and the traffic will be coming from the Elastic IP on the NAT instance and you can lock down traffic from that.
|
|
#
?
Mar 21, 2019 00:12
|
|
|
Or look at doing a VPN tunnel between the two virtual networks
|
|
#
?
Mar 21, 2019 22:03
|
|
|
Thanks Ants posted:Or look at doing a VPN tunnel between the two virtual networks In the end we just set up a client vpn from the workspaces to the azure network.
|
|
#
?
Mar 21, 2019 22:21
|
|
|
Cloud networking is magic. We needed to move some Azure services into a different region so I just built the vnet, moved the VPN tunnels from the old region to the new, then peered the two vnets, allowing the old one to use the gateway of the new one. Total downtime was about 30 mins which included the time to redo the VPN tunnels on our firewalls. Everything works as it did before, except the packets are going via our local region and I can bring new things up gradually without any disruption. I have no idea how that all works in the backend in a way that can maintain segregation but it’s impressive.
|
|
#
?
Apr 14, 2019 01:40
|
|
|
Thanks Ants posted:I have no idea how that all works in the backend in a way that can maintain segregation but it’s impressive. Overlays, overlays everywhere. I don't know what Azure are doing specifically, but knowing a bit about VXLAN and GRE can go a long way towards building understanding of cloud networking magic.
|
|
#
?
Apr 14, 2019 17:00
|
|
|
Yeah I figured something along those lines but it's the scale that is the  bit for me
|
|
#
?
Apr 14, 2019 18:24
|
|
|
if you're ever bored, this talk about how aws vpc networking works is really good: https://www.youtube.com/watch?v=Zd5hsL-JNY4 its mad old now, but like its basically the step by step story of "why we had to do this and how we did it"
|
|
#
?
Apr 14, 2019 21:42
|
|
|
...and here is his update, presented the next year at re:Invent (2015) https://youtu.be/3qln2u1Vr2E They are each 45 minutes long or so but are worth watching.
|
|
#
?
Apr 14, 2019 22:33
|
|
|
This was a real cool networking talk from the 2018 re:Invent. One of those sessions I was glad I went to even though it had no immediate value, because it was just Amazon nerds talking in depth about the kickass poo poo they get to do behind the scenes. Opened my eyes to things I never would have thought of. https://www.youtube.com/watch?v=tPUl96EEFps
|
|
#
?
Apr 15, 2019 05:53
|
|
|
Coming in knowing almost nothing, what’s the best method to invoke a Java executable against a file that appears in S3 and keep it as ephemeral as possible? I’d prefer if it could be a service like Elastic Beanstalk, but I’m not sure how friendly that is with executables that take arguments or properties. I could just invoke it from an OS running in an EC2 but I thought there must be a more AWS-friendly method.
|
|
#
?
Apr 18, 2019 00:13
|
|
|
PierreTheMime posted:Coming in knowing almost nothing, what’s the best method to invoke a Java executable against a file that appears in S3 and keep it as ephemeral as possible? I’d prefer if it could be a service like Elastic Beanstalk, but I’m not sure how friendly that is with executables that take arguments or properties. There’s a bunch of ways. You could use ECS and Fargate with a docker image, put the java code into lambda and go serverless, get the thing working on opsworks (which I think they have recipes pre made for java) to manage the ec2 and some options. Or use just ec2 and maybe auto scaling which you can set to 0 when you aren’t using it. Don’t think there’s a best way, it just depends on your needs and how you want to manage it.
|
|
#
?
Apr 18, 2019 00:41
|
|
|
JHVH-1 posted:There’s a bunch of ways. You could use ECS and Fargate with a docker image, put the java code into lambda and go serverless, get the thing working on opsworks (which I think they have recipes pre made for java) to manage the ec2 and some options. Or use just ec2 and maybe auto scaling which you can set to 0 when you aren’t using it. Yeah server-less would be my preference, it just needs to stream in a file from S3 source and output a result so nothing too complicated. I’ll check out the Lambda angle. Thanks
|
|
#
?
Apr 18, 2019 00:51
|
|
|
"Ephemeral as possible" kind of cries out for lambda, imo. You can configure an S3 bucket to invoke your function every time an object is uploaded, receiving info about the object as an argument. When it's done processing, it shuts off until the next invocation. Here are some random docs. The code sample is nodejs but java works fine, too. https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html
|
|
#
?
Apr 18, 2019 02:53
|
|
|
what's my best option for pushing records to kinesis from languages with poor support for the kinesis producer library? what i've considered so far: a: pushing directly to the stream using a native client b: writing a basic http wrapper around the kpl and pushing events to a pair of fargate containers running it, letting them batch them and push to the stream c: using cloudwatch events instead and taking advantage of it's ability to persist events to a kinesis stream d: using the kinesis log agent and just writing json lines to a file i don't like a because we're using some sketch programming languages and they have iffy quality clients i don't like b because i hate operating things and also latency is a concern i don't like c because i can't find out if order is preserved and also latency is a concern i really don't like d because i can't afford to lose events and orchestrating things so all logs are written and shipped seems hard have i missed something obvious?
|
|
#
?
Apr 19, 2019 04:07
|
|
|
I posted in the Java thread but I suppose I’ll ask here too: does anyone have any experience with the AWS SDK? I’m rewriting some code to read a file from an S3 bucket (related to my question previously) but instantiating the client connection takes literally 2+ minutes from an EC2 instance with direct access and nigh-instant CLI connectivity. The library is a little big but I can’t imagine it should take as long as it does. Once the connection is made it works but, slow as molasses. The same code parsing a file from local/network drives takes milliseconds.
|
|
#
?
Apr 19, 2019 14:20
|
|
|
Not java, but go’s and boto3. I don’t even think about the delay with those libs. That is, it’s as close to instant as network allows.
|
|
#
?
Apr 19, 2019 14:36
|
|
|
Sounds like there's something else going on. That operation should be near instantaneous.
|
|
#
?
Apr 19, 2019 17:10
|
|
|
Prediction: it is, somehow, the fault of DNS.
|
|
#
?
Apr 19, 2019 17:34
|
|
|
I imagine it is environmental but I just don’t know how to diagnose it. It’s not indicating anything specific but it could be defaulting to a different region, I suppose. I did try creating a “standard” client connection and setting the region and it didn’t seem to work any better. Maybe I’ll look into tracing the traffic from the call. I’d talk to the infra people about it but a) they probably won’t know or care and b) I’d prefer to figure it out on my own.
|
|
#
?
Apr 19, 2019 17:55
|
|
|
Can you post the relevant code? edit - nevermind, i found it in the java thread. Is there a reason you aren't using TransferManager? Rolling your own seems like a recipe for disaster. https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html deedee megadoodoo fucked around with this message at 21:17 on Apr 19, 2019 |
|
#
?
Apr 19, 2019 21:13
|
|
|
deedee megadoodoo posted:Can you post the relevant code? It doesn’t especially seem like rolling my own, considering I’m using code snippets straight off their site. I’m not looking to download the file, I’m looking to read it in a steam and pass that to Apache CSVParser and write out my own file after I parse the data. I could be wrong though, as I’m by no means an expert.
|
|
#
?
Apr 19, 2019 21:44
|
|
|
PierreTheMime posted:I posted in the Java thread but I suppose I’ll ask here too: does anyone have any experience with the AWS SDK? I’m rewriting some code to read a file from an S3 bucket (related to my question previously) but instantiating the client connection takes literally 2+ minutes from an EC2 instance with direct access and nigh-instant CLI connectivity. Is there some weird network setup that makes the sdk's attempt to fetch AWS credentials from the ec2 instance metadata endpoint time out before it uses some other source of creds?
|
|
#
?
Apr 20, 2019 11:55
|
|
|
Vanadium posted:Is there some weird network setup that makes the sdk's attempt to fetch AWS credentials from the ec2 instance metadata endpoint time out before it uses some other source of creds? I know for this current setup the IAM role is assigned to the EC2 so I can’t pass it defined credentials, which is apparently a normal thing but it may be the culprit. I might try getting another user-based credential created I can pass and see if that resolves the issue. I’m off till Tuesday but I’ll be happy to update the thread when I find out more info. Edit: aha, there’s a blurb on the site about it that might relate: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html posted:With some AWS SDKs, the developer can use a provider that manages the temporary security credentials transparently. (The documentation for individual AWS SDKs describes the features supported by that SDK for managing credentials.) Like I said though, the code does eventually work instead of timing out or failing, so I’m a little confused what round-about way it’s pulling creds that would take so long. Edit edit: Here’s the related SDK entry I’ll be trying out later: https://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionTokenJava.html PierreTheMime fucked around with this message at 12:20 on Apr 20, 2019 |
|
#
?
Apr 20, 2019 12:13
|
|
|
Yeah that sounds normal. I figured it might time out and then fallback to so something else, but that doesn't sound like it. Can you configure debug logging for the SDK and see if there's anything weird going on? I've mostly used the golang sdk, but I can just tell that to log every request it does and then figure out what's taking all the time.
|
|
#
?
Apr 20, 2019 12:25
|
|
|
Vanadium posted:Yeah that sounds normal. I figured it might time out and then fallback to so something else, but that doesn't sound like it. I’ll check on that as well, I’m sure you can and I just haven’t done it yet. Thanks for talking it through with me.
|
|
#
?
Apr 20, 2019 12:32
|
|
|
Update: Apparently the huge delay was actually just loading the library into memory, since the process does not have high priority and the original library is quite large. I trimmed down the imports and I can now get a credential in about 8-10 seconds, which still isn't blazing fast but is at least vaguely "okay". Still hunting for improvements though, as the CLI can access the EC2 credential in ~1.5 seconds.
|
|
#
?
Apr 24, 2019 18:36
|
|
|
|
| # ? May 17, 2024 14:29 |
|
|
Is there facility in sagemaker for deploying pre-trained Torch models? I know there’s a legacy mode for python endpoints, anything similar for lua?
|
|
#
?
Apr 24, 2019 20:55
|
|