|
I'm dumping files to S3 and on a schedule need to take all the new ones and convert them to custom avro and make avro files for every N files. I have a Python function do bottle up and convert, is there a more elegant way to do this than s3 sync to a computer with a real file system and push it back? I've used Kinesis firehouse on ingest but don't see anything that could accomplish what I want.
|
#
¿
Feb 18, 2022 04:14
|
|
|
|
| # ¿ May 1, 2024 20:27 |
|
|
I run a lot of work through SQS queues and want to display stats on a web portal: outstanding jobs, number of jobs over 1/6/24 hours, stuff like that. If I use boto3 I can get a few of these stats at a moment in time, but not the history. I guess I could snapshot this and roll my own stats on my Django database, but I assume I'm not the first person to need to solve this problem. What should I be looking at?
|
|
#
¿
Apr 24, 2022 03:23
|
|
|
I’m trying to use SES from Lambdas and other hosted stuff to send email. Is it worth making a subdomain to send or doing domain verification on the existing one? This will be internal emails saying reports are ready or go here, not for an ELP blast. Our current email domain is hosted Office365
|
|
#
¿
May 11, 2022 04:36
|
|
|
CarForumPoster posted:TIL I can get a desktop/GUI out from an AWS Lambda based docker container image. This makes diagnosing why some webscrapers are having issues much easier and maybe someone in this thread needs to know this. Interesting. Are you RDPing or shoving X down the pipe (lol that latency would be funnn)?
|
|
#
¿
May 12, 2022 01:24
|
|
|
I'd like to run a corporate Django site on Fargate, does AWS have anything like Azure App Proxy? I like the idea of having the Django app sitting behind a scalable LB that authenticates people against a directory (Azure AD in this case) and passes that info back. I've done this in the past in places that had PKI where the nginx -> Django backend would authenticate the user and pass back headers that the Django app used for authentication/authorization. Should I just look into the AWS LB sets more or is there something else to do this?
|
|
#
¿
Jun 9, 2022 04:18
|
|
|
Thanks! That link is exactly what I needed, and answers my follow on question about authenticating the headers. 
|
|
#
¿
Jun 9, 2022 16:48
|
|
|
This risks being a bigger SAML question, but what are my options for programmatic access? If I put an API on the Django website like django-rest-framework, do I make people log in via SAML and get tokens to feed into their API client?
|
|
#
¿
Jun 9, 2022 21:54
|
|
|
I see that SES can receive emails, has anyone used these as a forwarder too? I interact with some companies that can only send us reports via email. I'd love to give them a statements@domain.com that kicks off a Lambda / posts the attachment to S3, but in the meantime (while transitioning) I'd like to have it forward to the usual recipients. I guess I could fire a lambda that does an SES forward, but it's starting to sound Rube Goldbergian at that point.
|
|
#
¿
Jun 22, 2023 17:16
|
|
|
We have an Application Load Balancer that is ingress for a k8s application. Most of the time it works really well but occasionally it just times out. Running curl -v https://app.com doesn’t even negotiate TLS and then times out. Once it works it seems to be sticky. Intermittent so it’s hard to debug. Looking at the health checks for the app it seems fine. What should I be looking at to debug this? Shouldn’t the ALB negotiate TLS with a client first or is it “smart” and making sure the app is in a good state.
|
|
#
¿
Jun 28, 2023 12:35
|
|
|
Thanks everyone who helped, I solved this using your help and realize I forgot to follow up. We had 2/3 subnets (different AZs) that weren't plumbed correctly. Hard to debug because some sticky sessioning going on (so once it worked it was hard to reproduce). It didn't help between my general inexperience with Terraform and how hard it was to debug because I'm inexperienced attaching to k8s pods. Armed with your thoughts we sat down together and figured it out. Thanks for the assistance. And in great news I now know how to use curl with the -w command to time the various parts of the handshakes/HTTP transaction. CF won't let me paste it, but using the -w flag to write out things like time_connect made this a lot easier to debug than my "curl from various endpoints upstream and downstream of the LB, and use a wall clock".
|
|
#
¿
Jul 13, 2023 12:52
|
|
|
We have an application set running on EKS that seems to take way too much infrastructure. My 1.0 version (non-kubernetes) ran on 1 t2.medium and now the re-architected version takes 5 t3.larges. The CPU usage is very small across these, though... most of our workflows spend their time waiting for I/O. There's no memory or disk I/O pressure. The problem is IP addresses. If we try to run too many pods on our EKS we get a "can't schedule pod" message and EKS complains that it's out of IP addresses to allocate. Right now we are running 67 pods and going much higher would require a 6th instance in the cluster. Where should I look to research this problem and allow for more multiplexing on the same instances? I found this article about optimization, but things like network overlays don't seem like the first thing I'd want to reach for in terms of complexity.
|
|
#
¿
Sep 6, 2023 21:43
|
|
|
The Iron Rose posted:It depends on how many network interfaces the underlying instance has. If you have small workloads and small nodes, you can’t run all that many pods on them at once. At least when using the default CNI. A t3.small supports a grand total of 11 pods, including system daemonsets. Is there a way to get more pods on the same hardware? Currently we are just buying the "most economical" from a cost per interfaces standpoint, but it's really crummy as our instances sit idle almost all of the time. Should we switch to nitro?
|
|
#
¿
Sep 26, 2023 20:08
|
|
|
I have a pretty small Postgres RDSinstance on a db.t4g.medium that is capping out of IOPS and as a result there's a whole lot of WAL wait. We have a 20GB disk on gp2 so expected IOPS is 60. I shouldn't have let it get to this point, but is there a graceful way to increase IOPS with no downtime? My options appear to be:
Is there a good way to spin up another instance into the cluster with enough IOPS and gracefully transition to it? I know I could pgdump/restore but would rather not have downtime if possible.
|
|
#
¿
Apr 15, 2024 20:36
|
|
|
|
| # ¿ May 1, 2024 20:27 |
|
|
Happiness Commando posted:gp2 to gp3 should be totally safe and without downtime. Also you should have a backup, because that's a good practice. Documentation for aws cli here Docjowles posted:I'm reasonably sure you can convert from gp2 to gp3 online. You certainly can with EBS volumes so I don't know why RDS would be different. There might be some performance degradation during the move but 20GB should be very fast. Thank you both, I backed up and found where I could change it and it all happened online. Now getting 10x the IOPS and the RDS dashboard wait times look MUCH better.
|
|
#
¿
Apr 17, 2024 16:26
|
|