(Thread IKs:
hot cocoa on the couch)
|
Blink 182 is playing right now. The original members.
|
#
?
Apr 15, 2023 03:05
|
|
|
|
| # ? Jun 7, 2024 03:15 |
|
|
Saalkin posted:Someone post a God dang line up i know gorillaz is playing which i'd love to see, new album is good imo neato burrito posted:Blink 182 is playing right now. The original members. worth seeing too
|
|
#
?
Apr 15, 2023 03:12
|
|
|
I love tom delonge
|
|
#
?
Apr 15, 2023 03:14
|
|
|
aliens exist
|
|
#
?
Apr 15, 2023 03:15
|
|
|
Coming up: Blondie https://www.youtube.com/watch?v=pHCdS7O248g
|
|
#
?
Apr 15, 2023 03:31
|
|
|
El Jebus posted:I'm 5 miles away, avoiding the people that go to it. Does that count? Same. The 10 had so much more traffic than usual last night Hey. It's still Friday
|
|
#
?
Apr 15, 2023 03:37
|
|
|
piss skulled with the coworkers, now it's time for some gaming. friday, it's good.
|
|
#
?
Apr 15, 2023 03:50
|
|
|
fart
|
|
#
?
Apr 15, 2023 04:12
|
|
|
Dixville posted:Same. The 10 had so much more traffic than usual last night Yeah, did all my grocery shopping for the weekend on Wednesday. I won't have to leave the house for food, beer, or weed.
|
|
#
?
Apr 15, 2023 04:34
|
|
|
has PISS SKULLER the boat happened yet what;s the status
|
|
#
?
Apr 15, 2023 04:57
|
|
|
Saturday mornings alright for posting
|
|
#
?
Apr 15, 2023 05:13
|
|
|
jimmyjams posted:fart Thats the way to start Saturday
|
|
#
?
Apr 15, 2023 05:31
|
|
|
Actually went out with friends and had a fun night. Looking for a movie to play while I start playing a game.
|
|
#
?
Apr 15, 2023 05:34
|
|
|
One more for the road.
|
|
#
?
Apr 15, 2023 05:44
|
|
|
played a lot of minecraft and a little oblivion and blue dragon. kind of a psycho collection of games to run through in a day but im a crazy guy 
|
|
#
?
Apr 15, 2023 06:20
|
|
|
good friday overall
|
|
#
?
Apr 15, 2023 06:21
|
|
|
drank some beers and cooked some honey garlic sausages with a buddy havnt really socialized too much with my bros lately so it was nice to catch up and talk some poo poo solid friday
|
|
#
?
Apr 15, 2023 07:31
|
|
|
late nite burrito
|
|
#
?
Apr 15, 2023 07:41
|
|
|
Chinatown posted:late nite burrito A good username. And lifestyle.
|
|
#
?
Apr 15, 2023 07:43
|
|
|
i woke up at 10 to find a I missed a meeting with some intel guy at 9 am who was supposed to help us understand why we have garbage performance on our new servers. something something numa. i think I did like 10 minutes of work to hand off a thing to another team who was trying to a/b test which of our new datacenters was worse than the others. this is somewhat complicated actually because graphite is a garbage tsdb and thats where application metrics are stored. We should be using prometheus at this point tbh, but years of mismanagement have prevented that lol oh well. anyway that was hard so I just left it to the other guys and went back to bed for a bit longer. Eventually my phone started chiming so I had to go have some slack conversations and helped another guy with a terraform thing. i lasted about 2 hours before I just decided I didn't really want to be awake so I went back to bed at 12 until about 2:30 according to my chrome history.  i think at this point I decided to follow up with the graphite metrics guys to see their conclusions. Had some more conversations around the interpretations of their findings. helped a few other random people with random whatevers did some more low-intensity testing and follow ups on more the nightmare that is this stupid fuckin datacenter performance issues we've been having. one thing that came up was that there was a strong correlation between packet loss and bad performance. It was actually our strongest correlation we've found after like 3 weeks of chasing red herrings. Further investigation I found that we have all of our interrupt handling on a single core. That's apparently bad according to cloudflare particularly with numa systems. https://null.53bits.co.uk/index.php?page=numa-and-queue-affinity https://blog.cloudflare.com/how-to-achieve-low-latency/ https://blog.cloudflare.com/how-to-receive-a-million-packets/ quote:While a 10% penalty for running on a different NUMA node may not sound too bad, the problem only gets worse with scale. On some tests I was able to squeeze out only 250kpps per core. On all the cross-NUMA tests the variability was bad. The performance penalty across NUMA nodes is even more visible at higher throughput. In one of the tests I got a 4x penalty when running the receiver on a bad NUMA node. After some reading, it's actually all kinda hosed. Anyway im updating the intel ice nic drivers from 0.8.1-k to 1.11.14 as an action item. supposedly this helps with irq balancing. Crazy how old the ubuntu included driver is. im probably going to need to screw around with pinning of irqs to specific physical cores. idfk its all complicated and im way too tired to be dealing with this. there's also some poo poo about disabling c-states and modifying p-states and maybe the cpu frequency governor i probably need to pursue. I've been ignoring this angle for a while. i don't relaly think its seriously the problem anymore after finding the packet drop stuff. I have some PR for tuning tcp socket buffers. This is another one of those redherrings. I think there's still merit in it even if it's not my immediate problem. https://en.wikipedia.org/wiki/Bandwidth-delay_product Apparently this was a known bottleneck for kafka. It couldn't saturate its 10g nics until these were tuned. tbh my servers likely won't be pushing 10gbps on a single socket like kafka was, but i'll still tune it why not. these defaults were set in like 1999 for 10mbps nics on machines with 500mb of memory lol code:this has turned into another whole fuckin mess because look at this poo poo. the prometheus operator doesn't support arbitrary environment variables being passed in. Which is what I need in order to do the oidc iam federation authentication thing. this thing https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ https://github.com/prometheus-opera...efulset.go#L726 It was only recently that thanos-sidecar at got support for using non-static credentials. v.0.25.0 according to the tags here. I'm still running v0.21.0 from forever ago because neglect. so guess i need to update this while im at it. https://github.com/thanos-io/thanos...8f8f64bb328R103 Even though the operator doesn't support what I want. thats fine because I can just cheat and template in my own stuff as a work around and keep going. Except the way things are templated is all hosed so I've been wasting like 45 minutes unravelling this dogshit configuration. It's actually so bad. I've been underwater for the past 9 months so I haven't had time to really care about this, but man all of the new hires have hosed this up so bad. I hate it. I want to both fix this, but i just don't have time for anything anymore and haven't in so long. and also actually i don't want to fix it, i just want it to not be wasting my time. I ended up just going back to bed for another 2 hours before I was willing to come back to work for a little bit more. quote:level=error ts=2023-04-15T05:48:05.185550013Z caller=main.go:130 err="yaml: unmarshal errors:\n line 1: field aws_sdk_auth not found in type s3.Config\ncreate S3 client\ngithub.com/thanos-io/thanos/pkg/objstore/client.NewBucket\n\t/app/pkg/objstore/client/factory.go:77\nmain.runStore\n\t/app/cmd/thanos/store.go:243\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:188\nmain.main\n\t/app/cmd/thanos/main.go:128\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\ncreate bucket client\nmain.runStore\n\t/app/cmd/thanos/store.go:245\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:188\nmain.main\n\t/app/cmd/thanos/main.go:128\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\npreparing store command failed\nmain.main\n\t/app/cmd/thanos/main.go:130\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371" after more pissing around with this and reading about kernel networking details I decided to just go to the grocery store. I ran out of food a couple days ago. I bought veggies and eggs mostly. The pork loins I usually buy weren't on sale so I didn't bother. This was the first time I really left my house in more than 2 weeks I think. i honestly don't remember when I last did. guess i'll just keep fixing this thanos sidecar auth thing so metrics work again for the rest of the night. overall an extremely painfully normal day
|
|
#
?
Apr 15, 2023 07:44
|
|
|
this is what you get for not closing the friday thread hot coco on the clout
|
|
#
?
Apr 15, 2023 07:47
|
|
|
Chinatown posted:hot cocoa passed out on the couch
|
|
#
?
Apr 15, 2023 07:49
|
|
|
lol
|
|
#
?
Apr 15, 2023 07:51
|
|
|
hot cocoa on the couch posted:there's no doubt in my mind that the schedule will be perfect this friday
|
|
#
?
Apr 15, 2023 07:54
|
|
|
|
|
#
?
Apr 15, 2023 07:56
|
|
|
also lol that the effort poster is very bad at their job
|
|
#
?
Apr 15, 2023 07:57
|
|
|
Bad Purchase posted:also lol that the effort poster is very bad at their job that kind of language earns you a paystub in your PMs buddy
|
|
#
?
Apr 15, 2023 07:59
|
|
|
looking forward to it, on a saturday
|
|
#
?
Apr 15, 2023 07:59
|
|
|
im the technical team lead of a group of 10 im actually not bad at my job. i've just had an extremely difficult last 7 years of stressful project work and have started having severe sleeping problems again in the past couple weeks.
|
|
#
?
Apr 15, 2023 08:00
|
|
|
Sounds like you should go back to work then and stop posting on the forums forever
|
|
#
?
Apr 15, 2023 08:02
|
|
|
I'm sure life is real stressful when you blind move to a new city every 4 months
|
|
#
?
Apr 15, 2023 08:02
|
|
|
I give each city a full 12 months. I wish I went to knoxville instead though im taking a break, it's been a hard day.
|
|
#
?
Apr 15, 2023 08:05
|
|
|
you're gonna be welcomed with open arms in the neo-Confederate South, I can feel it
|
|
#
?
Apr 15, 2023 08:07
|
|
|
Methanar posted:im the technical team lead of a group of 10 lol
|
|
#
?
Apr 15, 2023 08:09
|
|
|
code:https://github.com/thanos-io/thanos/issues/5929 apparently it broke after 0.28? come on.
|
|
#
?
Apr 15, 2023 08:15
|
|
|
I woke up and had to pisssssssssss
|
|
#
?
Apr 15, 2023 08:22
|
|
|
hm nope still broken on 0.28.code:code:code:code:within the IAM trust of the role I want to assume, we pin to specific principals. The prometheus container obviously is using the prometheus SA. I guess I'll need to figure out how to make this an OR and include prometheus here. That's a terraform change. whatever guess i'll do it. https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html code:
|
|
#
?
Apr 15, 2023 08:27
|
|
|
Methanar posted:im actually not bad at my job. Methanar posted:apparently it broke after 0.28? come on.
|
|
#
?
Apr 15, 2023 08:27
|
|
|
wow its actually a pain in the rear end to do a logical OR in iam policies. thankfully somebody figured out the truth table for me https://dev.to/himwad05/aws-iam-how-to-achieve-logical-or-effect-with-multiple-iam-condition-operators-2h0p
|
|
#
?
Apr 15, 2023 08:31
|
|
|
|
| # ? Jun 7, 2024 03:15 |
|
|
drat, dog, i sure give a gently caress about any of that!
|
|
#
?
Apr 15, 2023 08:35
|
|