Infra/networking thread

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > YOSPOS > Infra/networking thread

Qtotonibudinibudet: Nov 7, 2011; Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

animist posted:

i contemplate your ops wisdom and am enlightened.

question: if the problem is that ObscureLang doesn't support authentication, tracing, etc, how does a service mesh help? it seems to me that those things interact with actual functionality in complex ways. so you either need to wire the service mesh in at the ObscureLang source code layer, or you'd need some hella complicated request inspection code at the service mesh layer.

like, how do you trace a request through a language that doesn't support tracing? do you just correlate incoming and outgoing requests by time received or something?

a distributed trace having a basic "entered this service, exited this service" span in the chain of services doing something with a request is still more than nothing. ideally your application does tracing and adds its own "also i made a db query in 0.03s" span, but you take what you can get

# ? May 28, 2020 22:56

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 17:47

freeasinbeer: Mar 26, 2015; by Fluffdaddy

yeah that�s a dirty secret, you still have to touch all those lovely apps and add instrumentation. if your gonna do that, why not just natively trace?

https://istio.io/faq/distributed-tracing/#how-distributed-tracing-works

# ? May 29, 2020 02:11

Qtotonibudinibudet: Nov 7, 2011; Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

ah, yeah, good point. copy-paste an existing trace-id header is easier than properly instrumenting, but still :effort:

# ? May 29, 2020 03:46

kitten emergency: Jan 13, 2008; get meow this wack-ass crystal prison

well i can talk to the tracing side of this

one big advantage of service mesh for tracing is that it neatly ignores the problem of "wahhh babby cant write like fifteen loving lines of inject/extract code and redeploy" or "for some reason this service eats and discards headers it didn't expect"* by moving trace context propagation out of band with the service. you're still able to get value from these traces, as they'll give you latency info and help you visualize/map out your service dependency graph. ideally you'd also have your services instrumented as well so you can get some depth, but the mesh helps bootstrap the whole shebang for you while also providing other handy services.

generally if you do have a service that eats your tracing headers and you can't get around it, your best bet is to hope that you don't really care what happens after that.

*note, istio specifically doesn't help here because it does require services to transparently forward headers. you can get around this problem with some DIY ingenuity, it really depends on how much you give a poo poo. the simplest way is to wrap your bad service in a nginx proxy or something and have that manage the trace headers for your service instance.

# ? May 29, 2020 04:18

kitten emergency: Jan 13, 2008; get meow this wack-ass crystal prison

also you should natively trace anyway, agents and auto-instrumentation is the path of sadness

# ? May 29, 2020 04:20

Qtotonibudinibudet: Nov 7, 2011; Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

uncurable mlady posted:

auto-instrumentation is the path of sadness

auto-anything is generally the path of sadness.

however, we live in a sad reality where some poor, pollyanna ops team wants to dance around ossified bigcorp app team that aint gonna do poo poo.

there's a lot of things you probably shouldn't do in the middleware that get done in the middleware because tinpot fiefdom does't want to do it properly

# ? May 29, 2020 06:07

ate shit on live tv: Feb 15, 2004; by Azathoth

CMYK BLYAT! posted:

auto-anything is generally the path of sadness.

however, we live in a sad reality where some poor, pollyanna ops team wants to dance around ossified bigcorp app team that aint gonna do poo poo.

there's a lot of things you probably shouldn't do in the middleware that get done in the middleware because tinpot fiefdom does't want to do it properly

It's just too hard. So this is why we need layer 2 adjacency and vmware bullshit.

# ? May 30, 2020 08:10

Schadenboner: Aug 15, 2011; by Shine

suffix posted:

ethernet bondage? you must be a twisted pair

# ? May 30, 2020 21:44

Jimmy Carter: Nov 3, 2005; THIS MOTHERDUCKER
FLIES IN STYLE

OkCupid ran their entire site on 5 servers in 2012 how did we stray so far from the god's light

# ? May 30, 2020 21:54

ate shit on live tv: Feb 15, 2004; by Azathoth

Jimmy Carter posted:

OkCupid ran their entire site on 5 servers in 2012 how did we stray so far from the god's light

The russian's used a pencil

# ? May 31, 2020 05:04

abigserve: Sep 13, 2009; this is a better avatar than what I had before

Jimmy Carter posted:

OkCupid ran their entire site on 5 servers in 2012 how did we stray so far from the god's light

yeah sure it ran on 5 but could it scale to 1000???!?!?

# ? May 31, 2020 05:16

Qtotonibudinibudet: Nov 7, 2011; Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

Jimmy Carter posted:

OkCupid ran their entire site on 5 servers in 2012 how did we stray so far from the god's light

and in C to boot

# ? May 31, 2020 05:34

Farmer Crack-Ass: Jan 2, 2001; this is me posting irl

Jimmy Carter posted:

OkCupid ran their entire site on 5 servers in 2012 how did we stray so far from the god's light

i don't deal with code so my cynical assumption is that a lot of "scaling" is basically "if we can fully automate spinning up and tearing down server instances, it won't matter how often our shoddy code crashes!"

# ? May 31, 2020 06:02

Jimmy Carter: Nov 3, 2005; THIS MOTHERDUCKER
FLIES IN STYLE

after doing some more research on this:

they decided to do their startup in Hard Mode and had a strong Not Invented Here culture and did everything from scratch in C++ where everything got turned into async calls, with a dev team of around 10, which included building their own web server

Then after making one too many high-quality data science posts, anyone with that talent got poached to work elsewhere, then Tinder invented the swiping UI paradigm when OKC didn't feel like jumping head-first into mobile and here we are now.

# ? May 31, 2020 06:16

Farmer Crack-Ass: Jan 2, 2001; this is me posting irl

OKCupid was able to make their own web server software alongside building their entire website with a dev team of ten people?

how many thousands of shithead devs does twitter employ again?

# ? May 31, 2020 21:17

the talent deficit: Dec 20, 2003; self-deprecation is a very british trait, and problems can arise when the british attempt to do so with a foreign culture

Farmer Crack-rear end posted:

OKCupid was able to make their own web server software alongside building their entire website with a dev team of ten people?

how many thousands of shithead devs does twitter employ again?

making a web server isn't that hard if you only need it to run your bespoke web application

# ? Jun 1, 2020 01:24

psiox: Oct 15, 2001; Babylon 5 Street Team

i still think it's cool that the livejournal people made lasting contributions to distributed computin'

also too bad okcupid is owned by the match people now or i'd be tempted to use it. oh well, too late to enter that game anyway now that we're in the cool zone

# ? Jun 1, 2020 03:18

mod saas: May 4, 2004; Grimey Drawer

hi i live under a rock what is this cool zone thing all about

# ? Jun 1, 2020 04:17

Bored Online: May 25, 2009; We don't need Rome telling us what to do.

mod saas posted:

hi i live under a rock what is this cool zone thing all about

your in it

# ? Jun 1, 2020 04:24

Captain Foo: May 11, 2004; we vibin'
we slidin'
we breathin'
we dyin'

mod saas posted:

hi i live under a rock what is this cool zone thing all about

search for cool zone on Twitter

# ? Jun 1, 2020 14:50

ate shit on live tv: Feb 15, 2004; by Azathoth

mod saas posted:

hi i live under a rock what is this cool zone thing all about

# ? Jun 2, 2020 01:16

cowboy beepboop: Feb 24, 2001

Jimmy Carter posted:

OkCupid ran their entire site on 5 servers in 2012 how did we stray so far from the god's light

doesn't stack overflow run on some tiny amount of servers as well

# ? Jun 2, 2020 03:59

MrMoo: Sep 14, 2000

PlentyOfFish was like 2 Windows servers and one dev? for a long time. Nice few write ups on HighScalability.com from 2009.

http://highscalability.com/plentyoffish-architecture

# ? Jun 2, 2020 16:56

distortion park: Apr 25, 2011

my stepdads beer posted:

doesn't stack overflow run on some tiny amount of servers as well

yeah it's all sql server and like one search index . https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/

quote:

When you remove redundancy here�s what Stack Exchange needs to run (while maintaining our current level of performance):

2 SQL servers (SO is on one, everything else on another�they could run on a single machine still having headroom though)
2 Web Servers (maybe 3, but I have faith in just 2)
1 Redis Server
1 Tag Engine server
1 elasticsearch server
1 Load balancer
1 Network
1 ASA
1 Router

(we really should test this one day by turning off equipment and seeing what the breaking point is)

that's fewer the startup i work for which has like 10 customers

# ? Jun 3, 2020 10:55

abigserve: Sep 13, 2009; this is a better avatar than what I had before

Running one server is easy - running a thousand servers that all do something different is hard and that's why we are where we are

# ? Jun 3, 2020 15:00

Bloody: Mar 3, 2013

Yeah, microservices are dumb

# ? Jun 3, 2020 17:26

GenJoe: Sep 15, 2010; Rehabilitated?

That's just a bullshit word.

while we're on this topic, I have a project that uses postgresql and I'm worried about its read/write performance, particularly when accessing it with dapper. I haven't done anything fancier than set up a base postgres with docker and make a few tables:

code:

        const int RUNS = 1000;
        
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        for (int i = 0; i < RUNS; i++)
        {
            var transaction = _connection.BeginTransaction();
            _connection.Query("SELECT id FROM backend_character where id = 2", transaction);
            transaction.Commit();   
        }

        stopwatch.Stop();

all on an SSD, takes 2.5 seconds, so 400 TPS.

pgbench is giving me these stats for a select-only test:

code:

pgbench -c 1 -T 30 -S
...
tps = 4302.334840

what am I supposed to expect as reasonable for modern DB performance? I'm not database guy and have no idea where we're at with modern server throughput etc.

GenJoe fucked around with this message at 20:17 on Jun 3, 2020

# ? Jun 3, 2020 20:07

Jonny 290: May 5, 2005; [ASK] me about OS/2 Warp

we have some former letsencrypt employees and they manage certs for over 100M domains on a half rack of servers somewhere

# ? Jun 3, 2020 20:15

animist: Aug 28, 2018

edge cases are hard but rebooting is easy

# ? Jun 3, 2020 21:54

distortion park: Apr 25, 2011

GenJoe posted:

while we're on this topic, I have a project that uses postgresql and I'm worried about its read/write performance, particularly when accessing it with dapper. I haven't done anything fancier than set up a base postgres with docker and make a few tables:
code:
        const int RUNS = 1000;
        
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        for (int i = 0; i < RUNS; i++)
        {
            var transaction = _connection.BeginTransaction();
            _connection.Query("SELECT id FROM backend_character where id = 2", transaction);
            transaction.Commit();   
        }

        stopwatch.Stop();
all on an SSD, takes 2.5 seconds, so 400 TPS.

pgbench is giving me these stats for a select-only test:
code:
pgbench -c 1 -T 30 -S
...
tps = 4302.334840
what am I supposed to expect as reasonable for modern DB performance? I'm not database guy and have no idea where we're at with modern server throughput etc.

idk what to expect in your exact case, but dapper is pretty fast (there are probably a bunch of driver options you can play with. you should also consider what parts are actually going to be sync etc, e.g. your demo is fully parallelizable and the db will generally handle this well.

# ? Jun 5, 2020 11:30

Bulgogi Hoagie: Jun 1, 2012; We

GenJoe posted:

while we're on this topic, I have a project that uses postgresql and I'm worried about its read/write performance, particularly when accessing it with dapper. I haven't done anything fancier than set up a base postgres with docker and make a few tables:
code:
        const int RUNS = 1000;
        
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        for (int i = 0; i < RUNS; i++)
        {
            var transaction = _connection.BeginTransaction();
            _connection.Query("SELECT id FROM backend_character where id = 2", transaction);
            transaction.Commit();   
        }

        stopwatch.Stop();
all on an SSD, takes 2.5 seconds, so 400 TPS.

pgbench is giving me these stats for a select-only test:
code:
pgbench -c 1 -T 30 -S
...
tps = 4302.334840
what am I supposed to expect as reasonable for modern DB performance? I'm not database guy and have no idea where we're at with modern server throughput etc.

in my experience if you want more perf from postgres give it a beefier cpu host to work on

# ? Jun 5, 2020 22:28

MrMoo: Sep 14, 2000

Why is there a transaction for a read-only select query? Also using prepared statements would usually be smarter.

# ? Jun 6, 2020 00:19

GenJoe: Sep 15, 2010; Rehabilitated?

That's just a bullshit word.

my example's doing 1k/s after removing those transactions, compared to pgbench's 4k/s, so I'm wondering how much of this is driver overhead and how much might be attributed to pgbench's schema being different from mine? (the underlying test in both is just a simple single-select query on an id column)

dapper doesn't let you use prepared statements, but the underlying ngpsql driver has an automatic prepare feature that caches common commands and prepares them for you, as long as you toggle it on.

I'll mess around with a connection pool, it looks like that should help. this is for a game (more of an MMO) so there's one long-lived server that needs to be able to fulfill a bunch of events simultaneously. pgbench, running with 16 connections and 4 threads, can do 33k read transactions per second which is more than enough.

# ? Jun 6, 2020 02:46

MrMoo: Sep 14, 2000

I would expect a simple lookup like that to be orders of 100,000s to millions per second, and not touching disk at all. There is definitely something highly inefficient in the configuration.

The test suite for Dapper appears to show parameterized queries, that should be the correct form to feed auto-prepare logic.

code:

_connection.Query("SELECT id FROM backend_character where id = @Id", new { Id = 2 });

https://github.com/StackExchange/Dapper/blob/master/Dapper.Tests.Performance/LegacyTests.cs#L156

# ? Jun 6, 2020 03:10

Qtotonibudinibudet: Nov 7, 2011; Omich poluyobok, skazhi ty narkoman? ya prosto tozhe gde to tam zhivu, mogli by vmeste uyobyvat' narkotiki

i lol at the recent spate of "we didn't choose kubernetes" posts on hn recently from companies that are mature enough to employ a team of infra people qualified to make and implement that decision

realistically 80% of the audience is immature companies that have maybe one person working infra full time, who will fail to implement kubernetes or non-kubernetes effectively regardless because lol at having one person responsible for that

# ? Jun 9, 2020 04:00

Nomnom Cookie: Aug 30, 2009

I rolled out k8s more or less singlehandedly at my last job and it was fine. poo poo, I wasn't even doing infra full time. It's not something that a fullstack dev who just installed Ubuntu for the first time last Thursday is going to be able to do successfully, though. It also requires being willing to adapt your requirements to what is easy and effective, instead of going full Aspie and insisting that everything must be exactly how you want it.

# ? Jun 9, 2020 06:49

Schadenboner: Aug 15, 2011; by Shine

Why did the packet wrangling thread turn into some sort of containerized microservice hell?

:ohdear:

# ? Jun 9, 2020 13:39

12 rats tied together: Sep 7, 2006

Farmer Crack-rear end posted:

i don't deal with code so my cynical assumption is that a lot of "scaling" is basically "if we can fully automate spinning up and tearing down server instances, it won't matter how often our shoddy code crashes!"

im sure some degree of that exists somewhere in the world, but with my current employer the company basically makes money per CPU cycle, so we have a fleet of edge servers that are dedicated to the part that makes us money and then we have a huge log pipeline that is responsible for turning that data into something we can bill clients with

in this case in order to make the most amount of money in any given second, we need exactly as much log processing infrastructure to process all of that data within an appropriate timeframe. the amount of input data scales with overall internet activity, which fluctuates as people wake up, go to work, get home from work, etc.

if your employer makes money off of the internet demand curve in any way, however bullshit it might be, they are going to benefit from regional scaling optimizations that follow it. it's both not reasonable to manually scale to follow this pattern and not possible right-size a "web scale" (lmao) service, in general

CMYK BLYAT! posted:

k8s isn't a troll: google want to provide some sort of lingua franca around managing computing resources in modern environments based on their practical experience running one. everyone else has done so on their particular cloud compute platform in myriad ways, and there are legion sysadmins saying "by god we can continue to use provider-native tools to do the same poo poo", and they're not wrong, but they're not providing a lingua franca,

this was me, the guy arguing against k8s because it is extra complexity for no new functionality, until we hit a developer:sysadmin ratio where I was being asked "what subnet should I pick?" upwards of 5x per day by people who didn't read the docs for picking a subnet.

the lingua franca goes from not mattering much to being the only reason you can focus for reasons that are totally out of your control, so, you should optimize towards it whenever possible. even if that means adding complexity.

# ? Jun 9, 2020 18:31

abigserve: Sep 13, 2009; this is a better avatar than what I had before

I think you won't find that many people arguing against cloud delivered k8s

On prem? Different story

# ? Jun 9, 2020 22:17

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 17:47

my homie dhall: Dec 9, 2010; honey, oh please, it's just a machine

imo on-prem is the best place for it. if you already have nice apis for controlling infrastructure k8s adds features you may not need. for example if I�m using ec2 I can already use things like security groups for restricting access between VMs or ELBs for L4/L7 load balancing. Kubernetes gives some nice APIs like NetworkPolicy, Ingress, and LoadBalancer for doing the same in a standardized way on-prem and you don�t have to worry about it being some janky API written by your network team. you also get things like service discovery via dns.

it�s ideal on-prem operationally speaking as well because when a vm goes bad in the cloud you delete it and the ASG creates a fresh one on a different hypervisor. you obviously can�t do that on-prem so you�d like to just have a pool of workers that can be pulled for maintenance and returned for service in a way where workloads will get rescheduled which is what k8s offers

# ? Jun 9, 2020 22:55

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > YOSPOS > Infra/networking thread

«‹›34 »