Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.
Don't make the mistake of viewing cargo cult behaviors as irrational. Many of these businesses are ad businesses. The job of an ad business is to follow trends. If your customers are buying more ads, and spending more on them, you hire and run lots of experiments to improve the ROI on your ads. If the well runs dry, you shift into austerity mode.

Pay attention to the economic analysts, but you can't follow them. You have to go where your customers are.

Adbot
ADBOT LOVES YOU

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...

Vulture Culture posted:

Don't make the mistake of viewing cargo cult behaviors as irrational. Many of these businesses are ad businesses. The job of an ad business is to follow trends. If your customers are buying more ads, and spending more on them, you hire and run lots of experiments to improve the ROI on your ads. If the well runs dry, you shift into austerity mode.

Pay attention to the economic analysts, but you can't follow them. You have to go where your customers are.

With the extremely delicious exception of Twitter, there's no need to actually fire an arbitrary chunk of your employees when you are and will be extremely profitable, especially when you can't actually articulate why except to vaguely gesture and mumble something that sounds like "inflation," while also performing stock buy backs.

When "the well runs dry" means "only $20B in profit instead of $30B" I'm less than empathetic.

12 rats tied together
Sep 7, 2006

if your employer is publicly traded or wants to be, they'll be firing a large chunk of people because the stock market rewards them for doing so, irrespective of whether or not that makes a lot of business sense

if they aren't publicly traded but are accountable to the whims of venture capital they might be in a similar situation depending on which bloodsucking ghoul organization they've accepted money from

Bruegels Fuckbooks
Sep 14, 2004

Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

Volmarias posted:

With the extremely delicious exception of Twitter, there's no need to actually fire an arbitrary chunk of your employees when you are and will be extremely profitable

Just saying "are and will be extremely profitable" excludes twitter, no specific exception needed.

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...

Bruegels Fuckbooks posted:

Just saying "are and will be extremely profitable" excludes twitter, no specific exception needed.

I don't have to specify, no, but I do enjoy doing so. :teslaget:

Nybble
Jun 28, 2008

praise chuck, raise heck
Heck even YC just laid off 20% of their staff on Monday, after their VP was crying crocodile tears on Friday that companies with money in SVB won’t be able to pay employees, despite FDIC always restoring full cash to depositors in their history.

it’s all self inflicted. insert the “buy bUY BUY sell seLL SELL” comic here.

prom candy
Dec 16, 2005

Only I may dance
Anyone have any experience with Meilisearch? I need some kind of searchable document store to power some search/filter pages and the Algolia plan that would cover our needs would cost more than our entire AWS spend.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

prom candy posted:

Anyone have any experience with Meilisearch? I need some kind of searchable document store to power some search/filter pages and the Algolia plan that would cover our needs would cost more than our entire AWS spend.
Since you're on AWS already, have you looked at Amazon Kendra?

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Volmarias posted:

With the extremely delicious exception of Twitter, there's no need to actually fire an arbitrary chunk of your employees when you are and will be extremely profitable, especially when you can't actually articulate why except to vaguely gesture and mumble something that sounds like "inflation," while also performing stock buy backs.

When "the well runs dry" means "only $20B in profit instead of $30B" I'm less than empathetic.
You should not have empathy for capital; they and you both know it's business. The instincts responsible for your self-preservation in a contraction should not assume that the people around you have no idea what they're doing.

prom candy
Dec 16, 2005

Only I may dance

Vulture Culture posted:

Since you're on AWS already, have you looked at Amazon Kendra?

Never even heard of it which is funny because I was googling for like "AWS Search Solutions" last night and it didn't come up. I'm not sure if this is quite what I'm after though, the dashboards we have are built around having lots of separate filters that people can tweak rather than open-ended text search. I don't think I need ML power for like "created by this person, this person, or this person between these two dates and matching these other 3 criteria" type searches. It's actually the kind of thing that I feel like MySQL should be good at but I've spent hours loving with indices and tweaking the queries and so on and with all the joins and filtering and sorting it's just too slow.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

prom candy posted:

Never even heard of it which is funny because I was googling for like "AWS Search Solutions" last night and it didn't come up. I'm not sure if this is quite what I'm after though, the dashboards we have are built around having lots of separate filters that people can tweak rather than open-ended text search. I don't think I need ML power for like "created by this person, this person, or this person between these two dates and matching these other 3 criteria" type searches. It's actually the kind of thing that I feel like MySQL should be good at but I've spent hours loving with indices and tweaking the queries and so on and with all the joins and filtering and sorting it's just too slow.
Joins and filters and sorts are three things that should really not be slow for moderately-sized data sets with a reasonably-designed schema.

If your data is relationally structured and all your full-text queries are going against FULLTEXT indexes, the query planner should give you all the tools you need to optimize those queries out, but you might need some kind of query logging or APM sampling in order to figure out what the queries are that are causing inefficient plans.

prom candy
Dec 16, 2005

Only I may dance

Vulture Culture posted:

Joins and filters and sorts are three things that should really not be slow for moderately-sized data sets with a reasonably-designed schema.

If your data is relationally structured and all your full-text queries are going against FULLTEXT indexes, the query planner should give you all the tools you need to optimize those queries out, but you might need some kind of query logging or APM sampling in order to figure out what the queries are that are causing inefficient plans.

What's moderately-sized in this case? The tables that we struggle with (and that join into each other frequently) each have about 5-10 million records. I think our schema is reasonably-designed but I'm a web developer, not a DBA.

Doktor Avalanche
Dec 30, 2008

5-10 million sounds like a lot but I don't have a proper frame of reference

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

prom candy posted:

What's moderately-sized in this case? The tables that we struggle with (and that join into each other frequently) each have about 5-10 million records. I think our schema is reasonably-designed but I'm a web developer, not a DBA.
5-10 million records in a table is large enough that you need to care a lot about performance to stay responsive, but it's not so large that you need to do much differently. At this size, what I would mostly worry about is making sure you're paging records efficiently, you're avoiding full table scans, and that your database instance has enough memory to keep your hottest data (for your use case, probably your join indexes) in memory as much as possible.

prom candy
Dec 16, 2005

Only I may dance

Vulture Culture posted:

5-10 million records in a table is large enough that you need to care a lot about performance to stay responsive, but it's not so large that you need to do much differently. At this size, what I would mostly worry about is making sure you're paging records efficiently, you're avoiding full table scans, and that your database instance has enough memory to keep your hottest data (for your use case, probably your join indexes) in memory as much as possible.

Thanks! I posted about this in the CI/devops thread too but I think there are probably a couple of indices I need to add for sorting and then it also might be time to stop displaying the total count of things to users because a lot of our `SELECT COUNT(*) FROM thing WHERE stuff` queries are slow even though EXPLAIN doesn't indicate there's anything wrong with them. We have a lot of "Viewing record 1-30 of 512,234" type messaging that probably doesn't need to be there.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

prom candy posted:

Thanks! I posted about this in the CI/devops thread too but I think there are probably a couple of indices I need to add for sorting and then it also might be time to stop displaying the total count of things to users because a lot of our `SELECT COUNT(*) FROM thing WHERE stuff` queries are slow even though EXPLAIN doesn't indicate there's anything wrong with them. We have a lot of "Viewing record 1-30 of 512,234" type messaging that probably doesn't need to be there.
COUNT is really slow on joined/filtered queries against large data sets, yeah.

Also, I'm assuming you're doing that pagination with LIMIT/OFFSET here. Have you looked at the statistical distribution of your offsets/page numbers? Large offsets in MySQL can be very slow.

e: most humans rarely go beyond page 10 of any results, so if you're seeing a lot of results too much beyond that, you've probably got some bot/scraper activity happening.

Vulture Culture fucked around with this message at 21:39 on Mar 14, 2023

CPColin
Sep 9, 2003

Big ol' smile.

Vulture Culture posted:

e: most humans rarely go beyond page 10 of any results, so if you're seeing a lot of results too much beyond that, you've probably got some bot/scraper activity happening.

At some point when I was working at Experts Exchange, we had a problem with bots picking up the "Last Page" links on big lists and slowing us down, so our solution was to see if the requested page was more than halfway through the list (we'd already done a COUNT to get the number of pages) and flip the sort, if so. Fortunately, nobody ever seemed to want to view the pages in the middle of big lists!

Aramoro
Jun 1, 2012




prom candy posted:

Never even heard of it which is funny because I was googling for like "AWS Search Solutions" last night and it didn't come up. I'm not sure if this is quite what I'm after though, the dashboards we have are built around having lots of separate filters that people can tweak rather than open-ended text search. I don't think I need ML power for like "created by this person, this person, or this person between these two dates and matching these other 3 criteria" type searches. It's actually the kind of thing that I feel like MySQL should be good at but I've spent hours loving with indices and tweaking the queries and so on and with all the joins and filtering and sorting it's just too slow.

I’ve used Solr for this and it was pretty good for creating a searchable document store from a relational database.

I used it to replace a legacy search so retaining the old search parameters and just swapping out the back end. Worked well and was pretty fast compared to the old query building.

Aramoro fucked around with this message at 22:42 on Mar 14, 2023

prom candy
Dec 16, 2005

Only I may dance

Vulture Culture posted:

COUNT is really slow on joined/filtered queries against large data sets, yeah.

Also, I'm assuming you're doing that pagination with LIMIT/OFFSET here. Have you looked at the statistical distribution of your offsets/page numbers? Large offsets in MySQL can be very slow.

e: most humans rarely go beyond page 10 of any results, so if you're seeing a lot of results too much beyond that, you've probably got some bot/scraper activity happening.

We have an infinite scroll setup but it's LIMIT/OFFSET based yeah. I imagine it's pretty unlikely anyone is loading up more than a few pages but we do have a media view that's actually pretty fun to scroll through. None of our endpoints are public and I doubt any of our customers are sophisticated enough to be using scrapers (most of them are barely sophisticated enough to use GUIs.) I'd like to switch to cursor based pagination but it gets tricky with non-unique fields. Really I need to get into a position where I can do schema migrations without downtime so I can actually experiment with things.

Aramoro posted:

I’ve used Solr for this and it was pretty good for creating a searchable document store from a relational database.

I used it to replace a legacy search so retaining the old search parameters and just swapping out the back end. Worked well and was pretty fast compared to the old query building.

There's a term I haven't heard in a while. I used to use this at the agency I worked at a decade+ ago. Totally fell off my radar, I'll take another look!

Aramoro
Jun 1, 2012




prom candy posted:


There's a term I haven't heard in a while. I used to use this at the agency I worked at a decade+ ago. Totally fell off my radar, I'll take another look!

Theres a solution to deploy it to k8s as well now.

Macichne Leainig
Jul 26, 2012

by VG
I started a temporary job reassignment this week, with the ulterior motive of getting a better understanding of that team's product and getting better cross-product knowledge across our whole company's development team.

Anyway, long story short, they don't even have me working on that product, they have me working on an identity management application instead, which isn't terrible but is also the most basic CRUD app imaginable and sort of nullifies the "cross-product knowledge" aspect we were hoping for. My supervisor is pissed.

thotsky
Jun 7, 2005

hot to trot
Got two offers, one from the biggest consultancy company here and one from a smaller one specializing in senior devs that has a flat, open and quite lucrative compensation scheme based on billable hours. If I go with the latter my TC will be about unchanged, whereas the former is probably not going to want to match that but has a higher guaranteed pay even if I am not on a project.

Doktor Avalanche
Dec 30, 2008

what's TC?

epswing
Nov 4, 2003

Soiled Meat

I was wondering this too, probably Total Compensation. I guess to account for salary + extras.

wilderthanmild
Jun 21, 2010

Posting shit




Grimey Drawer

Total Compensation. Typically Salary + Bonus + Stock. Sometimes you see people count other stuff, but unless it's something that can reasonably be turned into cash it doesn't really count.

Hughlander
May 11, 2005

wilderthanmild posted:

Total Compensation. Typically Salary + Bonus + Stock. Sometimes you see people count other stuff, but unless it's something that can reasonably be turned into cash it doesn't really count.

Like a place I worked tried to put all of what they covered in health care as TC to try to bring it up to an average salary around them rather than just admiting they were below mean.

wilderthanmild
Jun 21, 2010

Posting shit




Grimey Drawer

Hughlander posted:

Like a place I worked tried to put all of what they covered in health care as TC to try to bring it up to an average salary around them rather than just admiting they were below mean.

Yeah I had a recruiter years ago try to pull this with me. Trying to say when you hear that X company pays $200k or whatever it's really that they count everything to bump that number up. He was literally counting from insurance all the way down poo poo down to snacks in the office, company lunches, etc.

CPColin
Sep 9, 2003

Big ol' smile.

Not much :negative:

thotsky
Jun 7, 2005

hot to trot

Nothing much, what's TC with you?

StumblyWumbly
Sep 12, 2007

Batmanticore!

Hughlander posted:

Like a place I worked tried to put all of what they covered in health care as TC to try to bring it up to an average salary around them rather than just admiting they were below mean.

For folks with a family or persistent medical expenses, this is kinda fair, as long as they list it out separately.
E: It's the inverse of folks maxing salary but having poo poo health insurance.
E2: I guess it only applies to America, too

Hughlander
May 11, 2005

StumblyWumbly posted:

For folks with a family or persistent medical expenses, this is kinda fair, as long as they list it out separately.
E: It's the inverse of folks maxing salary but having poo poo health insurance.
E2: I guess it only applies to America, too

No they had poo poo insurance as well

YanniRotten
Apr 3, 2010

We're so pretty,
oh so pretty

Macichne Leainig posted:

I started a temporary job reassignment this week, with the ulterior motive of getting a better understanding of that team's product and getting better cross-product knowledge across our whole company's development team.

Anyway, long story short, they don't even have me working on that product, they have me working on an identity management application instead, which isn't terrible but is also the most basic CRUD app imaginable and sort of nullifies the "cross-product knowledge" aspect we were hoping for. My supervisor is pissed.

Just go back to your actual job, what are they gonna do to you? Make you do work they tricked you into doing and your management doesn't support you doing? Just ask your manager what they've got for you.

New Yorp New Yorp
Jul 18, 2003

Only in Kenya.
Pillbug

thotsky posted:

Got two offers, one from the biggest consultancy company here and one from a smaller one specializing in senior devs that has a flat, open and quite lucrative compensation scheme based on billable hours. If I go with the latter my TC will be about unchanged, whereas the former is probably not going to want to match that but has a higher guaranteed pay even if I am not on a project.

I've been a consultant for 11 years. The latter's pay scheme is going to gently caress you over hard in one way or another.

In a given week, I am billable 75% of the time if I'm lucky. Internal work, pre-sales and scoping new projects, etc eats up a tremendous amount of time.

This means one of two things:
1) You work 50-60 (or more!) hours a week to be billable for 40 hours.
2) You don't get paid anywhere near what you think you're going to get paid.

Some consulting companies will make part of your job performance billable hours, so scenario #1 is effectively mandatory. My company thankfully has no such policy. If I'm only billable for a third of the week because internal poo poo eats up all my time, I shrug and say "don't make me do so much non-billable poo poo and I'll bill more, okay it's Friday at 5 pm, talk to you on Monday"

thotsky
Jun 7, 2005

hot to trot

New Yorp New Yorp posted:

I've been a consultant for 11 years. The latter's pay scheme is going to gently caress you over hard in one way or another.

In a given week, I am billable 75% of the time if I'm lucky. Internal work, pre-sales and scoping new projects, etc eats up a tremendous amount of time.

This means one of two things:
1) You work 50-60 (or more!) hours a week to be billable for 40 hours.
2) You don't get paid anywhere near what you think you're going to get paid.

Some consulting companies will make part of your job performance billable hours, so scenario #1 is effectively mandatory. My company thankfully has no such policy. If I'm only billable for a third of the week because internal poo poo eats up all my time, I shrug and say "don't make me do so much non-billable poo poo and I'll bill more, okay it's Friday at 5 pm, talk to you on Monday"

I mean, with my last employer I was allocated like 5% because they were incompetent, but apparently consultants in this company don't really work with internal stuff or scoping much, and never sales. You're sold into a team and spend at least a year and sometimes multiple years there. It's possible to be unlucky though, I guess.

I have a friend who works there, and he's been allocated 100% for three years now.

thotsky fucked around with this message at 09:20 on Mar 17, 2023

Sistergodiva
Jan 3, 2006

I'm like you,
I have no shame.

Is there any SA dev discord or am I missremembering?

prom candy
Dec 16, 2005

Only I may dance
Being a dev that's not involved in the sales or scoping process can also be really frustrating when you get put on a project with ridiculous deliverables. My life at an agency improved immensely when I started getting involved in projects during the discovery and estimation phase, even though it's annoying.

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

prom candy posted:

Being a dev that's not involved in the sales or scoping process can also be really frustrating when you get put on a project with ridiculous deliverables. My life at an agency improved immensely when I started getting involved in projects during the discovery and estimation phase, even though it's annoying.
Being involved in scoping and estimating is great, and what's even better is knowing enough about the background of the requirements to be able to draw a line in the sand at implementation time about what is and isn't your/your team's job in making those happen

prom candy
Dec 16, 2005

Only I may dance

Vulture Culture posted:

Being involved in scoping and estimating is great, and what's even better is knowing enough about the background of the requirements to be able to draw a line in the sand at implementation time about what is and isn't your/your team's job in making those happen

The other nice thing is it gives you a chance to actually recommend solutions. Being able to hear someone ask for what sounds like an insane feature and then tease out what their actual business need is and recommend something that will solve their problem in 1/10th the time is a good skill to have and it comes up quite a bit. If you're always just building what people think they want instead of what they actually need there's going to end up with a lot of dissatisfaction on both sides of the table.

Love Stole the Day
Nov 4, 2012
Please give me free quality professional advice so I can be a baby about it and insult you

Vulture Culture posted:

to be able to draw a line in the sand at implementation time about what is and isn't your/your team's job in making those happen
Hello yes our team owns this component, but we don't have capacity at this time and would like to leverage the inner sourcing model.

Adbot
ADBOT LOVES YOU

Vulture Culture
Jul 14, 2003

I was never enjoying it. I only eat it for the nutrients.

Love Stole the Day posted:

Hello yes our team owns this component, but we don't have capacity at this time and would like to leverage the inner sourcing model.
My unironic favorite thing about shared ownership is that sometimes people will really surprise you with what they're able to do. On one project alone, I watched a data scientist submit a feature to an AWS SDK to get something working in a low-lift way, and another data-centered software engineer with very little Linux domain experience investigate and fix an SSSD configuration embedded into a Kubernetes sidecar through several layers of config management glue.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply