Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Jabor
Jul 16, 2010

#1 Loser at SpaceChem

Ihmemies posted:

Actually what the gently caress these tasks are..

So how I'm going to write an infinitely deep SQL query without recursion? Like I just can't use a "for" loop or anything :D Hnngh.. poo poo..

This sounds like it's asking "have you read the documentation/lecture notes closely enough to find the CONNECT BY operator"

Adbot
ADBOT LOVES YOU

GigaPeon
Apr 29, 2003

Go, man, go!
Anybody at PASS this week? I’ll be the guy forcing myself to socialize cause my boss said to ‘be sure to network and have fun’.

Hughmoris
Apr 21, 2007
Let's go to the abyss!

GigaPeon posted:

Anybody at PASS this week? I’ll be the guy forcing myself to socialize cause my boss said to ‘be sure to network and have fun’.

I am not but I'd really like to be there. More importantly, I'd like to be on a team where people want to go to those types of events. Yes, I am a nerd.

GigaPeon
Apr 29, 2003

Go, man, go!

Hughmoris posted:

I am not but I'd really like to be there. More importantly, I'd like to be on a team where people want to go to those types of events. Yes, I am a nerd.

Yeah I’m glad I’m here. The other DB nerds couldn’t make the time for it with their own stuff going on. It’s just regular goon-style social anxiety. Like going to a wedding where you don’t know anyone.

GigaPeon fucked around with this message at 02:28 on Nov 15, 2023

Hughmoris
Apr 21, 2007
Let's go to the abyss!

GigaPeon posted:

Yeah I’m glad I’m here. The other DB nerds couldn’t make the time for it with their own stuff going on. It’s just regular goon-style social anxiety. Like going to a wedding where you don’t know anyone.

Yeah, flying solo at those things can always be a little intimidating at the start. Just remember that mostly everyone is there because they want to talk cool tech with like-minded strangers, so chat them up.

Let us know your thoughts on the event as a whole, after things wrap up. I'm tempted to try and make the SQLBits 2024 in Scotland.

MrMoo
Sep 14, 2000

It's in England though?

https://sqlbits.com/news/sqlbits-2024-unveiled/

Hughmoris
Apr 21, 2007
Let's go to the abyss!

Thanks. For some reason I had it in my had as Edinburgh.

England could be cool, too.

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Hughmoris posted:

England could be cool, too.

(a thousand goons hurtle into the thread at breakneck speed, eager to tell you about all the ways England is not cool)

GigaPeon
Apr 29, 2003

Go, man, go!
My takeaway so far is that I want Redgate’s toys and gizmos but my boss probably won’t buy them.

This thing is hosted by Redgate though so I’m probably being brainwashed.

Also there’s so much AI poo poo. How are people just cool with hooking up their environments to the internet and sending it to a third party?

GigaPeon fucked around with this message at 05:02 on Nov 17, 2023

Hughmoris
Apr 21, 2007
Let's go to the abyss!

GigaPeon posted:

My takeaway so far is that I want Redgate’s toys and gizmos but my boss probably won’t buy them.

This thing is hosted by Redgate though so I’m probably being brainwashed.

Also there’s so much AI poo poo. How are people just cool with hooking up their environments to the internet and sending it to a third party?

Yeah, I get tired of reading random articles about AI so I can't even imagine how much that crap is getting pushed there.

Also, are you seeing much of anything Postgres there? I saw they had a few sessions lined up on the topic but I imagine they're shoved off in a corner somewhere.

GigaPeon
Apr 29, 2003

Go, man, go!
There’s definitely non MS stuff going on here. Postgres even has a vendor booth. We’re a MS shop though so I haven’t dug too deep.

I feel like I could give some of these talks. I figured I could learn some cool tricks at the Git + SQL talk so I wouldn’t have to do weird things to version views and procs but they’re just like ‘here’s what a branch is’


Cool thanks. Also at a whiskey based after party and found someone to talk about smoked meats with so that helps with anxiety.

GigaPeon fucked around with this message at 07:04 on Nov 17, 2023

Lib and let die
Aug 26, 2004

GigaPeon posted:

Also there’s so much AI poo poo. How are people just cool with hooking up their environments to the internet and sending it to a third party?

Look, we have to take risks, it's part of doing business.

At least, that's what Scott Brighton, CEO of Bonterera, parent company of NGP VAN, the Democrat Party's purpose-built voter engagement and fundraising platform has to say about leveraging AI.

God I can't wait 'til '24 kicks into high gear and the bottom falls out.

mortarr
Apr 28, 2005

frozen meat at high speed

GigaPeon posted:

My takeaway so far is that I want Redgate’s toys and gizmos but my boss probably won’t buy them.

This thing is hosted by Redgate though so I’m probably being brainwashed.

Also there’s so much AI poo poo. How are people just cool with hooking up their environments to the internet and sending it to a third party?

Tell me more about the latest redgates gizmos - I used to use their sql toolbelt in another life, and their dependency sniffing tool was a legit lifesaver for the end-of-lifed products I was extracting data from at the time, but I never really got into much else of theirs.

GigaPeon
Apr 29, 2003

Go, man, go!

mortarr posted:

Tell me more about the latest redgates gizmos - I used to use their sql toolbelt in another life, and their dependency sniffing tool was a legit lifesaver for the end-of-lifed products I was extracting data from at the time, but I never really got into much else of theirs.

Here's what I was looking at

SQL Prompt: https://www.red-gate.com/products/sql-prompt/ People at the summit I was talking to were big on this one. Saying it saved them 100s of hours. Refactoring and templates and macros. Let's you define or pick a house style. Of course now they're using ChatGPT integration for some stuff. "Make this sp more efficient". Talking to a Redgate guy at an afterparty, he said that they're planning on making it so you can limit training to your own data for security concerns. I've been using their online formatter for a long time (https://www.red-gate.com/website/sql-formatter) so I was interested in the more robust features of this

Flyway: https://www.red-gate.com/products/flyway/ Devops for DBs. I've always wished that SSMS had some sort of Git integration and that seems to do that as well as automating deployments. I'm mostly an Enterprise Apps guy so I don't build a lot of my own stuff, so I'm sure I won't get to play with it a lot.

Test Data Manager: https://www.red-gate.com/products/test-data-manager/ This is their new thing. Generate large sets of test data based on your own criteria. Also data anonymization/sanitization stuff for PII.

MrMoo
Sep 14, 2000

GigaPeon posted:

SQL Prompt

Flyway

Flyway is pretty nice, although you have to do the work to make it powerful.



That is an impressive choice of styling.

kumba
Nov 8, 2003

I posted my food for USPOL Thanksgiving!

enjoy the ride

Lipstick Apathy
i use SQLPrompt and it is indeed very handy

Lib and let die
Aug 26, 2004

Commas before, you filthy animals

kumba
Nov 8, 2003

I posted my food for USPOL Thanksgiving!

enjoy the ride

Lipstick Apathy
i have successfully convinced my team to start all of their where clauses with

WHERE 1=1

and sqlprompt is very good at making things like this easier

Generic Monk
Oct 31, 2011

GigaPeon posted:

Here's what I was looking at

SQL Prompt: https://www.red-gate.com/products/sql-prompt/ People at the summit I was talking to were big on this one. Saying it saved them 100s of hours. Refactoring and templates and macros. Let's you define or pick a house style. Of course now they're using ChatGPT integration for some stuff. "Make this sp more efficient". Talking to a Redgate guy at an afterparty, he said that they're planning on making it so you can limit training to your own data for security concerns. I've been using their online formatter for a long time (https://www.red-gate.com/website/sql-formatter) so I was interested in the more robust features of this

a few of the older members of my team use this and swear by it. it looks ok but i do balk at the concept of having to pay for plugins to fix the barebones functionality of SSMS. datagrip is in the same ballpark price wise and is generally just way more full featured and integrated; it’s the nicest way i’ve found to write sql.

Lib and let die
Aug 26, 2004

SQL Prompt is really handy for refactoring but I don't know that I'd pay redgate prices just for that. We were always told that its use was non-optional because of stuff like detecting UPDATE or DELETE statements without WHERE clauses and less about making our code pretty.

Tax Oddity
Apr 8, 2007

The love cannon has a very short range, about 2 feet, so you're inevitably out of range. I have to close the distance.
I always found SQL Complete to be better than SQL Prompt, with more features and better formatting at a lower price. They're both great though.

I don't work for them or anything but I was at a company where all of us were using SQL Prompt and I personally evaluated and compared it to SQL Complete. After my evaluation was done we made the switch. This was a few years ago so things could have changed since then.

Just-In-Timeberlake
Aug 18, 2003
SQL Prompt was very good, but I swear with each update it’s gotten worse. I spend an amount of time I’m not happy to be spending fighting it some days.

The snippets manager is a huge timesaver for sure, but it does some real dumb things sometimes. For example, we have a table called Customers, and there is a built in snippet for creating a user that you trigger by typing in ‘cu’, which is fine! Except when I type in ‘Select customerId from cu’ and hit tab expecting the table name to be inserted and not the code to create a new db user, because that’s totally a thing I want in a from clause.

mortarr
Apr 28, 2005

frozen meat at high speed
Thanks all, interesting to see the inclusion of chatgpt although I guess it was inevitable.

kumba posted:

i have successfully convinced my team to start all of their where clauses with

WHERE 1=1

and sqlprompt is very good at making things like this easier

I thought I was the only person who did the 1=1 thing. So handy.

GigaPeon
Apr 29, 2003

Go, man, go!

Lib and let die posted:

SQL Prompt is really handy for refactoring but I don't know that I'd pay redgate prices just for that. We were always told that its use was non-optional because of stuff like detecting UPDATE or DELETE statements without WHERE clauses and less about making our code pretty.

I always write my UPDATE/DELETEs as SELECT statements first because of that one time.

Thanks for all the pointers to Prompt alternatives. Not sure the powers that be will shell out $200/head/year.

GigaPeon fucked around with this message at 00:32 on Nov 22, 2023

Ranzear
Jul 25, 2013

Lib and let die posted:

Commas before, you filthy animals

Sincerely appreciating this way of coping with dogmatic intolerance of trailing commas in TYOOL 2024.

Seventh Arrow
Jan 26, 2005

lazerwolf posted:

I’d be in for a data thread as well. Transitioning my role from Full stack developer to more of a Data Engineering role so I’m interested in discussing more.

A Data Engineering thread is finally upon us, huzzah!

https://forums.somethingawful.com/showthread.php?threadid=4050611&pagenumber=1#lastpost

thanks to monochromagic!

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself
i have a crazy query that I need a bit of help optimizing. most of it is fast, but there's one piece of it that is slowing it down to a screeching halt

Basically I have an endpoint that is returning me a paginated array of posts that belong to threads, which themselves belong to forums. I want to get the users active ban. This all works fine. The part that's slowing this down is the "author_total_posts" because I want to return the count of every single post the user made

JavaScript code:
Array<{
  "id": 123,
  "body": "hello world",
  "author_username": "teen phone cutie",
  "author_id": 1,
  "thread_id": 123,
  ...more_post_and_thread_info
  "active_offense": {
    "is_ban": true,
    "id": 123,
    ...more_info_about_offense
  },
  "author_total_posts": 400 // this is the data that I need that's causing the slowdown
}>
The SQL to generate me this response looks something like this:

SQL code:
SELECT   posts_1.id,
                posts_1.body,
                posts_1.thread_id,
                posts_1.author_id,
                threads.title,
                users.username,
                users.byline,
                forums.id,
                forums.title,
                Group_concat(offenses_1.id),
                Group_concat(offenses_1.is_ban),
                Group_concat(COALESCE(offenses_1.end_time, "2024-01-14T13:01:52.352914Z")),
                Group_concat(offenses_1.post_id),
                offense_lat.id,
                offense_lat.is_ban,
                offense_lat.end_time,
                post_count_lat.count_1
FROM            posts AS posts_1
INNER JOIN      threads
ON              posts_1.thread_id = threads.id
AND             threads.is_deleted = false
AND             posts_1.thread_id = 11
INNER JOIN      forums
ON              forums.id = threads.forum_id
INNER JOIN      users
ON              users.id = posts_1.author_id
LEFT OUTER JOIN offenses AS offenses_1
ON              offenses_1.post_id = posts_1.id
AND             offenses_1.approved_user_id > 0
LEFT OUTER JOIN lateral
                (
                         select   offenses_2.id                      AS id,
                                  offenses_2.end_time                AS end_time,
                                  offenses_2.is_ban                  AS is_ban,
                                  offenses_2.post_id                 AS post_id,
                                  offenses_2.private_message_post_id AS private_message_post_id
                         FROM     offenses                           AS offenses_2
                         WHERE    (
                                           offenses_2.is_ban = true
                                  OR       offenses_2.end_time > "2024-01-14T13:01:52.352914Z")
                         AND      offenses_2.approved_user_id > 0
                         AND      offenses_2.offender_id = posts_1.author_id
                         ORDER BY offenses_2.is_ban DESC,
                                  offenses_2.end_time DESC limit 1) AS offense_lat
ON              true = 1
LEFT OUTER JOIN lateral
                (
                           SELECT     count(posts_2.thread_id) AS count_1
                           FROM       posts                    AS posts_2
                           INNER JOIN threads
                           ON         threads.id = posts_2.thread_id
                           WHERE      posts_1.author_id = posts_2.author_id
                           AND        threads.is_deleted = false) AS post_count_lat
ON              true = 1
WHERE           threads.id = 11
GROUP BY        posts_1.id,
                offense_lat.id,
                post_count_lat.count_1
ORDER BY        posts_1.posted_at ASC
LIMIT 25
OFFSET 475
The problem I'm having is despite this being a paginated request to get 25 posts, that last LEFT OUTER JOIN takes like 2 seconds to process, and I can't figure out a more optimal way to get the total users posts for each post that I want to return.

Hopefully this makes sense. I can provide more information about the tables if need be but yeah the gist is just that last LEFT OUTER JOIN takes forever to get the total post count for each post author that's being returned in a list of posts

teen phone cutie fucked around with this message at 02:03 on Jan 15, 2024

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe
is that absolutely the smallest example you can create that exhibits the undesired behaviour?

you refer to "that second LEFT OUTER JOIN" but then you refer to it as a query that counts posts. which appears to be the third LEFT OUTER JOIN in that query; the second "lateral" one.

I don't understand why you need to use this fancy LATERAL feature that I've never encountered before; why do you need to do that rather than just put the join logic in the ON clause? What, if anything, is different if you do? (this applies to both "lateral" joins)

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe
In most cases where you refer to multiple instances of a table you give each instance a unique alias, but you didn't do that for "threads". It's referenced in the outermost query and in the counting query as "threads". I can't imagine that "threads" in the counting query could sensibly refer to anything but the most local one, but for consistency if nothing else you might as well make sure.

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself

Hammerite posted:

is that absolutely the smallest example you can create that exhibits the undesired behaviour?

you refer to "that second LEFT OUTER JOIN" but then you refer to it as a query that counts posts. which appears to be the third LEFT OUTER JOIN in that query; the second "lateral" one.

I don't understand why you need to use this fancy LATERAL feature that I've never encountered before; why do you need to do that rather than just put the join logic in the ON clause? What, if anything, is different if you do? (this applies to both "lateral" joins)

yep sorry i meant the third. The lateral thing I picked up from stackoverflow, so I'm not sure if there's an easier way to accomplish what I want. SQL is not my biggest strength.

As I understand it, the reason I need the lateral subquery is because the main query is getting a page of posts.

So let's say for example the query is getting page 301 of the "SELECT * FROM Questions WHERE Type = 'Stupid'" thread. BUT I want to also get all the posts you've made on SA total and list it under your avatar. So in your case you have "14748" posts and I need to get that from all threads that are not deleted.

So the thread in the outer-most query is just this thread, but in the lateral subquery, it's all threads you've posted in that are not deleted

Hammerite posted:

In most cases where you refer to multiple instances of a table you give each instance a unique alias, but you didn't do that for "threads". It's referenced in the outermost query and in the counting query as "threads". I can't imagine that "threads" in the counting query could sensibly refer to anything but the most local one, but for consistency if nothing else you might as well make sure.

so you're saying make 2 different aliases for each, the outer and the subquery?

teen phone cutie fucked around with this message at 02:22 on Jan 15, 2024

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself

Hammerite posted:

is that absolutely the smallest example you can create that exhibits the undesired behaviour?

When I remove the last outer join, the query goes from 2 seconds to 100ms, so I'm 100% certain it's the problem. Unless, you're suggesting I try something else?

RandomBlue
Dec 30, 2012

hay guys!


Biscuit Hider

teen phone cutie posted:

When I remove the last outer join, the query goes from 2 seconds to 100ms, so I'm 100% certain it's the problem. Unless, you're suggesting I try something else?

How does that last query in the outer join run by itself for a single poster id or all posters since you're using it that way? Have you run an explain plan on that?

e: Oh, just read what that lateral join does, that's likely the problem. Don't see why you can't just use a standard outer join here with grouping, that should still work with pagination.

RandomBlue fucked around with this message at 02:45 on Jan 15, 2024

McGlockenshire
Dec 16, 2005

GOLLOCKS!

teen phone cutie posted:

When I remove the last outer join, the query goes from 2 seconds to 100ms, so I'm 100% certain it's the problem. Unless, you're suggesting I try something else?

What's EXPLAIN say?

Hughmoris
Apr 21, 2007
Let's go to the abyss!

McGlockenshire posted:

What's EXPLAIN say?

Yeah, in my limited experience this is the first place to start when you want to make a query run faster. Look at the executed query plan. Share here if you're able to.

Jabor
Jul 16, 2010

#1 Loser at SpaceChem
Not directly answering your question, but in many cases where something turns out to be too expensive to compute on every single page load, the right answer is to store the computed result somewhere and update it every time it changes.

redleader
Aug 18, 2005

Engage according to operational parameters
use a separate query to get the total count. less to do means less chance the execution engine will gently caress you over

Hammerite
Mar 9, 2007

And you don't remember what I said here, either, but it was pompous and stupid.
Jade Ear Joe

Jabor posted:

Not directly answering your question, but in many cases where something turns out to be too expensive to compute on every single page load, the right answer is to store the computed result somewhere and update it every time it changes.

This is probably the most practical solution to your immediate problem. Denormalise by storing a user's number of posts as a column in the User table. If it's important that the number be accurate and you're worried that you'll miss cases where it should be updated, there are strategies for dealing with that (triggers, scheduling tasks to update the column)

teen phone cutie posted:

so you're saying make 2 different aliases for each, the outer and the subquery?

I don't think so. I mean that where you have "INNER JOIN threads" in the outer query, to be consistent with your practice for other tables it should have been "INNER JOIN threads AS threads_1" and where you have "INNER JOIN threads" in the subquery it would be "INNER JOIN threads AS threads_2".

What happens if you replace the problematic JOIN with this:

code:
LEFT OUTER JOIN (
    SELECT
        "posts_2"."author_id",
        COUNT(*) AS "count_1"
    FROM
        "posts" AS "posts_2"
        INNER JOIN "threads" AS "threads_2" ON
            "threads_2"."id" = "posts_2"."thread_id"
    WHERE
        "threads_2"."is_deleted" = FALSE
    GROUP BY
        "posts_2"."author_id"
) AS "post_count_lat" ON
    "post_count_lat"."author_id" = "posts_1"."author_id"
 
I only just noticed that the outer query has a GROUP BY clause, and I don't understand why that is there or why those specific columns are the grouping ones. If it's necessitated by the uses of Group_concat in the outer query, doesn't it make more sense to put that grouping in a subquery defining offenses_1?

nielsm
Jun 1, 2009



Or you could put the posts per user per thread logic into a CTE which is easy to join with.

teen phone cutie
Jun 18, 2012

last year i rewrote something awful from scratch because i hate myself

McGlockenshire posted:

What's EXPLAIN say?

Hammerite posted:

This is probably the most practical solution to your immediate problem. Denormalise by storing a user's number of posts as a column in the User table. If it's important that the number be accurate and you're worried that you'll miss cases where it should be updated, there are strategies for dealing with that (triggers, scheduling tasks to update the column)

I don't think so. I mean that where you have "INNER JOIN threads" in the outer query, to be consistent with your practice for other tables it should have been "INNER JOIN threads AS threads_1" and where you have "INNER JOIN threads" in the subquery it would be "INNER JOIN threads AS threads_2".

What happens if you replace the problematic JOIN with this:

code:
LEFT OUTER JOIN (
    SELECT
        "posts_2"."author_id",
        COUNT(*) AS "count_1"
    FROM
        "posts" AS "posts_2"
        INNER JOIN "threads" AS "threads_2" ON
            "threads_2"."id" = "posts_2"."thread_id"
    WHERE
        "threads_2"."is_deleted" = FALSE
    GROUP BY
        "posts_2"."author_id"
) AS "post_count_lat" ON
    "post_count_lat"."author_id" = "posts_1"."author_id"
 
I only just noticed that the outer query has a GROUP BY clause, and I don't understand why that is there or why those specific columns are the grouping ones. If it's necessitated by the uses of Group_concat in the outer query, doesn't it make more sense to put that grouping in a subquery defining offenses_1?

lemme give this stuff a shot when i get some time. will report back!

Adbot
ADBOT LOVES YOU

kiwid
Sep 30, 2013

Can someone help me understand what is happening in this WHERE clause?

SQL code:
WHERE (
	(
		NULL IS NULL AND v.contract_nbr = 0
	) 
	OR
	(
		v.contract_nbr > 0 AND (
			(
				NULL IS NULL AND 'Y' = 'Y'
			) OR (
				NULL IS NOT NULL AND offsite_storage_id IS NULL
			)
		)
	)
)
When would NULL IS NOT NULL ever be true?

Does this entire mess just reduce down to v.contract_nbr >= 0?

kiwid fucked around with this message at 20:02 on Jan 31, 2024

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply