Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Supersonic
Mar 28, 2008

You have used 43 of 300 characters allowed.
Tortured By Flan
Whoops, wrong thread.

Adbot
ADBOT LOVES YOU

Macichne Leainig
Jul 26, 2012

by VG
I definitely struggled with ORMs until I had a good handle on SQL proper. Being able to debug the SQL statements that the ORM prepares is a pretty good skill to have when you inevitably run into some weird edge case.

I think SQLAlchemy in particular has one oddity where if you try to count from a table, it does a SELECT * FROM instead of something like SELECT 1 FROM which causes some performance issues as the target table increases in size and complexity.

Ranzear
Jul 25, 2013

Every ORM I've interacted with I end up falling back to just preparing SQL statements and plugging stuff in, especially if I need multiple reads and writes in a proper transaction to potentially roll back.

Protocol7 posted:

I think SQLAlchemy in particular has one oddity where if you try to count from a table, it does a SELECT * FROM instead of something like SELECT 1 FROM which causes some performance issues as the target table increases in size and complexity.

That will depend on the server actually. MariaDB doesn't care, for instance, and because it's the default in some/most distros it's probably not even a common problem.

redleader
Aug 18, 2005

Engage according to operational parameters

Protocol7 posted:

I think SQLAlchemy in particular has one oddity where if you try to count from a table, it does a SELECT * FROM instead of something like SELECT 1 FROM which causes some performance issues as the target table increases in size and complexity.

this occurs when your database is an idiot

Computer viking
May 30, 2011
Now with less breakage.

I've just spent some time doing random bugfixes in a Qt application I once wrote for internal use at work - it prints labels to a network connected label printer.

One of the things I spent some time on was moving it from Qt 4.something to 6.2, to try to keep up with the supported versions.

We still have people using Windows 7, because it's the only thing officially supported on the relevant internal network.

Qt6 has removed support for windows 7. Sigh.

Computer viking fucked around with this message at 14:45 on Jan 7, 2022

Macichne Leainig
Jul 26, 2012

by VG

redleader posted:

this occurs when your database is an idiot

Well it's MySQL, so... yes?

nightbae smokewheat
Feb 11, 2011

I'm goofing around with image recognition software for a hobby project, and my end goal is analogous to car counting as demonstrated here:

https://www.youtube.com/watch?v=n_Og933KtAE

So far, I've figured out how to install labelImg and ImageAI and use these to annotate a training image dataset, and to train a YOLOv3 model that can detect car analogs in still images and video. Today I realized that to do counting I need to perform not only object detection, but object tracking as well, and I don't think that ImageAI supports the latter.

1) Do I need to drop ImageAI and cobble together something like this or this instead?

2) In case there is a go-to solution for this that I'm too stupid to have found already: can anyone recommend their favorite open source solution for object detection and tracking in video?

CarForumPoster
Jun 26, 2013

⚡POWER⚡

nightbae smokewheat posted:

I'm goofing around with image recognition software for a hobby project, and my end goal is analogous to car counting as demonstrated here:

https://www.youtube.com/watch?v=n_Og933KtAE

So far, I've figured out how to install labelImg and ImageAI and use these to annotate a training image dataset, and to train a YOLOv3 model that can detect car analogs in still images and video. Today I realized that to do counting I need to perform not only object detection, but object tracking as well, and I don't think that ImageAI supports the latter.

1) Do I need to drop ImageAI and cobble together something like this or this instead?

2) In case there is a go-to solution for this that I'm too stupid to have found already: can anyone recommend their favorite open source solution for object detection and tracking in video?

I have a kinda not helpful answer because it sounds like you're looking for something that has abstracted away the challenge of associating the individual objects in the frames to common IDs. I am not sure if such a package exists but in radar parlance this is called "track association" and the way it's often performed for 3D space is using an initial set of detections that are sufficiently proximal to feed a Kalman filter, then predict the volume where the tracked object "could" go, given the kinematics of the object. If a second detection is sufficiently "similar" and in the correct volume in space, associate them and update the Kalman filter.

No idea if this is what is done in video image track association, but googling that term along with "YOLOv3" I found this which seems like what you want: https://github.com/theAIGuysCode/yolov3_deepsort

Track association in 3D when someone really wants to break tracks is a hard topic mathematically. You can see why creating a bunch of targets (e.g. w/chaff) or towed decoys can work. Detection is the easy part.

EDIT: I replied without reading your questions. The answer to 1 is: Yea probably. It looks like the ImageAI repo isnt actively supported. They talk about switching to PyTorch but last commit was May 2021. I dont see any obvious indicators they do track associations.

CarForumPoster fucked around with this message at 00:36 on Jan 8, 2022

nightbae smokewheat
Feb 11, 2011

Thanks; yes, I have seen deepsort come up in a few searches too so I will start looking into that this weekend. Fortunately I'm not a state-level actor, but instead just a garden-variety dumbass, so I don't have to worry about my car analogues dropping chaff or otherwise actively breaking track. :toot:

nightbae smokewheat fucked around with this message at 02:31 on Jan 9, 2022

Computer viking
May 30, 2011
Now with less breakage.

Though that said, being able to flag it when cars do break out of their path, intersect, and/or release fragments could perhaps be useful on its own.

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...

Computer viking posted:

Though that said, being able to flag it when cars do break out of their path, intersect, and/or release fragments could perhaps be useful on its own.

Relevant:

https://www.youtube.com/watch?v=k7YVxLLIuGM

CarForumPoster
Jun 26, 2013

⚡POWER⚡

nightbae smokewheat posted:

Thanks; yes, I have seen deepsort come up in a few searches too so I will start looking into that this weekend. Fortunately I'm not a state-level actor, but instead just a garden-variety dumbass, so I don't have to worry about my car analogues dropping chaff or otherwise actively breaking track. :toot:

Unless in a controlled environment, you will find nature has many ways to break your tracks. Whatever you wanna use, anticipate that the same thing in reality will end up getting multiple tracks assigned to it at unexpected times over the duration of interest.

Computer viking
May 30, 2011
Now with less breakage.


:hmmyes:

hbag
Feb 13, 2021

i uh
im getting this in python
AttributeError: 'dict' object has no attribute 'descendants'

this makes no sense, i thought dicts did have descendants, that was the whole point
idk heres my code lmao
Python code:
            for post in posts:
                #resp_object[post] = 'wahooo'
                resp_object['cum'].append('wahoo')
neither of these worked

nightbae smokewheat
Feb 11, 2011

hbag posted:

cum wahoo
                    /

nielsm
Jun 1, 2009



Well, what are the objects inside your resp_object dictionary actually? Are they lists or dictionaries or some custom class of yours or a class from some framework you haven't said anything about? Are you sure that the object you pull out of the dictionary is even the right one?

nightbae smokewheat
Feb 11, 2011

Update:

nightbae smokewheat posted:

1) Do I need to drop ImageAI and cobble together something like this or this instead?

2) In case there is a go-to solution for this that I'm too stupid to have found already: can anyone recommend their favorite open source solution for object detection and tracking in video?

I've gotten this working. I did indeed need to drop labelImg and ImageAI, and instead cobble together my own solution using Roboflow and a Google Colab notebook implementing this and a few other tutorials. End result is that I have a Yolov4 model trained for my use case that does a great job making detections in video, and a tuned Deepsort implementation that does a merely-okay job collecting the detections into tracks. With further tuning, I think Deepsort might be made great too. As far as I can tell there is no go-to open-source solution for this, and instead anyone wanting to get this working has to repeat all my cobbling. Worst part is working with the limitations of Google Colab's free-tier GPU (un)availability.

DoctorTristan
Mar 11, 2006

I would look up into your lifeless eyes and wave, like this. Can you and your associates arrange that for me, Mr. Morden?

hbag posted:

i uh
im getting this in python
AttributeError: 'dict' object has no attribute 'descendants'

this makes no sense, i thought dicts did have descendants, that was the whole point
idk heres my code lmao
Python code:
            for post in posts:
                #resp_object[post] = 'wahooo'
                resp_object['cum'].append('wahoo')
neither of these worked


What this guy said, also - not saying this is drawn from experience or anything - but you can never really predict where a piece of code you write is going to end up or who might demand to see it, so maybe dance like nobody’s watching but pick variable names like they’ll be read out in a televised courtroom

Blinkz0rz
May 27, 2001

MY CONTEMPT FOR MY OWN EMPLOYEES IS ONLY MATCHED BY MY LOVE FOR TOM BRADY'S SWEATY MAGA BALLS

hbag posted:

i uh
im getting this in python
AttributeError: 'dict' object has no attribute 'descendants'

this makes no sense, i thought dicts did have descendants, that was the whole point
idk heres my code lmao
Python code:
            for post in posts:
                #resp_object[post] = 'wahooo'
                resp_object['cum'].append('wahoo')
neither of these worked

From a quick Google this seems to be a beautiful soup issue and not one related to python specifically. Maybe read those docs?

hbag
Feb 13, 2021

DoctorTristan posted:

What this guy said, also - not saying this is drawn from experience or anything - but you can never really predict where a piece of code you write is going to end up or who might demand to see it, so maybe dance like nobody’s watching but pick variable names like they’ll be read out in a televised courtroom

trust me im not pushing anything with "cum wahoo" in it to my git repo

anyway heres more of the code lmao

Python code:
def viewthread_actual(thread_id):
    if verifyCall():
        sesh = requests.Session()
        sesh.cookies = http.cookiejar.MozillaCookieJar("cookies.txt")
        forumAuth(sesh)

	# query thread
        if 'page' not in request.args:
            page_num = 1
        else:
            page_num = request.args.get('page')
        threadRaw = sesh.post(f'https://forums.somethingawful.com/showthread.php?threadid={thread_id}&userid=0&perpage=40&pagenumber={page_num}')
        thread = BeautifulSoup(threadRaw.content, 'html.parser')
        
        # grab thread
        if thread.select_one('.userinfo .author') != None:
            resp_object = {'bing bing': 'wahoo'}
            posts = BeautifulSoup.findAll({'class': 'post'})
            for post in posts:
                #resp_object[post].append('wahooo')
                resp_object['cum'].append('wahoo')
        else:
            resp_object = {'error': 'Thread not found.'}
    else:
        resp_object = {'error': 'Unauthorized Access. Ensure your PUBLIC token is somewhere on your profile, and you\'ve included your PRIVATE key in your request.'}
    
    return resp_object
im pretty sure it is a beautifulsoup issue which is why i commented out the line that uses poo poo from beautifulsoup and just tried to set a fixed key pair but that didnt seem to work either

pokeyman
Nov 26, 2006

That elephant ate my entire platoon.

hbag posted:

trust me im not pushing anything with "cum wahoo" in it to my git repo

anyway heres more of the code lmao

Python code:
def viewthread_actual(thread_id):
    if verifyCall():
        sesh = requests.Session()
        sesh.cookies = http.cookiejar.MozillaCookieJar("cookies.txt")
        forumAuth(sesh)

	# query thread
        if 'page' not in request.args:
            page_num = 1
        else:
            page_num = request.args.get('page')
        threadRaw = sesh.post(f'https://forums.somethingawful.com/showthread.php?threadid={thread_id}&userid=0&perpage=40&pagenumber={page_num}')
        thread = BeautifulSoup(threadRaw.content, 'html.parser')
        
        # grab thread
        if thread.select_one('.userinfo .author') != None:
            resp_object = {'bing bing': 'wahoo'}
            posts = BeautifulSoup.findAll({'class': 'post'})
            for post in posts:
                #resp_object[post].append('wahooo')
                resp_object['cum'].append('wahoo')
        else:
            resp_object = {'error': 'Thread not found.'}
    else:
        resp_object = {'error': 'Unauthorized Access. Ensure your PUBLIC token is somewhere on your profile, and you\'ve included your PRIVATE key in your request.'}
    
    return resp_object
im pretty sure it is a beautifulsoup issue which is why i commented out the line that uses poo poo from beautifulsoup and just tried to set a fixed key pair but that didnt seem to work either

It doesn't match the error message you posted before, but I think you need to put a value in resp_object for the key 'cum' before you try to append to it?

After that, I'm not sure exactly what using BeautifulSoup elements as dict keys will do, but my guess is "end poorly". In the commented-out bit, did you mean to do resp_object[‘post’] or something?

hbag
Feb 13, 2021

pokeyman posted:

It doesn't match the error message you posted before, but I think you need to put a value in resp_object for the key 'cum' before you try to append to it?

After that, I'm not sure exactly what using BeautifulSoup elements as dict keys will do, but my guess is "end poorly". In the commented-out bit, did you mean to do resp_object[‘post’] or something?

No, I meant to do resp_object[post], which I know isn't great but it's just while i get it up and running.
As for your first point, I also tried doing resp_object['foo'] = 'bar', but that had the same error.

For context, I'm trying to get a JSON object of variable length, since this function I'm writing reads a thread and has each post in its own key pair in the response object. So, naturally, it'd need to append each new post it finds to the object.

pokeyman
Nov 26, 2006

That elephant ate my entire platoon.

hbag posted:

No, I meant to do resp_object[post], which I know isn't great but it's just while i get it up and running.
As for your first point, I also tried doing resp_object['foo'] = 'bar', but that had the same error.

For context, I'm trying to get a JSON object of variable length, since this function I'm writing reads a thread and has each post in its own key pair in the response object. So, naturally, it'd need to append each new post it finds to the object.

Well I'm baffled on the descendants error. The "each post in the response" part makes total sense, but I would've expected resp_object[post.id] = post or something, i.e. using a string value as the key.

hbag
Feb 13, 2021

pokeyman posted:

Well I'm baffled on the descendants error. The "each post in the response" part makes total sense, but I would've expected resp_object[post.id] = post or something, i.e. using a string value as the key.

yeah it MIGHT be because its trying to name the key after the entire 'post' object or some poo poo, ill try some more poo poo and see if that works

edit: actually wait no that doesnt make sense either because it has the exact same error when i just try it with a hardcoded value

KillHour
Oct 28, 2007


hbag posted:

yeah it MIGHT be because its trying to name the key after the entire 'post' object or some poo poo, ill try some more poo poo and see if that works

edit: actually wait no that doesnt make sense either because it has the exact same error when i just try it with a hardcoded value

Yeah, here's the error you get if you try to assign a key that doesn't have a string representation:



I recommend futzing around with it in interactive mode if possible. It makes it easier to really dig in and see what is breaking where, and then you can easily dump whatever you need out to console.

hbag
Feb 13, 2021

oh gdi i was looking at the wrong thing
it was being thrown by line 34, which is this:

Python code:
posts = BeautifulSoup.findAll({'class': 'post'})

pokeyman
Nov 26, 2006

That elephant ate my entire platoon.
Welp. Well if it helps, I have limitless patience for anyone trying to use the SA Forums's unofficial API.

I complimented someone the other day about how good they'd gotten at picking out the problematic line from a backtrace. And I took a minute to phrase it because I was worried it would sound patronizing, but I was completely genuine. Very useful skill!

hbag
Feb 13, 2021

pokeyman posted:

Welp. Well if it helps, I have limitless patience for anyone trying to use the SA Forums's unofficial API.

I complimented someone the other day about how good they'd gotten at picking out the problematic line from a backtrace. And I took a minute to phrase it because I was worried it would sound patronizing, but I was completely genuine. Very useful skill!

is there another unofficial API or are you talking about the one i wrote (at https://api.fyad.club) that this is for

KillHour
Oct 28, 2007


hbag posted:

oh gdi i was looking at the wrong thing
it was being thrown by line 34, which is this:

Python code:
posts = BeautifulSoup.findAll({'class': 'post'})


Well that's a lot easier to answer. First, findAll has been deprecated and moved to find_all: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#method-names

Next, find_all doesn't expect a dictionary by itself for that function signature: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#find-all

I think what you're trying to do is
code:
posts = BeautifulSoup.find_all(attrs={'class': 'post'})
The 'attrs=' is important there because you need to tell python you're setting the "attrs" property of the function, not the "name" property, which is the default property for a function call with one argument.

Edit: You can also simplify that to this
code:
posts = BeautifulSoup.find_all(class_='post')
Note the underscore on the end of 'class_', because 'class' by itself is a reserved keyword in Python.

KillHour fucked around with this message at 06:25 on Jan 17, 2022

pokeyman
Nov 26, 2006

That elephant ate my entire platoon.

hbag posted:

is there another unofficial API or are you talking about the one i wrote (at https://api.fyad.club) that this is for

I mean the HTML-outputting interface usually accessed with a web browser. (I've helped maintain Awful.app for a few years.)

Nice WordArt!

KillHour
Oct 28, 2007


How dare you get probed before I can find out if your code works now you monster :mad:

hbag
Feb 13, 2021

yeah i figured it out i was trying to use BeautifulSoup (the package) instead of... the actual thread object i created
which is lol

anyway this works
Python code:
posts = thread.findAll({'class': 'post'})

DoctorTristan
Mar 11, 2006

I would look up into your lifeless eyes and wave, like this. Can you and your associates arrange that for me, Mr. Morden?

KillHour posted:

Well that's a lot easier to answer. First, findAll has been deprecated and moved to find_all:

Is this… package maintainers actually deciding to give a poo poo about PEP8? :aaaaa:

leper khan
Dec 28, 2010
Honest to god thinks Half Life 2 is a bad game. But at least he likes Monster Hunter.

DoctorTristan posted:

Is this… package maintainers actually deciding to give a poo poo about PEP8? :aaaaa:

More likely people complaining, so only as a second order concern.

hbag
Feb 13, 2021

camelcase is for functions, underscores are for variables

Doom Mathematic
Sep 2, 2008

hbag posted:

camelcase is for functions, underscores are for variables

Same thing, old boy.

Gin_Rummy
Aug 4, 2007
Can anyone ELI5 exactly what companies/job postings mean when they refer to experience working with cloud infrastructure like AWS and/or GoogleCloud?

Armauk
Jun 23, 2021


Gin_Rummy posted:

Can anyone ELI5 exactly what companies/job postings mean when they refer to experience working with cloud infrastructure like AWS and/or GoogleCloud?

Understanding functions, cloud storage, cloud databases, etc. and feeling comfortable using those products in those specific environments.

ultrafilter
Aug 23, 2007

It's okay if you have any questions.


Posting here cause I can't think of any better place to ask:

My team is using GitLab to host our repositories and do CI on merge requests. We want to split our CI build into different stages so that we can get useful information about what's failing, but GitLab has the possibility of copying the artifacts from any given stage to a different directory or even a different machine to do the next stage. Our artifacts are about 2GB total, so that hurts. Is there a way to get it to run the entire pipeline in a single directory? I know that Jenkins can do that, so it's not a foreign concept, but searching any of the obvious keywords doesn't help.

Adbot
ADBOT LOVES YOU

Xerophyte
Mar 17, 2008

This space intentionally left blank

ultrafilter posted:

Posting here cause I can't think of any better place to ask:

My team is using GitLab to host our repositories and do CI on merge requests. We want to split our CI build into different stages so that we can get useful information about what's failing, but GitLab has the possibility of copying the artifacts from any given stage to a different directory or even a different machine to do the next stage. Our artifacts are about 2GB total, so that hurts. Is there a way to get it to run the entire pipeline in a single directory? I know that Jenkins can do that, so it's not a foreign concept, but searching any of the obvious keywords doesn't help.

Looking at issues 17497, 118599 and 28724 in their tracker: this seems to not currently be possible, but going by the discussion in 28724 it's a candidate for maybe 14.10.

Some people in one of the threads worked around it by making sure that each individual executors had a unique tag and then requiring the same tag for each stage of the pipeline to ensure it all ran on the same machine. I don't know Gitlab's build system but presumably that stops you from actually distributing your builds across multiple machines, so sounds like the very bad type of workaround.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply