Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

QuarkJets: Sep 8, 2008

sugar free jazz posted:

I’m trying to get a package I cloned from github, that doesn’t have a setup.py, imported into a conda env and I have no fuckin idea what to do

Do I just load the dependencies and run the functions and just do it manually or wtf

Can you link to the github project?

Check for a setup.cfg or a pyproject.toml; you should be able to pip install a directory that contains one of these. A setup.cfg is equivalent to a setup.py, it's a bit more modern way of configuring a package (setup.py is gauche because you shouldn't need to write code just to define a package)

Sometimes people organize their repositories in funny ways, so it's worth checking subdirectories for these files as well

# ? Jul 5, 2022 00:11

Adbot: ADBOT LOVES YOU

# ? May 14, 2024 13:36

sugar free jazz: Mar 5, 2008

QuarkJets posted:

Can you link to the github project?

Check for a setup.cfg or a pyproject.toml; you should be able to pip install a directory that contains one of these. A setup.cfg is equivalent to a setup.py, it's a bit more modern way of configuring a package (setup.py is gauche because you shouldn't need to write code just to define a package)

Sometimes people organize their repositories in funny ways, so it's worth checking subdirectories for these files as well

Heres the project

https://github.com/ryanjgallagher/core_periphery_sbm

Afaict it�s just four .py files, a read me with nothing about installation and a license

# ? Jul 5, 2022 00:15

bigperm: Jul 10, 2001; some obscure reference

I have no opinion on nested list comprehensions being good or bad or neutral but I was curious to see if any of the top python projects on github use them and if so what do they look like.

Methodology:
I cloned some repos and grepped for:

code:

\[\[\w*\sfor

Results:

Django uses exactly one:
https://github.com/django/django/blob/main/django/db/models/sql/compiler.py

Python code:

# django/django/db/models/sql/compiler.py:
param_rows = [[p for ps in row for p in ps] for row in param_rows]

Rich - none.
Flask - none.
Requests - none.

Keras uses two:
https://github.com/keras-team/keras/blob/master/keras/datasets/imdb.py
https://github.com/keras-team/keras/blob/master/keras/datasets/reuters.py

Python code:

# keras/keras/datasets/imdb.py:
xs = [[w for w in x if skip_top <= w < num_words] for x in xs]
# keras/keras/datasets/reuters.py:
xs = [[w for w in x if skip_top <= w < num_words] for x in xs]

Pandas gave two hits but I don't know if either of them count.
https://github.com/pandas-dev/pandas/blob/main/pandas/tests/frame/methods/test_filter.py
https://github.com/pandas-dev/pandas/blob/main/pandas/tests/groupby/test_quantile.py

Python code:

# pandas/pandas/tests/frame/methods/test_filter.py:
exp = df[[x for x in df.columns if "BB" in x]]
# pandas/pandas/tests/groupby/test_quantile.py:
idx_codes = [[x for x in range(min(nrow, 4)) for _ in q]] * len(groupby) + [
        list(range(len(q))) * min(nrow, 4)

Numpy has three all in the same file.
https://github.com/numpy/numpy/blob/main/numpy/f2py/tests/test_character.py

Python code:

# numpy/numpy/f2py/tests/test_character.py:
expected = np.array([[c for c in s] for s in a], dtype='u1')
# numpy/numpy/f2py/tests/test_character.py:
a = np.array([[c for c in s] for s in expected], dtype='u1')
# numpy/numpy/f2py/tests/test_character.py:
expected = np.array([[[c for c in item] for item in row] for row in a],

It doesn't look like they are used very much in popular python projects on github.

bigperm fucked around with this message at 02:53 on Jul 5, 2022

# ? Jul 5, 2022 01:08

QuarkJets: Sep 8, 2008

sugar free jazz posted:

Heres the project

https://github.com/ryanjgallagher/core_periphery_sbm

Afaict it�s just four .py files, a read me with nothing about installation and a license

Looks like they didn't bother creating an installable package, so you have choices here:

1. Modify your PYTHONPATH environment variable or sys.path to point to whatever directory core_periphery_sbm is in (easy mode but not robust at all; this is fine if you're just loving around in a notebook or something)

2. Do the little bit of additional work to make this an installable package (basically it just needs a very minimal setup.cfg, at which point you should be able to run "pip install" against it; you could create a branch, add this file in, try it out, and then ping the authors with a pull request)

3. Make this project a git submodule of your own project, and do whatever needs to be done to make it install properly as part of your project

# ? Jul 5, 2022 01:09

sugar free jazz: Mar 5, 2008

QuarkJets posted:

Looks like they didn't bother creating an installable package, so you have choices here:

1. Modify your PYTHONPATH environment variable or sys.path to point to whatever directory core_periphery_sbm is in (easy mode but not robust at all; this is fine if you're just loving around in a notebook or something)

2. Do the little bit of additional work to make this an installable package (basically it just needs a very minimal setup.cfg, at which point you should be able to run "pip install" against it; you could create a branch, add this file in, try it out, and then ping the authors with a pull request)

3. Make this project a git submodule of your own project, and do whatever needs to be done to make it install properly as part of your project

I super appreciate it! I am in fact just fuckin around at this point and option 1 worked perfectly.

# ? Jul 5, 2022 03:45

Gangsta Lean: Dec 3, 2001; Calm, relaxed...what could be more fulfilling?

bigperm posted:

I have no opinion on nested list comprehensions being good or bad or neutral but I was curious to see if any of the top python projects on github use them and if so what do they look like.

Methodology:
I cloned some repos and grepped for:
code:
\[\[\w*\sfor

You�re probably missing out on a ton of stuff by assuming that there�s an interior list. This form is probably way more common.

code:

[x + y for y in range(10, 50, 10) for x in range(5)]

# ? Jul 6, 2022 03:10

tays revenge: Aug 29, 2009

code:

x = a.__add__(b)

# ? Jul 6, 2022 03:58

Armitag3: Mar 15, 2020; Forget it Jake, it's cybertown.

Gangsta Lean posted:

You�re probably missing out on a ton of stuff by assuming that there�s an interior list. This form is probably way more common.
code:
[x + y for y in range(10, 50, 10) for x in range(5)]

i cannot wrap my head around this form

e: well this is understandable actually. last time i came across this double for was for zipping two things or whatever and didn�t get it at the time

# ? Jul 6, 2022 12:37

bigperm: Jul 10, 2001; some obscure reference

Gangsta Lean posted:

You�re probably missing out on a ton of stuff by assuming that there�s an interior list. This form is probably way more common.
code:
[x + y for y in range(10, 50, 10) for x in range(5)]

Is that really a nested list comprehension though? The python docs say

https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions posted:

A list comprehension consists of brackets containing an expression followed by a for clause, then zero or more for or if clauses.

And only refer to nested list comprehensions when the initial expression is another list comprehension.

quote:

5.1.4. Nested List Comprehensions
The initial expression in a list comprehension can be any arbitrary expression, including another list comprehension.

# ? Jul 6, 2022 15:58

LochNessMonster: Feb 3, 2005; I need about three fitty

I'm trying to build some pagination and I'm having some brain farts on how to do this in a pythonic way.

API 1 has a limit of 100 items per call, I want to use the results from this one to query API 2 which only allows calls with 10 items. API 1 returns a list from which I'd like to take 10 items at a time as input for the 2nd query. The "best" thing I could come up with is something like this.

Python code:

def query_api2(api1_results: list) -> dict:
	api2_final_results = {}
	while len(api1_results) > 0:
		temp_list = api1_results[:10]
		api2_results = call_to_api2(temp_list)
		del(results[:10]
		temp_list = []
		api2_final_results.update(api2_results)
	return api2_final_results

# ? Jul 6, 2022 16:01

boofhead: Feb 18, 2021

LochNessMonster posted:

I'm trying to build some pagination and I'm having some brain farts on how to do this in a pythonic way.

For this kinda stuff I just chunk it like this:

Python code:

list_to_chunk = list(range(1,101))
chunk_size = 10
chunked_list = [list_to_chunk[i:i+chunk_size] for i in range(0, len(list_to_chunk), chunk_size)]
print(chunked_list)
# [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 
# ...  [91, 92, 93, 94, 95, 96, 97, 98, 99, 100]]

Then you just loop through chunked_list

# ? Jul 6, 2022 16:14

ExcessBLarg!: Sep 1, 2001

bigperm posted:

Is that really a nested list comprehension though?

Technically no, but it's a related feature that has the same benefits and issues as actual nested list comprehensions, but with a different resulting structure. I don't know if it has a real name, but I'd call it a "chained comprehension" since it has the same behavior as itertools.chain.

FWIW, the Ruby equivalent of a nested list comprehension is nested maps, while the equivalent of chained comprehensions is nested flat_maps. So it's definitely a concept that extends across languages.

# ? Jul 6, 2022 17:48

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

That style is what I always considered nested list comprehension and also found very difficult to parse. (the one without a set of internal brackets)

# ? Jul 6, 2022 18:24

LochNessMonster: Feb 3, 2005; I need about three fitty

boofhead posted:

For this kinda stuff I just chunk it like this:

Python code:

list_to_chunk = list(range(1,101))
chunk_size = 10
chunked_list = [list_to_chunk[i:i+chunk_size] for i in range(0, len(list_to_chunk), chunk_size)]
print(chunked_list)
# [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 
# ...  [91, 92, 93, 94, 95, 96, 97, 98, 99, 100]]

Then you just loop through chunked_list

That looks like a way nicer solution than what I was trying to do. Unfortunately it only seems to work for the first list.

My result looks like this

code:

[[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [], [], [], [], [], [], [], [], []

edit:

writing is differently to make my brain parse what's going on properly helped me identify the issue

Python code:

chunked_list = []
for i in range(0, len(list_to_chunk), chunk_size):
	chunked_list.append(list_to_chunk[i:i+chunk_size])
	chunk_size += chunk_size

adding that last line fixed the issue, which is elegant enough for me, but I'd like to understand if there's a way to put it into the oneliner you showed as an example.

edit2: the above didn't actually fix anything, and I realize I'm now getting increasingly large batch sizes as I'm increasing the chunk_size. "i" does get incremented by 10 for each iteration so it should've added it. Switched the chunked_list = part to chunked_list.append(), but that didn't work either

edit 3: I'm a moron and wrote i:+chunk_size instead of i:i+chunk_size. Solution works perfectly fine...

LochNessMonster fucked around with this message at 19:52 on Jul 6, 2022

# ? Jul 6, 2022 19:09

ExcessBLarg!: Sep 1, 2001

LochNessMonster posted:

edit 3: I'm a moron and wrote i:+chunk_size instead of i:i+chunk_size. Solution works perfectly fine...

I'm really surprised that Python doesn't have a slice_at_offset_with_length function somewhere in the stdlib, since having such would avoid this mistake, or having to use the walrus operator when your starting offset has to be computed but (max) list length is known.

# ? Jul 6, 2022 20:52

LochNessMonster: Feb 3, 2005; I need about three fitty

ExcessBLarg! posted:

I'm really surprised that Python doesn't have a slice_at_offset_with_length function somewhere in the stdlib, since having such would avoid this mistake, or having to use the walrus operator when your starting offset has to be computed but (max) list length is known.

I�ve been googling for some time to prevent asking for something that already exists but came to the conclusion that I didn�t know what the thing I was looking for is called or it didn�t exist. Seemed like a pretty straightforward use case too.

# ? Jul 7, 2022 09:07

Hughmoris: Apr 21, 2007; Let's go to the abyss!

A question about a domain I have zero experience in:

Goal: I want to know whenever the UPS or FedEx truck pass by my townhome.

I have a clear view of the street (minimal traffic) which is about 20 ft from my front door. Can I set point a webcam out the window and use object detection + python script to ping me when the Fedex or UPS truck drive by? Is that a huge can of worms or pretty approachable given today's libraries?

# ? Jul 9, 2022 20:26

QuarkJets: Sep 8, 2008

Hughmoris posted:

A question about a domain I have zero experience in:

Goal: I want to know whenever the UPS or FedEx truck pass by my townhome.

I have a clear view of the street (minimal traffic) which is about 20 ft from my front door. Can I set point a webcam out the window and use object detection + python script to ping me when the Fedex or UPS truck drive by? Is that a huge can of worms or pretty approachable given today's libraries?

It's highly approachable for someone willing to dig in and play around with stuff - numpy, scipy, PIL, and opencv provide a ton of tools that could be useful, and there are many approaches you could try. As a low-effort attempt I might be tempted to try a correlation-based detector. Basically take some images of the FedEx or UPS truck in front of your house. From then on, take any arbitrary image from your camera and try to correlate it with your example images. If the correlation score is high enough, then that kind of truck is probably there. You can use time-boxing to discriminate between A) the truck driving on your street vs B) the truck parked in front of your house. You can use color analysis to determine the color of the truck.

There are also neural networks that you could dip into using. I'd consider that overkill for this problem

# ? Jul 9, 2022 21:20

boofhead: Feb 18, 2021

The raspberry pi thread could also be a good resource, I think people have done similar things with that. Might not necessarily be in python but it could be helpful as a reference

# ? Jul 9, 2022 21:33

QuarkJets: Sep 8, 2008

To put this in perspective, this kind of problem would be considered challenging but achievable for high school robotics clubs. At the national FIRST tournaments a lot of teams show up with CV implementations, both for autonomous segments and for driver assistance (teams are allowed to implement aim-botting). Getting a good-enough solution here is something that just about anyone willing to touch computers should be able to do, but it'll take time as you learn new things. You can ask questions here any time.

# ? Jul 9, 2022 21:51

Hughmoris: Apr 21, 2007; Let's go to the abyss!

Sweet. I'll start poking around and see what I can come up with. Thanks!

# ? Jul 9, 2022 22:19

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

IMO it would be much easier to do the actually UPS or not UPS classification with a neural net, particularly FastAI.

You might use some existing OpenCV code to actually capture the image of the cars but then then the final classification you�ll likely have much easier and better results by passing all the �car� labeled images to a neural net.

# ? Jul 9, 2022 22:26

QuarkJets: Sep 8, 2008

CarForumPoster posted:

IMO it would be much easier to do the actually UPS or not UPS classification with a neural net, particularly FastAI.

You might use some existing OpenCV code to actually capture the image of the cars but then then the final classification you�ll likely have much easier and better results by passing all the �car� labeled images to a neural net.

OP this illustrates a common AI workflow; OpenCV is pretty efficient, and it's common to use various OpenCV algorithms for preprocessing prior to training or classification by a neural network. I'd view an OpenCV implementation as laying the groundwork for a more accurate CNN implementation

# ? Jul 9, 2022 22:48

KICK BAMA KICK: Mar 2, 2009

Small wrinkle: I think both those (and other) package services use additional unmarked probably rental trucks at high-volume holiday times. Likely nbd for your use case but just an example of the kinds of things you'll need to think about as your code touches the real world.

# ? Jul 10, 2022 00:13

QuarkJets: Sep 8, 2008

Yeah that's probably true, but the OpenCV implementation can be robust to that; basically any delivery truck shape parking in front of your house is going to correlate well. A CNN can be made more robust by training against a lot more truck shapes but that's more work

# ? Jul 10, 2022 01:44

Rescue Toaster: Mar 13, 2003

Is there a python library for providing a command prompt to the terminal where I can easily configure options and help screens and the like? Similar to argparse but working on stdin in real time in a loop, basically. Features from a modern shell like tab completion and the like would be great. This could be used either when running from a shell, or possibly attached to a tty instead of a shell, that sort of thing.

Even if there's not something as full featured as argparse for that, maybe even a recommended low level lexer/parser library that's easy to use would do I guess.

It's very hard to search for 'python command prompt' 'python terminal' 'python shell' and the like so maybe this is a super simple thing and I just can't find it.

# ? Jul 11, 2022 20:26

necrotic: Aug 2, 2005; I owe my brother big time for this!

Check out click

https://click.palletsprojects.com/en/8.1.x/

It�s my go to for python CLI development

edit: it doesn�t do the REPL style thing it sounds like you want, though?

Something like with can make any command a prompt style thing: https://github.com/mchav/with

necrotic fucked around with this message at 20:39 on Jul 11, 2022

# ? Jul 11, 2022 20:36

SurgicalOntologist: Jun 17, 2004

For CLIs I too use click for anything moderately complicated and docopt for smaller scripts.

If you're looking for something interactive, perhaps prompt_toolkit is what you're looking for. For something more flexible (i.e. not necessarily prompt based) urwid is super powerful.

# ? Jul 11, 2022 20:44

Rescue Toaster: Mar 13, 2003

Yeah I'm talking interactive like the REPL, almost like implementing a shell yourself (like bash) or something.

I think the python-prompt-toolkit looks closest to what I'm thinking, but it lacks the then built-in argument parsing you get from something like argparse or click. Possibly one of those could be used to parse out the input once prompt-toolkit has obtained a string.

The holy grail would be an interactive argparse/click that could do tab completion for individual parameters, basically. But using prompt-toolkit for the menus & transitions, any extra user input I want to prompt for, and then when I do want to accept a command string w/ parameters, feed it into one of the argument parsing libraries, I could probably make that all work well enough.

vvv Awesome thanks very much!

Rescue Toaster fucked around with this message at 21:36 on Jul 11, 2022

# ? Jul 11, 2022 21:02

SurgicalOntologist: Jun 17, 2004

Looks like it exists: https://github.com/click-contrib/click-repl

Edit: continuing to browse this list, looks like that's not your only choice.

SurgicalOntologist fucked around with this message at 21:34 on Jul 11, 2022

# ? Jul 11, 2022 21:31

Hed: Mar 31, 2004; Fun Shoe

Ran into this one when I was running through test cases too quickly, and immediately assumed the function where I was building out the Decimal was wrong.

It turns out, calling Decimal() on the string was the right way, and in my tests I wrote it the way at the bottom! Always read your Expected vs. Actual to determine where the problem is

Python code:

>>> Decimal('920')
Decimal('920')
>>> Decimal('920') / Decimal(100)
Decimal('9.2')   # wtf it's right in the interpreter?
>>> Decimal(9.20)
Decimal('9.199999999999999289457264239899814128875732421875')     # oh god drat it

# ? Jul 12, 2022 23:47

Falcon2001: Oct 10, 2004; Eat your hamburgers, Apollo.; Pillbug

Hed posted:

Ran into this one when I was running through test cases too quickly, and immediately assumed the function where I was building out the Decimal was wrong.

It turns out, calling Decimal() on the string was the right way, and in my tests I wrote it the way at the bottom! Always read your Expected vs. Actual to determine where the problem is
Python code:
>>> Decimal('920')
Decimal('920')
>>> Decimal('920') / Decimal(100)
Decimal('9.2')   # wtf it's right in the interpreter?
>>> Decimal(9.20)
Decimal('9.199999999999999289457264239899814128875732421875')     # oh god drat it

I look forward to fourteen pages about floating point number formatting weirdness.

# ? Jul 14, 2022 20:28

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Falcon2001 posted:

I look forward to fourteen pages about floating point number formatting weirdness.

slap a ceil, 9.2, 9.199999999999 theyre all about 10

# ? Jul 14, 2022 20:57

DoctorTristan: Mar 11, 2006; I would look up into your lifeless eyes and wave, like this. Can you and your associates arrange that for me, Mr. Morden?

Falcon2001 posted:

I look forward to 14.000000000001 pages about floating point number formatting weirdness.

# ? Jul 14, 2022 23:28

pmchem: Jan 22, 2010

IEEE-754 more like IEEE-7.54 am i right

# ? Jul 15, 2022 01:58

OnceIWasAnOstrich: Jul 22, 2006

SurgicalOntologist posted:

Looks like it exists: https://github.com/click-contrib/click-repl

I just want to say this thing is wonderful. I already had a moderately-complicated Python app using Click that has a very slow startup time on most commands because it needs to read and set a bunch of cloud metadata. An import and one line of code and this gave it a fully-featured REPL with tab completion and everything that saves substantial amounts of time when running multiple commands sequentially.

# ? Jul 15, 2022 03:40

Seventh Arrow: Jan 26, 2005

So, did a bit of experimentation today. There was a project in Codecademy about classes and they wanted a printout of the string representation method and the official solution used a string concatenation:

code:

def __repr__(self):
  return self.name + ' menu available from ' + str(self.start_time) + " - " + str(self.end_time)

I had just learned about f-strings and was wondering if I could use that instead of an ugly concat. Well according to this site, you could!

code:

 def __repr__(self):
    return f"{self.name} menu available from {self.start_time} - {self.end_time}"

Lo and behold, the same printout with half the calories! Or something. Would there ever be a reason to use .format() or a string concat instead of f-strings?

Also, I feel like Codecademy kind of rushed this complex subject (classes, not f-strings) and I feel like I've been smacked with a frying pan. Can someone point me to a decent tutorial on classes?

# ? Jul 15, 2022 04:07

Jigsaw: Aug 14, 2008

Seventh Arrow posted:

So, did a bit of experimentation today. There was a project in Codecademy about classes and they wanted a printout of the string representation method and the official solution used a string concatenation:
code:
def __repr__(self):
  return self.name + ' menu available from ' + str(self.start_time) + " - " + str(self.end_time)
I had just learned about f-strings and was wondering if I could use that instead of an ugly concat. Well according to this site, you could!
code:
 def __repr__(self):
    return f"{self.name} menu available from {self.start_time} - {self.end_time}"
Lo and behold, the same printout with half the calories! Or something. Would there ever be a reason to use .format() or a string concat instead of f-strings?

Also, I feel like Codecademy kind of rushed this complex subject (classes, not f-strings) and I feel like I've been smacked with a frying pan. Can someone point me to a decent tutorial on classes?

I can think of two reasons to use .format() or a string concat over an f-string. One is if you need a backslash somewhere inside the formatted expression, since f-strings don�t support that but .format() does (and with a string concat you can just break things up as needed). Another is for backward compatibility, since f-strings weren�t introduced until version 3.6. There might be other reasons, but those are the things that immediately come to mind.

# ? Jul 15, 2022 04:45

QuarkJets: Sep 8, 2008

Seventh Arrow posted:

So, did a bit of experimentation today. There was a project in Codecademy about classes and they wanted a printout of the string representation method and the official solution used a string concatenation:
code:
def __repr__(self):
  return self.name + ' menu available from ' + str(self.start_time) + " - " + str(self.end_time)
I had just learned about f-strings and was wondering if I could use that instead of an ugly concat. Well according to this site, you could!
code:
 def __repr__(self):
    return f"{self.name} menu available from {self.start_time} - {self.end_time}"
Lo and behold, the same printout with half the calories! Or something. Would there ever be a reason to use .format() or a string concat instead of f-strings?

Also, I feel like Codecademy kind of rushed this complex subject (classes, not f-strings) and I feel like I've been smacked with a frying pan. Can someone point me to a decent tutorial on classes?

I think it's best to avoid string concat. I sometimes see it used as a nice lazy shortcut for building a string based on conditionals:

Python code:

some_string = 'all '
some_string += "the marbles " if some_condition else "your base are "
some_string += "belong to us"

That block of code sucks for a bunch of reasons.

I think it's cleaner if you use f-strings. Temporary variables can improve readability

Python code:

some_string = f"all {'the marbles' if some_condition else 'your base are'} belong to us"
# or I think this is more readable:
middle = "the marbles" if some_condition else "your base are"
some_string = f"all {middle} belong to us"

Or list-joining is fine too, and still a lot better than string concatenation. Although it's technically a little more code, it's good for building very long strings and way more performant than concatenation

Python code:

string_parts = ['all']
string_parts.append("the marbles" if some_condition else "your base are")
string_parts.append("belong to us")
some_string = ' '.join(string_parts)

QuarkJets fucked around with this message at 04:59 on Jul 15, 2022

# ? Jul 15, 2022 04:54

Adbot: ADBOT LOVES YOU

# ? May 14, 2024 13:36

QuarkJets: Sep 8, 2008

I don't know of any decent class tutorials because every single one that I've ever seen deep throats the OOP cancer :shrug:

# ? Jul 15, 2022 04:58

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »