Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

duck monster: Dec 15, 2004

I am currently overbrimming with the desire to beat Homebrew's python maintainers to death (with angry words).

YET AGAIN, a random non python install just randomly upgrades python breaking the gently caress out of every goddamn virtualenv on my machine. People have been banging on about this being unnaceptable on variour homebrew related git issues for years, and ..... nothing gets done. The irritating thing is python has *multiple* methods of letting multiple pythons co-exist. Theres no need to uninstall the old versions.... Gah...

At least a warning that "Hey this package will destroy your dev environment, just so you know" and an option to pull the eject lever.....

< / rant >

# ? Nov 17, 2020 04:48

Adbot: ADBOT LOVES YOU

# ? May 22, 2024 17:52

Dominoes: Sep 20, 2007

Fork.

# ? Nov 17, 2020 05:48

NinpoEspiritoSanto: Oct 22, 2013

Use pyenv on OSX imo

# ? Nov 17, 2020 09:48

OnceIWasAnOstrich: Jul 22, 2006

duck monster posted:

I am currently overbrimming with the desire to beat Homebrew's python maintainers to death (with angry words).

YET AGAIN, a random non python install just randomly upgrades python breaking the gently caress out of every goddamn virtualenv on my machine. People have been banging on about this being unnaceptable on variour homebrew related git issues for years, and ..... nothing gets done. The irritating thing is python has *multiple* methods of letting multiple pythons co-exist. Theres no need to uninstall the old versions.... Gah...

At least a warning that "Hey this package will destroy your dev environment, just so you know" and an option to pull the eject lever.....

< / rant >

As much as I used it for years and got my whole PhD doing work on a Mac Pro primarily with Homebrew...it is a trash package manager. You can pin a formula but the Homebrew rule that you can't build against outdated formulas really works against Python here. I don't think there is a good way to maintain Python on Homebrew as it exists when by rule you can't NOT upgrade dependencies. I believe if you pin a particular formula it will at least ask for confirmation before upgrading it. I believe you could also alias --confirm onto your brew commands to get it to behave more like a sane package manager by default.

Use anything else to manage your Python environment on macOS, whether it is pyenv or conda or whatever else.

# ? Nov 17, 2020 15:38

mr_package: Jun 13, 2000

I have kind of a dumb question about Poetry. The dir structure by default seems fine but I often find I need to put the actual app.py script in root level, e.g. one level above the main package that Poetry creates, because it's normal to write an app functionality as a package and then import it into the main script users actually run. For example I wrote a package that reads/writes JSON files, does operations based on the data, etc. Then I wrote a cli using Click and it is one dir above so it can import modules, classes, etc. from the main package. This cli doesn't have much logic, just an interface basically to the package.

Anyway in the default dir structure that means your main.py or app.py is in the root dir, parallel to your module dir and 'tests' dirs. Does this seem weird? Should I be using the --src option and putting my root app there instead? Am I doing this wrong? Should my main app.py go in the same dir as the package? My intent is that anyone can use the functionality with whatever they want (e.g. run it in their own code, write their own frontend, whatever). But I might be actually unusual use case where I'm writing a lot of internal tools that need to be open in this way in case anyone wants to incorporate into CI pipeline for example. They can call the script or just import the package. But maybe having the main app / cli / whatever in the package would work fine too?

I remember running into a ton of import issues and one point which may also have been related. But I try to use empty __init__ files and explicit imports so that I don't have to maintain the init files (I often use sub modules for my io for example).

# ? Nov 19, 2020 23:06

NinpoEspiritoSanto: Oct 22, 2013

I use a src tree but just make a package directory and put your stuff in there. So alongside pyproject.toml you have app/__init__.py app/db/models.py app/cli.py etc.

Always use absolute imports imo so import app.db.models or from app.db import models etc.

# ? Nov 20, 2020 00:17

duck monster: Dec 15, 2004

OnceIWasAnOstrich posted:

As much as I used it for years and got my whole PhD doing work on a Mac Pro primarily with Homebrew...it is a trash package manager. You can pin a formula but the Homebrew rule that you can't build against outdated formulas really works against Python here. I don't think there is a good way to maintain Python on Homebrew as it exists when by rule you can't NOT upgrade dependencies. I believe if you pin a particular formula it will at least ask for confirmation before upgrading it. I believe you could also alias --confirm onto your brew commands to get it to behave more like a sane package manager by default.

Use anything else to manage your Python environment on macOS, whether it is pyenv or conda or whatever else.

(which still kinda vomits on itself, but at least the environment rebuild is a bit more descriptive.

I generally use Pipenv . But the work dev SOE is vanilla virtualenv.

The problem in this case was that it actually somehow managed to break loving PyCharm which seemed to be packaged with a version of setuptools thats incompatible with 3.9. I ended up just going and manually rebuilding it.

Unfortunately the Homebrew python maintainers are *really* uninterested in the problem , and its giving me flashback to the bad old days when Debian ruby maintainers would purposefully disable gem to encourage "correct package installation" (which apparently is "install everything globally via apt get and tough poo poo if your favorite gem isnt there, just grab a tar ball"

Though my suggestion that they put a warning up that python developers should uninstall homebrew was perhaps a liitle on the trollish side, ill confess

# ? Nov 21, 2020 12:51

Ghost of Reagan Past: Oct 7, 2003; rock and roll fun

The homebrew maintainers also made it impossible to install older packages from a git commit, which you might want for development purposes, because in the Real World we have to use non-bleeding-edge software for compatibility reasons. Basically making it useless to developers. This is recent and they've been absolutely unresponsive and unsympathetic to the mess that this causes. It's not the "intended workflow", which they just changed without warning because they clearly don't care about how people actually use brew. It's always been garbage and now it's somehow worse than it's ever been.

# ? Nov 22, 2020 03:52

Jeesis: Mar 4, 2010; I am the second illegitimate son of gawd who resides in hoaven.

Trying to unfuck some code, for background I am going from python 2.5 to python 3.8

code:

PATTERN = re.compile(r'some regex')

string = PATTERN.sub(process_match, string)

def process_match(match):
    groups = match.groupdict()
    if groups['butts']:
        return process_butts(match)`

I am trying to move the 'process_butts' and other functions into a class so everything will be delegated to their specific functions, as process_match only checks the resulting match and delegates work to other functions.

So I guess my question is, is the sub function calling the process_match function with the given 'string' variable as match or there some other black magic fuckery going on? If I only want the possible group match should I just do;

code:

match = PATTERN.match(string)
groups = match.groupdict()
if groups['butts']:
    replace_var = process_butts(string)

string = string.replace(match, replace_var)

Or something similar?

The caveat of working on this, is that this is for a template tag in django with nothing actually calling it right now so I have no fuckin way to test it without doing a fuckton more work and im just trying to work it out in my head but this code is so fucky its making my brain crap out.

# ? Nov 23, 2020 18:20

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

Jeesis posted:

Trying to unfuck some code, for background I am going from python 2.5 to python 3.8

Well, the original appears to be passing the match object which is a lot more than just the string so I guess it depends what

pre:

process_butts

is doing.

I also don't know what string.replace is, if it was a method in python 2 it's gone in python 3. You could use re.sub or I think replace on the match object.

# ? Nov 23, 2020 19:00

Dr Subterfuge: Aug 31, 2005; TIME TO ROC N' ROLL

quote:

str.replace(old, new[, count])
Return a copy of the string with all occurrences of substring old replaced by new. If the optional argument count is given, only the first count occurrences are replaced.

Definitely a thing in 3.x.

# ? Nov 23, 2020 20:43

Jeesis: Mar 4, 2010; I am the second illegitimate son of gawd who resides in hoaven.

Phobeste posted:

Well, the original appears to be passing the match object which is a lot more than just the string so I guess it depends what
pre:
process_butts
is doing.

I also don't know what string.replace is, if it was a method in python 2 it's gone in python 3. You could use re.sub or I think replace on the match object.

Yeh I am not really sure what is even going on here but I think I have it somewhat working, at least it will import. Going to consult the person who wrote this as they still work here but my gently caress is some fucky nested bullshit fuckery. There is a 30 line comment trying to explain just how the regex blob that unfurled to 79 char lines is 7 lines long works :shepface:

# ? Nov 23, 2020 21:06

Jakabite: Jul 31, 2010

I'm currently making a game with pygame, and it's going okay. I'm enjoying the experience anyway, having not programmed at all since university and never in Python. It's 2D space combat game, made to mimic the sort of combat you see in The Expanse - namely, realistic physics, vast distances, and the ship doing a lot of the heavy lifting with the player mostly just giving the ship commands. One mechanic I'm working on right now is getting the ship to move through a coordinate. So far I'm only focused on getting it working from standstill so the ship's current velocity doesn't need taking into account, and so far I've got the ship finding a target 'rotation', spinning the ship to that rotation, then engaging thrust. Mostly working okayishly except with shorter distances. What I'm wondering is, since the game will have a lot of systems like this and largely be based around the 'ship computer' calculating correct angles, thrusts, velocities etc., would I potentially be better off working with PID instead, and simulating this in the game? If so, what's the best way of doing that? I'm assuming there are modules for PID control but honestly I'm not sure if it'll be necessary or not? I'm leaning towards it but it being better but obviously it does increase the complexity of things somewhat.

That said I did work with PID a fair bit during my undergrad and wouldn't mind refreshing those skills.

# ? Nov 23, 2020 21:07

salisbury shake: Dec 27, 2011

Loezi posted:

I'm reading up on coroutines but getting tripped over by what's the current preferred method of doing stuff.

Various older tutorials are full of examples like
Python code:
def bare_bones():
    print("My first Coroutine!")
    while True:
        value = (yield)
        print(value)

coroutine = bare_bones()
next(coroutine)
coroutine.send("First Value")
But more recent PEPs are full of await and __aiter__and async def and all this stuff, and the asyncio module says "Support for generator-based coroutines is deprecated and is scheduled for removal in Python 3.10." which I'm not clear on whether it applies to just that module or the whole concept and it's a whole big mess.

Does anyone have a good, up to date tutorial on this stuff?

I interpreted that deprecation warning as meaning that use of generator-based coroutines for async/await will be deprecated.

This test still works on the 3.10 alphas:

Python code:

Python 3.10.0a2 (default, Nov  3 2020, 18:29:35) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> def test():
...   x = yield
...   print(x)
... 
>>> t = test()
>>> next(t)
>>> t.send(1)
1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

duck monster posted:

I am currently overbrimming with the desire to beat Homebrew's python maintainers to death (with angry words).

snip

This is because Homebrew's popularity is the result of trends and not because it is a superior package manager. Try out Macports or Fink.

# ? Nov 27, 2020 19:29

thotsky: Jun 7, 2005; hot to trot

Maybe this is more of an AWS centric thing than a python issue, but I'm having some trouble when working locally with Lambda Layers in my SAM/Python Lambda project.
I put my layer code in "/layers/python/example_layer.py" while my lambda lives in "/lambdas/example_lambda/example_lambda.py" and I import the layer using "import example_layer" in my lambda.
SAM managed to deploy both just fine, and it works as expected in AWS, but on my local machine pylint does not like the import statement (unable to import example_layer).

I figure that SAM packages layers correctly as long as they're in a folder named "python", but maybe python needs the layer code to be below the lambda code in my folder structure for it to be found when working locally. I can't really move it though, because I have more than one lambda, all in their own separate folders, and duplicating it would sort of defeat the point of a shared layer. Suggestions?

# ? Dec 3, 2020 16:03

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

I have the following code I whipped up just out of curiosity:

code:

import time

x = 2

while True:
    a = time.perf_counter_ns()
    x = x ** 2
    b = time.perf_counter_ns()
    y = (b - a)
    print(f'Digit size: {len(str(x))} - NS to calc: {y}')

I get output like this:

code:

Digit size: 1 - NS to calc: 1369
Digit size: 2 - NS to calc: 730
Digit size: 3 - NS to calc: 530
Digit size: 5 - NS to calc: 610
Digit size: 10 - NS to calc: 740
Digit size: 20 - NS to calc: 920
Digit size: 39 - NS to calc: 920
Digit size: 78 - NS to calc: 810
Digit size: 155 - NS to calc: 940
Digit size: 309 - NS to calc: 1360
Digit size: 617 - NS to calc: 2140
Digit size: 1234 - NS to calc: 4709
Digit size: 2467 - NS to calc: 13960
Digit size: 4933 - NS to calc: 30299
Digit size: 9865 - NS to calc: 57508
Digit size: 19729 - NS to calc: 57599
Digit size: 39457 - NS to calc: 122307
Digit size: 78914 - NS to calc: 258633
Digit size: 157827 - NS to calc: 541475
Digit size: 315653 - NS to calc: 1131908
Digit size: 631306 - NS to calc: 2390754
...

2390754 nanoseconds is like 0.002 seconds and it's definitely taking my computer more like 2-3 seconds to bash together two 310,000 digit numbers. Is having to do this big calculation loving with Python's timekeeping? Or am I not understanding how to use some of the tools in time correctly?

# ? Dec 9, 2020 18:50

mr_package: Jun 13, 2000

edit: it's the len(str(x))

mr_package fucked around with this message at 19:52 on Dec 9, 2020

# ? Dec 9, 2020 19:49

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

ooooh, I didn't even think that would be a serious issue

# ? Dec 9, 2020 20:03

OnceIWasAnOstrich: Jul 22, 2006

Epsilon Plus posted:

ooooh, I didn't even think that would be a serious issue

Yep, creating and working with massive Python strings is definitely going to add a lot of overhead compared to simple integer or even bignum math. Depending on what you are curious about, using log10 gets you the same info without that particular performance hit although with exponentially-increasing bignums you'll hit some performance problems soon enough anyway.

# ? Dec 9, 2020 20:25

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

I just had a student a while back ask me about the maximum integer size in Python, I told her "there's not really one" and she was interested in the implications of that - I literally whipped it together in like ten seconds and threw in len(str(x)) just to indicate how big the numbers would get. Using math.log10 is way better, I'm just a math idiot and didn't realize that was a possibility.

# ? Dec 9, 2020 20:31

NinpoEspiritoSanto: Oct 22, 2013

Yeah python is good at a lot of stuff but Ive run into it being bizarrely slow with a lot of string ops.

# ? Dec 10, 2020 12:13

necrotic: Aug 2, 2005; I owe my brother big time for this!

That's converting an int to a ~600kb string. ~~2ms seems reasonable for that in python.~~ Err, I guess 2-3 seconds.

# ? Dec 10, 2020 21:59

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

yeah now that I realize the nature of the issue I'm sure that taking a very, very, very large number and converting it into its string representation is going to be at least a little time-consuming

# ? Dec 10, 2020 23:00

Zugzwang: Jan 2, 2005; You have a kind of sick desperation in your laugh.; Ramrod XTreme

My computer sure did some interesting stuff when I tried to create a ~5 GB string from a list and write it to a text file all in one go. Only made that mistake once.

# ? Dec 10, 2020 23:25

Analytic Engine: May 18, 2009; not the analytical engine

Anyone use Jupyter notebooks? And if you were to interview someone on their knowledge of them, what would you ask? I have a call coming up focused on Python and notebooks and it's been years since I used them so I don't know the cool new features or popular libraries (outside scientific computing)

# ? Dec 15, 2020 04:53

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Analytic Engine posted:

Anyone use Jupyter notebooks? And if you were to interview someone on their knowledge of them, what would you ask? I have a call coming up focused on Python and notebooks and it's been years since I used them so I don't know the cool new features or popular libraries (outside scientific computing)

I really like notebooks and I hire SW people to write python code. We actually have a "production" (runs nightly with nbconvert) bit of code in a notebooks, I love them that much. I can't imagine quizzing someone on how to use an incredibly simple to learn tool though.

This might help tho:
https://www.youtube.com/watch?v=9Q6sLbz37gk

If you're the one doing the quizzing, prob don't.

# ? Dec 15, 2020 05:04

QuarkJets: Sep 8, 2008

They may just ask you to do something in a notebook, that would be a straightforward test

# ? Dec 15, 2020 05:49

Cosa Nostra Aetate: Jan 1, 2019

Analytic Engine posted:

Anyone use Jupyter notebooks? And if you were to interview someone on their knowledge of them, what would you ask? I have a call coming up focused on Python and notebooks and it's been years since I used them so I don't know the cool new features or popular libraries (outside scientific computing)

Probably worth making sure you know some of the 'magic' commands and introspection stuff: https://ipython.readthedocs.io/en/stable/interactive/python-ipython-diff.html .

# ? Dec 15, 2020 22:42

Analytic Engine: May 18, 2009; not the analytical engine

Thanks

# ? Dec 16, 2020 03:56

hhhmmm: Jan 1, 2006; ...?

Analytic Engine posted:

Anyone use Jupyter notebooks? And if you were to interview someone on their knowledge of them, what would you ask? I have a call coming up focused on Python and notebooks and it's been years since I used them so I don't know the cool new features or popular libraries (outside scientific computing)

Except when how the plots are produced is essential, I find that notebooks tend to be bad for long term code quality. Results have been better when Ive pushed teams to get proper IDEs

# ? Dec 19, 2020 00:59

Dr Subterfuge: Aug 31, 2005; TIME TO ROC N' ROLL

hhhmmm posted:

Except when how the plots are produced is essential, I find that notebooks tend to be bad for long term code quality. Results have been better when Ive pushed teams to get proper IDEs

Still thinking about this proud Netflix post:

https://netflixtechblog.com/notebook-innovation-591ee3221233?gi=2d21ad3f17d6

# ? Dec 19, 2020 05:31

QuarkJets: Sep 8, 2008

hhhmmm posted:

Except when how the plots are produced is essential, I find that notebooks tend to be bad for long term code quality. Results have been better when Ive pushed teams to get proper IDEs

Yeah this is the big pitfall of jupyter notebooks, it's easy to get results that are not easily reproducible or simply wrong if you're not careful

# ? Dec 19, 2020 06:07

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

QuarkJets posted:

Yeah this is the big pitfall of jupyter notebooks, it's easy to get results that are not easily reproducible or simply wrong if you're not careful

Its not a bug its a feature, the notebook will reproduce the results...it's code

It also gives you the freedom to run things iteratively or out of order. I use notebooks to generate/analyze different metrics for our business, for example. I dont always want the same metrics and I frequently want to add new stuff that I wont always use.

# ? Dec 19, 2020 06:19

QuarkJets: Sep 8, 2008

CarForumPoster posted:

Its not a bug its a feature, the notebook will reproduce the results...it's code

It also gives you the freedom to run things iteratively or out of order. I use notebooks to generate/analyze different metrics for our business, for example. I dont always want the same metrics and I frequently want to add new stuff that I wont always use.

Right, nonlinear code execution is not a bug (I never said that it was) but it is a pitfall

# ? Dec 19, 2020 07:14

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

QuarkJets posted:

Right, nonlinear code execution is not a bug (I never said that it was) but it is a pitfall

It�s not a bug it�s a feature is a common quote and clearly I am disagreeing that it is a pitfall in many situations

# ? Dec 19, 2020 07:25

QuarkJets: Sep 8, 2008

CarForumPoster posted:

It’s not a bug it’s a feature is a common quote and clearly I am disagreeing that it is a pitfall in many situations

That's kind of the definition of a pitfall though: something that is usually not a problem but can become one if you're either unaware or just not thinking about it. I use this nonlinearity feature too, I think most notebook users do because it's very useful, but it's absolutely a pitfall in that it can surprise you if you're not careful

# ? Dec 19, 2020 08:02

crazysim: May 23, 2004; I AM SOOOOO GAY

Analytic Engine posted:

Anyone use Jupyter notebooks? And if you were to interview someone on their knowledge of them, what would you ask? I have a call coming up focused on Python and notebooks and it's been years since I used them so I don't know the cool new features or popular libraries (outside scientific computing)

Not interview fodder as it's too new and maybe radical and I don't know how used it is outside of fast.ai so probably ignore this.

I noticed that the fast.ai ml framework is developed using nbdev, a tool also from them: https://github.com/fastai/nbdev .

nbdev is used to transform the output of the notebooks to plain Python stuff.

It appears to me to be a very interesting way to use jupyter Python notebooks for literate programming.

# ? Dec 19, 2020 10:10

thotsky: Jun 7, 2005; hot to trot

I always thought notebooks were mainly used in university courses that teach programming to non-programmers. I would not neccesarily look down my nose if it popped up in a CV, but it would be if that was the only programming experience.

# ? Dec 19, 2020 10:53

QuarkJets: Sep 8, 2008

Notebooks are pretty common in academic research and are prolific among AI practitioners, like it's not uncommon at all to learn that a machine learning project was developed entirely in Jupyter since a lot of neural networks are largely developed by tinkering

On a CV you would list Jupyter notebooks alongside framework skills like Tensorflow, Spark, etc.

QuarkJets fucked around with this message at 11:24 on Dec 19, 2020

# ? Dec 19, 2020 11:22

Adbot: ADBOT LOVES YOU

# ? May 22, 2024 17:52

abelwingnut: Dec 23, 2002

as a data guy who needs to run lots of one-off scripts and snippets here and there, i love notebooks. they�re a critical piece of my workflow within the anaconda platform.

# ? Dec 19, 2020 16:12

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »