Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

QuarkJets: Sep 8, 2008

Thermopyle posted:

TLDR: If you accept that good, high-quality code doesn't encourage the usage of PYTHONPATH and If you accept that the ideal project doesn't encourage the usage of PYTHONPATH...then my assertion is that you should signal to other developers that you're delivering good, high-quality code by not using PYTHONPATH (or hiding your usage of it). This is not to show off, but to help them better manage the limited confidence bandwidth they have.

I think we've already established that projects shouldn't be setting PYTHONPATH. If someone is writing code that sets PYTHONPATH, or requires a user to set PYTHONPATH in order to use it, then that is brittle as hell and something needs to change. PYTHONPATH should not be used in a context where other developers would ever see you using it, it's something that you should really only set for a specific, transient reason; that's the entire scope of what I've been talking about, from the very beginning.

And I don't mean that it's okay to set it in a deployment script or something: it's not. You should almost never be setting it outside of altering a single command in your current shell.

This all goes for setting PATH and LD_LIBRARY_PATH as well. You really shouldn't be setting any kind of PATH variable in an operational context, but obviously it's useful to be able to sometimes alter these variables temporarily. This goes way, way out of scope of what we've been talking about : temporary alteration by a single developer, on a single terminal, to test changes prior to deployment, which I feel like is also the only time you should be using pip -e? I seriously hope that no one here is deploying editable environments. Neither of these tools (PYTHONPATH or pip -e) should be used in an operational context. These tools are for testing code that has not yet made it into an installable package.

quote:

I do not think I'm superior to any developer for using PYTHONPATH. That's ridiculous.

Then don't compare using a well-documented environmental variable to violating PEP8, because that's how it comes across. Saying that suggests that developers who set PYTHONPATH only do so out of ignorance, which implies some degree of inferiority. Another example:

quote:

Manually setting PYTHONPATH is less than optimal because it's an indication that the author isn't aware of all the built-in facilities and conventions that are out there. It's not similar to choosing datetime instead of arrow or whatever. It's more akin to someone using some a for loop to concatenate each letter of a string instead of just using + without any comments about why they did it that way.

Like, seriously? This loop-concatenation implementation obviously has massive technical and stylistic flaws, from how you've described it. That conveys inferiority on the part of whoever wrote it. Surely the non-ignorant developer is in some way superior to the ignorant developer?

I would say that someone who requires users to set PYTHONPATH is operating on the same level of knowledge as the flawed technical example that you've written. But a developer setting it for themselves once, for a single command? Nah

quote:

I prefer to use the tools python provides...

PYTHONPATH is one of those tools.

Convention is good. I've encountered many developers who occasionally use PYTHONPATH. I've read textbooks that describe its use. I've seen lectures that describe its use. Apparently, you yourself use it. To me, it seems like it's extremely conventional.

QuarkJets fucked around with this message at 11:24 on Jun 1, 2020

# ? Jun 1, 2020 11:21

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 18:49

Wallet: Jun 19, 2006

QuarkJets posted:

Then they came back around and compared using PYTHONPATH to violating PEP8, which doesn't make any sense to me and seems condescending, at best. That's why I brought up the arrow and pathlib examples; I like these newer modules, but I don't care if someone else wants to use datetime and os.path, sometimes those are actually the better choices.

I think they're more comparable than you seem to. I've seen people violate PEP8 intentionally for good reason (which I guess is technically following PEP8) and I've seen lots of people violate it out of ignorance, but until you look a bit closer it's sometimes very hard to tell which is which. This all started with a couple of people saying that using PYTHONPATH has a code smell, which is�to me�directly comparable to people violating PEP8. It makes me a bit leery of what other conventions the code might be violating unless there's a clear reason it was done. There are no circumstances in which I am evaluating people's usage of PATH in code that never leaves their local machine.

# ? Jun 1, 2020 15:03

Jose Cuervo: Aug 25, 2004

I have a set of 4 Jupyter Notebooks which, when run one after the other, act like a pipeline (i.e., the output from the end of the first notebook is read in by the second notebook, and so on). Is there a way to write a python script to call each notebook in order, while waiting for each notebook to finish before starting the next?

EDIT: After some searching I just found papermill which it looks like will do exactly what I want.

Jose Cuervo fucked around with this message at 19:02 on Jun 1, 2020

# ? Jun 1, 2020 18:40

QuarkJets: Sep 8, 2008

Wallet posted:

Snip. It makes me a bit leery of what other conventions the code might be violating unless there's a clear reason it was done. There are no circumstances in which I am evaluating people's usage of PATH in code that never leaves their local machine.

We're not talking about usage in code. I'd be leery of code that's invoking pip -e or PYTHONPATH, too

# ? Jun 1, 2020 22:58

Cosa Nostra Aetate: Jan 1, 2019

As the original PYTHONPATH edit recommender, I feel like I've now attained a new level of enlightenment about what "code smell" means and what people are complaining about when they complain about Pythons build and deploy tools.

It looked like the question was about testing something that was just being learned (no requirements, Pipfile, or environment.yml were in the folder structure). So I wasn't thinking about deployment, just how frustrating it is not to just get something to run.

# ? Jun 2, 2020 02:10

Dominoes: Sep 20, 2007

Re conventions and idioms: Knowing when to break the rules is a subtle art. And there are strong, and weak conventions. Strong might be snake_case, while weak might be duck typing. Today I decided to stick to the weak conventions in Python and Rust in a multi-language library, even at the cost of API consistency, which would have been a justification for breaking them:

A measurement could use an onboard, or offboard temperature source. In Python, I set up the relevant functions with a keyword argument, where if you didn't specify it, you'd use the onboard temp reading, and if you specified t=my_sensor.reading() etc, it would use the one you specified. In Rust, I made the functions always take a TempSource enum with variants OnBoard and OffBoard(reading).

My reasoning: Python APIs tend to be quick and dirty, and use kwargs. Rust APIs favor explicitness. In this case, the Python API has the adv of being terse, and the Rust API has the adv of forcing you to be aware that there are 2 things you can set the temp source. I could have had them both take an enum or something, I'm happy with the way this worked.

So:

Python code:

pH = sensor.read()
pH = sensor.read(t=23.)

Rust code:

let pH = sensor.read(OnBoard);
let pH = sensor.read(OffBoard(23.));

See if you can spot the strong convention I broke!

I also have a C++ version. I used a third option, function overloading, but this resulted in a load of DRY I'm not happy about. I don't think I can even do enums-that-hold values, or kwargs in C++, so w/e.

Dominoes fucked around with this message at 23:37 on Jun 2, 2020

# ? Jun 2, 2020 23:32

QuarkJets: Sep 8, 2008

Why not make your Python API explicit, also? Python lets you make dirty APIs but you don't have to make them dirty.

# ? Jun 3, 2020 00:01

Dominoes: Sep 20, 2007

Hmm. What do you think? Enum with method that returns the offboard temp if applicable? (Could return None, or raise an exception if you call ONBOARD.temp()) Explicit, and matches the other API.

# ? Jun 3, 2020 00:04

QuarkJets: Sep 8, 2008

Dominoes posted:

Hmm. What do you think? Enum with method that returns the offboard temp if applicable? (Could return None, or raise an exception if you call ONBOARD.temp()) Explicit, and matches the other API.

Define 2 classes, OnBoard and OffBoard. OffBoard requires a positional argument. OnBoard requires no arguments, it inherits from OffBoard and just sets the positional argument to None. Shove any other sensor-specific attributes into OnBoard and OffBoard. You could even define a readTemperatore() method, which sensor.read() invokes.

Python code:

pH = sensor.read(OnBoard())
pH = sensor.read(OffBoard(23.))

You could even make the API identical, defining 1 class (OffBoard) and creating an instance of it (OnBoard = OffBoard(None)). Then the API looks like this:

Python code:

pH = sensor.read(OnBoard)
pH = sensor.read(OffBoard(23.))

This would break a PEP8 rule, but it's a convention that I've seen commonly and pretty nice for having API symmetry.

QuarkJets fucked around with this message at 00:46 on Jun 3, 2020

# ? Jun 3, 2020 00:42

Dominoes: Sep 20, 2007

Done. That was cleaner to implement, and a better API than what I was attempting.

How would you compare equality / match the variant? isinstance?

Dominoes fucked around with this message at 01:02 on Jun 3, 2020

# ? Jun 3, 2020 00:58

QuarkJets: Sep 8, 2008

Save the positional argument as an attribute, and later check its state. If it's None, that's an internal sensor. If it's a number, that's whatever the number denotes. Two instances are the same if they have the same attribute value

(sidenote, I noticed you were using a float for the positional argument of OffBoard; what do the numbers mean?)

# ? Jun 3, 2020 06:20

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

QuarkJets posted:

Define 2 classes, OnBoard and OffBoard. OffBoard requires a positional argument. OnBoard requires no arguments, it inherits from OffBoard and just sets the positional argument to None. Shove any other sensor-specific attributes into OnBoard and OffBoard. You could even define a readTemperatore() method, which sensor.read() invokes.
Python code:
pH = sensor.read(OnBoard())
pH = sensor.read(OffBoard(23.))
You could even make the API identical, defining 1 class (OffBoard) and creating an instance of it (OnBoard = OffBoard(None)). Then the API looks like this:
Python code:
pH = sensor.read(OnBoard)
pH = sensor.read(OffBoard(23.))
This would break a PEP8 rule, but it's a convention that I've seen commonly and pretty nice for having API symmetry.

In this case you'd probably want to make an "onboardsensor" and "offboardsensor" inheriting from sensor with a read method.

Switching behavior in an oo Lang usually means inheritance for better or worse as the nicest api. Maybe even have it be a Mixin with the sensor type for other behavior.

# ? Jun 3, 2020 07:13

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

Dominoes posted:

Done. That was cleaner to implement, and a better API than what I was attempting.

How would you compare equality / match the variant? isinstance?

If you are using isinstance for single dispatch that's usually a code smell that hints at using inheritance since single dispatch is built in.

There's a neat way to do multiple dispatch with decorators (onboard/off board, temperature/pH etc) but that's getting pretty esoteric

# ? Jun 3, 2020 07:14

Dominoes: Sep 20, 2007

Quark - that works. The number's the reading you got from the offboard sensor.

isinstance does feel like a code smell here - it's unusual. (And doesn't work with Quark's second, abbreviated version) The dual inheriting class sounds like a way to make this work too.

# ? Jun 3, 2020 11:58

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Jose Cuervo posted:

I have a set of 4 Jupyter Notebooks which, when run one after the other, act like a pipeline (i.e., the output from the end of the first notebook is read in by the second notebook, and so on). Is there a way to write a python script to call each notebook in order, while waiting for each notebook to finish before starting the next?

EDIT: After some searching I just found papermill which it looks like will do exactly what I want.

Yep, Papermill.

Also a quick and dirty way is to run them from within another notebook or using the command line in a shell script/batch file like this:

code:

call jupyter nbconvert --ExecutePreprocessor.timeout=-1 --ExecutePreprocessor.allow_errors=True --to html --execute FILENAME.ipynb

I do this with a jupyter notebook I run as a daily task on Win 10 that I want it to run every cell regardless of errors. (It does a bunch of web scraping and API scraping gluing that should continue even with errors, then feeds that to a database and sends some instant messages.)

# ? Jun 3, 2020 13:13

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

Dominoes posted:

So:
Python code:
pH = sensor.read()
pH = sensor.read(t=23.)
Rust code:
let pH = sensor.read(OnBoard);
let pH = sensor.read(OffBoard(23.));
See if you can spot the strong convention I broke!

I also have a C++ version. I used a third option, function overloading, but this resulted in a load of DRY I'm not happy about. I don't think I can even do enums-that-hold values, or kwargs in C++, so w/e.

Using camelCase for variable names, hate to see it lol

In cpp you might do something like this with default arguments, or inheritance, or template specialization

# ? Jun 3, 2020 13:39

Hadlock: Nov 9, 2004

I am rewriting/rearchitecting our analytics team's daily workflow ETL thing for the database. It's a mishmash of organically written scripts over the past 7 years.

I wanted to write a wrapper script* to run these and make it portable for <reasons>

We're a python shop and that's all the analytics team knows, so wanted to write the wrapper script in python. Workflow looks something like this:

django manage.py <something> > my-etl-stuff.py
external-dataload1.py
externa-dataload2.py
api-requests.py
pentaho-etl.py
sftp.py
normalize-data.py
send-email.py

This could be a 10 line python script, is there a framework I should know about that would be better. I think I'm looking for error handing, better logging and the ability to run some of these scripts (api-requests.py, pentaho-etl.py, sftp.py)in parallel, plus like, some notifications and tie in with our alerting platform.

*ref

# ? Jun 5, 2020 19:03

12 rats tied together: Sep 7, 2006

We're using Airflow at my employer for that, which is way overkill for what you asked, but does what you need to in a reasonably robust way. I would recommend taking a look at it if you anticipate a huge growth in either the number of ETL jobs you are running or huge growth in job complexity such as inter-job dependencies, retry behaviour, conditional code paths, etc.

I pretty much set this thing up with a couple of demo jobs for examples of basic functionality (we are really big on EMR) and didn't touch it for 6 months and now we have like 70 something jobs running, the data science teams have loved it.

edit: link https://airflow.apache.org/

12 rats tied together fucked around with this message at 19:19 on Jun 5, 2020

# ? Jun 5, 2020 19:16

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

12 rats tied together posted:

We're using Airflow at my employer for that, which is way overkill for what you asked, but does what you need to in a reasonably robust way. I would recommend taking a look at it if you anticipate a huge growth in either the number of ETL jobs you are running or huge growth in job complexity such as inter-job dependencies, retry behaviour, conditional code paths, etc.

I pretty much set this thing up with a couple of demo jobs for examples of basic functionality (we are really big on EMR) and didn't touch it for 6 months and now we have like 70 something jobs running, the data science teams have loved it.

edit: link https://airflow.apache.org/

managing airflow is...not fun if you don't set it up right

it is however really easy to use for ds teams, so go hog wild

# ? Jun 6, 2020 01:33

KICK BAMA KICK: Mar 2, 2009

Just realized I've been saying "in-jinx" in my head for like a year rather than "engine-X" so now I'm curious -- I see the etymology but do you pronounce it "gyoo-nicorn" or "gun-icorn"?

# ? Jun 7, 2020 01:21

12 rats tied together: Sep 7, 2006

It's even worse for me: "gee-unicorn"

edit: I was also saying "uwsgi" by pronouncing each letter name in sequence like it was some sort of organization name.

# ? Jun 7, 2020 01:46

UraniumAnchor: May 21, 2006; Not a walrus.

Maybe the "g" is silent and it's just "unicorn".

# ? Jun 7, 2020 02:13

punished milkman: Dec 5, 2018; would have won

i feel like a dumbass calling wsgi 'whiskey'

# ? Jun 7, 2020 02:52

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

For me it�s:
Engine-x
Goo-nicorn
W-S-G-I

Are those not right? I wanna be 1337

# ? Jun 7, 2020 03:16

Hed: Mar 31, 2004; Fun Shoe

punished milkman posted:

i feel like a dumbass calling wsgi 'whiskey'

I usually just say whizz-ghee

Like this guy!

# ? Jun 7, 2020 04:59

death cob for cutie: Dec 30, 2006; dwarves won't delve no more
too much splatting down on Zot:4

at work we pronounce it "goo-nicorn", no Y sound in there

we're also kind of dinguses, tho

# ? Jun 14, 2020 04:10

Hadlock: Nov 9, 2004

CarForumPoster posted:

For me it�s:
Engine-x
Goo-nicorn
W-S-G-I

engine-ex
gun-icorn :page3:

you-wiz-ghee

lin-uhx
jiff
gib-z

my temporary boss calls CLI "the klee"

system control, not system-kettle (sysctl)

Hadlock fucked around with this message at 04:55 on Jun 14, 2020

# ? Jun 14, 2020 04:52

qsvui: Aug 23, 2003; some crazy thing

gooey

# ? Jun 14, 2020 17:39

Data Graham: Dec 28, 2009; 📈📊🍪😋

Company way back at the beginning of my career had a product with both a CLI and a brand-new-fangled Web User Interface, which was pronounced "Wooey"

The CLI by extension (I guess) was the "silly" :psyduck:

# ? Jun 14, 2020 17:41

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

I've got a little thing splitting some addresses for a quick n dirty one off test.

code:

This works:
joel_df.loc[18274, 'ZIP'] = add_df['ZIP']

And this works:
    for column in add_df.index:
        joel_df.loc[index, column] = add_df[column]

Works but this gives a syntax error with the last line:

code:

    [(joel_df.loc[index, column] = add_df[column]) for column in add_df.index]

Am I losing my mind? Whats wrong with that syntax?

CarForumPoster fucked around with this message at 17:51 on Jun 14, 2020

# ? Jun 14, 2020 17:49

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

CarForumPoster posted:

Works but this gives a syntax error with the last line:
code:
    [(joel_df.loc[index, column] = add_df[column]) for column in add_df.index]
Am I losing my mind? Whats wrong with that syntax?

Assignments can't be used like that, basically. Elements of list comprehensions at least nominally have to be able to evaluate to something, and assignment expressions can't be used in the rhs of an expression. You need to refactor this so that you don't have to use assignment, somehow (don't know how dataframes work off the top of my head)

# ? Jun 14, 2020 18:05

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Phobeste posted:

Assignments can't be used like that, basically. Elements of list comprehensions at least nominally have to be able to evaluate to something, and assignment expressions can't be used in the rhs of an expression. You need to refactor this so that you don't have to use assignment, somehow (don't know how dataframes work off the top of my head)

I just did this instead:

code:

    for column in add_df.index:
        joel_df.loc[index, column] = add_df[column]

But huh didn't know that. Guess I had never tried.

# ? Jun 14, 2020 18:17

wolrah: May 8, 2006; what?

Data Graham posted:

Company way back at the beginning of my career had a product with both a CLI and a brand-new-fangled Web User Interface, which was pronounced "Wooey"

The CLI by extension (I guess) was the "silly"

I've always preferred "Web GUI" or just "Web UI". Trying to work web in to the acronym just doesn't come out smooth.

The ones I've never come up with a good answer for though are console-based interfaces that aren't command lines, "curses" type interfaces for example, the sort of things that'd have been a precursor to GUIs from the terminal era. It's not a CLI, it's not a GUI, but what? TUI? CUI?

# ? Jun 14, 2020 18:49

Bruegels Fuckbooks: Sep 14, 2004; Now, listen - I know the two of you are very different from each other in a lot of ways, but you have to understand that as far as Grandpa's concerned, you're both pieces of shit! Yeah. I can prove it mathematically.

wolrah posted:

I've always preferred "Web GUI" or just "Web UI". Trying to work web in to the acronym just doesn't come out smooth.

The ones I've never come up with a good answer for though are console-based interfaces that aren't command lines, "curses" type interfaces for example, the sort of things that'd have been a precursor to GUIs from the terminal era. It's not a CLI, it's not a GUI, but what? TUI? CUI?

I like TUI. Granted it's a retronym (e.g. people didn't call text based user interfaces TUI back in the day.) https://en.wikipedia.org/wiki/Text-based_user_interface

# ? Jun 14, 2020 19:08

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

Phobeste posted:

Assignments can't be used like that, basically. Elements of list comprehensions at least nominally have to be able to evaluate to something, and assignment expressions can't be used in the rhs of an expression. You need to refactor this so that you don't have to use assignment, somehow (don't know how dataframes work off the top of my head)

I am unfamiliar with them either and it looks like they don't have a simple setter/getter set up so all I could think of was an ugly soup using getattr and setattr. That's strange to me because I would think that kind of list comprehension would be common enough.

# ? Jun 14, 2020 19:35

Data Graham: Dec 28, 2009; 📈📊🍪😋

Bruegels Fuckbooks posted:

I like TUI. Granted it's a retronym (e.g. people didn't call text based user interfaces TUI back in the day.) https://en.wikipedia.org/wiki/Text-based_user_interface

Haha yeah, I remember in the DOS days these were considered "graphical" UIs :newlol:

# ? Jun 14, 2020 19:51

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

Rocko Bonaparte posted:

I am unfamiliar with them either and it looks like they don't have a simple setter/getter set up so all I could think of was an ugly soup using getattr and setattr. That's strange to me because I would think that kind of list comprehension would be common enough.

Yeah, but it�s really hard to define an ergonomic api for multidimensional data as I continue to face to my despair at my day job

# ? Jun 14, 2020 20:14

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

Phobeste posted:

Yeah, but it�s really hard to define an ergonomic api for multidimensional data as I continue to face to my despair at my day job

What I was hoping to find was a set(value, *args) kind of thing. I don't think there's anything cooler to try unless a lot of the classic Python dunders or functions can take *args. Like, I don't think getattr can take multiple dimensions at a time.

I had a partial interest in this because I've been neck-deep in Python syntax lately and was looking for cool tricks.

# ? Jun 14, 2020 21:12

Proteus Jones: Feb 28, 2013

Bruegels Fuckbooks posted:

I like TUI. Granted it's a retronym (e.g. people didn't call text based user interfaces TUI back in the day.) https://en.wikipedia.org/wiki/Text-based_user_interface

I remember them being called CURSES interfaces in the mid/late 90s early '00s, since a large amount of them were built (at least in BSD/Linux) using the curses terminal control library.

# ? Jun 14, 2020 23:35

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 18:49

Bad Munki: Nov 4, 2008; We're all mad here.

This is the right place to get some guru eyes on, yeah? I'm trying to run a command via Popen, and that command runs its own shell type thing, and ultimately I want to feed commands to that shell so I can periodically run a command. My script is largely lifted from stack overflow (https://stackoverflow.com/questions/19880190/interactive-input-output-using-python, the response a few down starting with "Here is an interactive shell."). If I replace the command I really want (standalone terraria server) with, say, cmd.exe (yeah, this is on windows, which surely doesn't help things) I can run commands through my proxy shell just fine. I can even run the game server itself just fine. But further input doesn't seem to make it to the game server's shell's input. Doesn't matter if I Popen cmd.exe and then start the server by hand from within there, or if I Popen the game server executable itself. Any commands input I try to send to the game server's shell just...evaporates? I dunno. But I know the underlying thing is working because I can proxy cmd.exe without issue. Is there some quirk of Windows I'm not aware of to get stuff to forward past the shell to the game server's input?

Am I missing something super obvious here or what?

code:

import sys
import os
import subprocess
from subprocess import Popen, PIPE
import threading

class ServerProxy(object):
    def __init__(self):
        pass

    def run(self):
        env = os.environ.copy()
        p = Popen(
            ['TerrariaServer.exe', '-config', 'serverconfig.txt'],
            cwd='D:/Steam/steamapps/common/Terraria',
            stdin=PIPE,
            stdout=PIPE,
            stderr=subprocess.STDOUT, shell=True, env=env)
        sys.stdout.write('Starting Terraria server...\n')

        def writeall(p):
            while True:
                data = p.stdout.read(1).decode("cp437")
                if data is not None:
                    sys.stdout.write(data)
                    sys.stdout.flush()

        writer = threading.Thread(target=writeall, args=(p,))
        writer.start()

        try:
            while True:
                d = sys.stdin.read(1)
                if not d:
                    break
                self._write(p, d.encode(encoding='cp437'))
        except EOFError:
            pass

    def _write(self, process, message):
        process.stdin.write(message)
        process.stdin.flush()

server = ServerProxy()
server.run()

Bad Munki fucked around with this message at 04:53 on Jun 15, 2020

# ? Jun 15, 2020 04:50

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »