Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Yeah, having to use PYTHONPATH is a code smell.

Adbot
ADBOT LOVES YOU

bobmarleysghost
Mar 7, 2006



Yea I haven't had to touch that at all in my extremely short foray into coding in python, this is the first time I'm running into this issue.


The code is invoked from the application/main.py module, but even that doesn't work without PYTHONPATH being declared:
code:
Traceback (most recent call last):
  File "application/main.py", line 2, in <module>
    from db import DB
  File "/home/bmg/cam-data-harvester/application/db/__init__.py", line 6, in <module>
    from application.config import Config as app_config
ModuleNotFoundError: No module named 'application'
I really want to know what I'm doing wrong.

And this is more less what's inside of main.py:

code:
from db import DB
from db.models import EventsData, LastEvent
from datetime import datetime

def main():
    #stuff here removed for legibility


if __name__ == "__main__":
    main()



And the imports freom db/__init__.py:

code:
import os
import sqlite3
from sqlalchemy import create_engine, MetaData
from sqlalchemy.orm import sessionmaker, exc

from application.config import Config as app_config
from .models import Base

bobmarleysghost fucked around with this message at 00:24 on May 30, 2020

KICK BAMA KICK
Mar 2, 2009

Does application import from db? Could be a circular import problem.

bobmarleysghost
Mar 7, 2006



application/__init__.py is empty.

I feel like I haven't structured the code properly..

QuarkJets
Sep 8, 2008

Thermopyle posted:

Yeah, having to use PYTHONPATH is a code smell.

In this case yeah, but not usually. Messing with sys.path is for sure a code smell, using PYTHONPATH usually just means that you're using code that's not in a stanard installed location. If you use venv or conda then you're using PYTHONPATH.

QuarkJets fucked around with this message at 01:10 on May 30, 2020

SurgicalOntologist
Jun 17, 2004

Are you sure your own package is installed? It looks like you're using poetry and I don't know the ins and outs of that, but if your package is installed it should be on the path regardless.

Your stack trace also indicates that python is not able to find your application at all, so I don't think the problem is anything within application. Circular import, for example, would have a different traceback.

bobmarleysghost
Mar 7, 2006



Well I guess I'm super dumb - I had main.py inside of application/
I moved it to the root of the project and it works without having to use PYTHONPATH.

Do any of you have good reading material on how to properly structure a python project?

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

QuarkJets posted:

If you use venv or conda then you're using PYTHONPATH.

By this definition most everything you do on your PC you're setting different environment variables because somewhere in the machinery of the deployment or build something somewhere is setting them.

Of course, I'm talking about having to set PYTHONPATH yourself. In most cases where people set it themselves there's a better way to do what they're trying to do.

I'm sure there's some valid cases, but they're not usual and typical...which is why I said "code smell" and not "you've hosed up by doing it".

Thermopyle fucked around with this message at 02:05 on May 30, 2020

QuarkJets
Sep 8, 2008

Thermopyle posted:

By this definition most everything you do on your PC you're setting different environment variables because somewhere in the machinery of the deployment or build something somewhere is setting them.

Of course, I'm talking about having to set PYTHONPATH yourself. In most cases where people set it themselves there's a better way to do what they're trying to do.

I'm sure there's some valid cases, but they're not usual and typical...which is why I said "code smell" and not "you've hosed up by doing it".

Most things that you do on a computer aren't setting your PYTHONPATH, that would be pretty unusual. But it's pretty normal to set that while you develop stuff, because git repos aren't usually located in the default search path. So setting it as part of an installation step is code smell, but setting it because you haven't installed some code to a standard location yet is pretty normal

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

QuarkJets posted:

Most things that you do on a computer aren't setting your PYTHONPATH, that would be pretty unusual.

I'm not sure how that's relevant? I didn't claim that.

QuarkJets posted:

But it's pretty normal to set that while you develop stuff, because git repos aren't usually located in the default search path. So setting it as part of an installation step is code smell, but setting it because you haven't installed some code to a standard location yet is pretty normal

Yes, it's fairly common but it's also the wrong way to do it. editable packages exist for a reason.

QuarkJets
Sep 8, 2008

Why does that make using PYTHONPATH wrong?

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

QuarkJets posted:

Why does that make using PYTHONPATH wrong?

Hmm, calling something a code smell doesn't mean it's wrong, per se. It doesn't mean you've broken something. It means its a warning that others should watch out for wrong things.

If someone is commonly telling others to set PYTHONPATH or commonly setting it themselves, it's evidence that they're not careful about what they're doing. There's facilities specifically built for many of the common reasons PYTHONPATH is used and they're almost always better. In many of the cases where these other facilities are not obviously and immediately better, they're better in the long run.

For example, there's an extensive history of various python packaging and build tools being broken by users editing their own PYTHONPATH that can be completely avoided by just using the appropriate tools.

Now, you might say, "I'll just be very careful" and that might work great! But it's still a smell. We should be writing software that is resilient to failure and also, importantly, signaling this to others. Seeing someone choosing the fragile, brittle course when there are other options should make you wary about what else they've done.

It's just a smell. Everything might be (probably is!) fine. It's just an indication. There are certainly times where PYTHONPATH editing is the correct thing to do.

QuarkJets
Sep 8, 2008

To be more specific:

Thermopyle posted:

QuarkJets posted:

But it's pretty normal to set that while you develop stuff, because git repos aren't usually located in the default search path. So setting it as part of an installation step is code smell, but setting it because you haven't installed some code to a standard location yet is pretty normal

Yes, it's fairly common but it's also the wrong way to do it. editable packages exist for a reason.

What's wrong about using PYTHONPATH here? I'm asking because I'd like to know if there's some hidden pitfall that I haven't considered to modifying PYTHONPATH for development sessions.

I have an operational venv that I could clone and then pollute with a git repo, but A) that's kind of a time-and-space waster if the venv is large and B) the git repo may not have the same file structure as the code would once installed into the venv, for whatever reason. Editing files in the venv directly would be bad practice since those changes would be untracked. So editing PYTHONPATH here seems like the best option, but you're saying that's bad because editable packages exist. Could you elaborate? Your followup was kind of generalized, I'd like to get to the bottom of this situation in particular.

Phobeste
Apr 9, 2006

never, like, count out Touchdown Tom, man

QuarkJets posted:

To be more specific:

quote:

Yes, it's fairly common but it's also the wrong way to do it. editable packages exist for a reason.

What's wrong about using PYTHONPATH here? I'm asking because I'd like to know if there's some hidden pitfall that I haven't considered to modifying PYTHONPATH for development sessions.

I have an operational venv that I could clone and then pollute with a git repo, but A) that's kind of a time-and-space waster if the venv is large and B) the git repo may not have the same file structure as the code would once installed into the venv, for whatever reason. Editing files in the venv directly would be bad practice since those changes would be untracked. So editing PYTHONPATH here seems like the best option, but you're saying that's bad because editable packages exist. Could you elaborate? Your followup was kind of generalized, I'd like to get to the bottom of this situation in particular.

Well, I think the point he's making is "you could do that but there's better ways and not knowing those ways might mean you don't know other best practices" which is true but kind of a dickish thing to snidely assert without actually providing an alternative.

The alternative, btw, is "editable packages". Use the -e option to pip install or poetry install or whatever and it'll put a .egg-link file in your default interpreter site-packages that contains the path for the interpreter to look to find a file. That's the intended workflow as far as I know for installing a package that you're also working on.

Anyway, using PYTHONPATH instead _will_ obviously work but if you do the editable package thing you're putting more of the work onto the interpreter's built in loading methods which means your method of using the package is closer to the nominal end user's which means you might miss fewer errors around prod/dev configuration differences.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

QuarkJets posted:


What's wrong about using PYTHONPATH here? I'm asking because I'd like to know if there's some hidden pitfall that I haven't considered to modifying PYTHONPATH for development sessions.

Like I said, there's nothing wrong with it in the sense that you'll break something immediately and obviously.


Phobeste posted:

Well, I think the point he's making is "you could do that but there's better ways and not knowing those ways might mean you don't know other best practices" which is true but kind of a dickish thing to snidely assert without actually providing an alternative.


Your post basically covers my answer.

(Except I did tell him to use editable packages already.)

Thermopyle fucked around with this message at 19:38 on May 30, 2020

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
My own take is that PYTHONPATH during development is totally fine since your source is God-knows-where, but you should try to deploy in a way that doesn't require it when releasing.

Dominoes
Sep 20, 2007

PYTHONPATH is so awk. I'll go a step further and say it's a smell for the entire venv workflow.

QuarkJets
Sep 8, 2008

It's pretty intuitive for nix users, but may seem weird if you primarily use Windows

Thermopyle posted:

Like I said, there's nothing wrong with it in the sense that you'll break something immediately and obviously.



Your post basically covers my answer.

(Except I did tell him to use editable packages already.)

So in this case it's just personal preference?

Dominoes
Sep 20, 2007

It's totally personal pref. And nailed the Windows thing.

I think the biggest arg agaisnt PYTHONPATH and venvs is keeping track of state, assuming you have a workflow that skips cd between venv and proj paths.

Dominoes fucked around with this message at 00:29 on May 31, 2020

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Rocko Bonaparte posted:

My own take is that PYTHONPATH during development is totally fine since your source is God-knows-where, but you should try to deploy in a way that doesn't require it when releasing.

Well that's part of it. Why is you source God-knows-where and why can't you pip -e it?

Dominoes
Sep 20, 2007

To clarify on the stateful thing: Unix-familiarity or not, I don't like executing code from my proj's folder to have different results based on hidden variables. It's a trap. I built a whole python/dep manager to hide this workflow behind something that's tied to your project vice system config.

Malcolm XML
Aug 8, 2009

I always knew it would end like this.
PYTHONPATH fuckery is a big indication that you are doing something stupid

QuarkJets
Sep 8, 2008

Dominoes posted:

To clarify on the stateful thing: Unix-familiarity or not, I don't like executing code from my proj's folder to have different results based on hidden variables. It's a trap. I built a whole python/dep manager to hide this workflow behind something that's tied to your project vice system config.

I don't really know what you mean by "hidden" variable; it's a well-documented feature of Python and it's probably being used by your python/dep manager, unless you're actually altering PATH (which at the end of the day basically has the same effect).

The whole point of setting PYTHONPATH is to alter how your code behaves, so I don't see that as a criticism. It's the kind of alteration that should be temporary, not something that you have set all the time:
code:
PYTHONPATH=/some/new/path your_python_scripy.py
e.g. you invoke your script this way while you work on an altered dependency, and then when you're done you install your altered dependency and launch the script normally to test it.

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!

Thermopyle posted:

Well that's part of it. Why is you source God-knows-where and why can't you pip -e it?

"Just clone it from git and pull if you need the latest. That's how we deploy. :smug: :smug: :smug:"

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

QuarkJets posted:

code:
PYTHONPATH=/some/new/path your_python_scripy.py
e.g. you invoke your script this way while you work on an altered dependency, and then when you're done you install your altered dependency and launch the script normally to test it.

Addressing that scenario is exactly why editable packages, aka pip -e and the like, exist.

Your method works and doesn't break anything...much like using camelCase method names definitely works and doesn't break anything.

However, when I see a developer using camelCase it makes me wonder what horrors lie beneath. Maybe there's nothing else and everything is fine. But it's a code smell that makes me wonder.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Thermopyle posted:

Addressing that scenario is exactly why editable packages, aka pip -e and the like, exist.

Your method works and doesn't break anything...much like using camelCase method names definitely works and doesn't break anything.

However, when I see a developer using camelCase it makes me wonder what horrors lie beneath. Maybe there's nothing else and everything is fine. But it's a code smell that makes me wonder.

I didn't know that camelCase was the name for this. I've had applicants that do this in their Python coding interviews or on their githubs and I noticed the same thing.

CarForumPoster fucked around with this message at 19:26 on May 31, 2020

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

CarForumPoster posted:

I didn't know that camelCase was the name for this. I've had applicants that do this in their Python coding interviews or on their githubs and I noticed the same thing.

snake_case is the Python convention!

You probably shouldn't use PascalCase or kebab-case either. :D

Hed
Mar 31, 2004

Fun Shoe

Thermopyle posted:

kebab-case

Thank you.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Thermopyle posted:

You probably shouldn't use PascalCase

Well, except for classes.

Dominoes
Sep 20, 2007

I sometimes break the rules for notation that follow a convention a non-programming context, like measurement units or math/science. Wrong case, non-ascii etc. I think most people go with the programming-language conventions in this case, but it's w/e.

QuarkJets
Sep 8, 2008

Thermopyle posted:

Addressing that scenario is exactly why editable packages, aka pip -e and the like, exist.

Your method works and doesn't break anything...much like using camelCase method names definitely works and doesn't break anything.

However, when I see a developer using camelCase it makes me wonder what horrors lie beneath. Maybe there's nothing else and everything is fine. But it's a code smell that makes me wonder.

:jerkbag:

You still haven't explained why you feel like using PYTHONPATH is a sin, just that you prefer using pip -e for the same purposes. Really it just comes across as a personal bias, like you'd look down your nose at someone for using datetime instead of arrow or path instead of pathlib. It's all starting to come across as kind of petty. Unlike specifications for case conventions or other style choices, "PYTHONPATH exists but you shouldn't ever use it" isn't written anywhere in a PEP, but you're pretending like it is because you're more comfortable doing something else.

Dominoes
Sep 20, 2007

QuarkJets posted:

I don't really know what you mean by "hidden" variable;
Hidden wasn't a great word choice. `System` or `system state` are better. Getting someone else's Python code working can be a pain due to version and dependency mismatches that are also dependent on system state. It's better now that you're less likely to run into Py2, but still.

NinpoEspiritoSanto
Oct 22, 2013




There's no need to be so combative or protective over such things. It's not unusual in software engineering to use one way until a better way comes along. PYTHONPATH is one of those things that it's just better to not develop a habit for because yeah it works great until it doesn't. It's a solved problem with venvs and pip -e anyway and in a far more robust fashion. What's being perceived as arrogant from Thermopyle is just an engineer wondering why you wouldn't do it the Right way instead. Anecdotal, but plenty of people run foul of loving with PYTHONPATH and going to python IRC for help.

There should be one and preferably only one way to do it.

Dominoes
Sep 20, 2007

Bundy posted:

There should be one and preferably only one way to do it.
That one didn't work out too well.

QuarkJets
Sep 8, 2008

Bundy posted:

There's no need to be so combative or protective over such things. It's not unusual in software engineering to use one way until a better way comes along. PYTHONPATH is one of those things that it's just better to not develop a habit for because yeah it works great until it doesn't. It's a solved problem with venvs and pip -e anyway and in a far more robust fashion. What's being perceived as arrogant from Thermopyle is just an engineer wondering why you wouldn't do it the Right way instead. Anecdotal, but plenty of people run foul of loving with PYTHONPATH and going to python IRC for help.

There should be one and preferably only one way to do it.

My question to Thermopyle was "why is this alternative the Right way?" and the answer boiled down to "well it isn't, really, but I prefer it". That was cool and good, we had reached agreement.

Then they came back around and compared using PYTHONPATH to violating PEP8, which doesn't make any sense to me and seems condescending, at best. That's why I brought up the arrow and pathlib examples; I like these newer modules, but I don't care if someone else wants to use datetime and os.path, sometimes those are actually the better choices.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Because I keep posting about the subject it probably seems like I care more about PYTHONPATH than I do. What I actually do care about is clear communication, so here we are. I just barely care enough about PYTHONPATH to have made the offhanded initial post. I also care a lot about the issue and usefulness of best practices and conventions in general, so, dear reader, if you don't care about PYTHONPATH just pretend I'm talking about any best practice or convention.

TLDR: If you accept that good, high-quality code doesn't encourage the usage of PYTHONPATH and If you accept that the ideal project doesn't encourage the usage of PYTHONPATH...then my assertion is that you should signal to other developers that you're delivering good, high-quality code by not using PYTHONPATH (or hiding your usage of it). This is not to show off, but to help them better manage the limited confidence bandwidth they have.

QuarkJets posted:

You still haven't explained why you feel like using PYTHONPATH is a sin,

I feel like I have. :(


So, I'm not sure how to explain this another way so to me it feels like I'm just saying what I already said, but I'll try again:

The reasons to not use PYTHONPATH are like 90% social and 10% technical.

(As an aside, I have a feeling that describes a large portion of so-called programming problems)

I do not think I'm superior to any developer for using PYTHONPATH. That's ridiculous. I guarantee there's tens of thousands developers telling everyone to use PYTHONPATH who are more competent than me. I'm barely competent. A big thing that keeps me treading water is all the developers around who follow best practices and I don't think you're going to find many project managers for leading projects who are going to say using PYTHONPATH instead of editable packages is a best practice.

It's not that I prefer using pip -e instead of PYTHONPATH, it's that most good and well maintained code, projects, project contribution guidelines, workplace guidelines that I've been in contact with recommend editable package approach and most of the times I see PYTHONPATH recommended it's people throwing poo poo at the wall. I mean, maybe my experience is not representative, but that's all anyone can go by. I have spent a lot of time consulting on related issues so my gut tells me I'm exposed to a wider variety of projects than your average developer, but who really knows?

This paragraph is only tangentially related to my point, but I think it's probably the information you're looking for: The reason editable packages exist is largely because setuptools/distutils/pip/etc have all had to deal with bugs stemming from people setting PYTHONPATH. That's the technical reason to use editable packages...they're more robust. If you want more details you'll have to go dig through the corresponding mailing lists from years ago when I was a regular on them. I don't remember all the details at this point.

Manually setting PYTHONPATH is less than optimal because it's an indication that the author isn't aware of all the built-in facilities and conventions that are out there. It's not similar to choosing datetime instead of arrow or whatever. It's more akin to someone using some a for loop to concatenate each letter of a string instead of just using + without any comments about why they did it that way. I prefer to use the tools python provides...but more importantly I prefer other developers to demonstrate they grasp the environment they're working in because if I have to use their code I'm already accepting a boat load of stuff on faith. Maybe they had a good reason to. Probably their code is fine. Likely their code is better on all dimensions than anything I could write. But, it raises questions from the get-go and that's a disservice to users of your code.

Sometimes I use PYTHONPATH because I'm lazy, but I don't tell anyone else to use PYTHONPATH nor will you find it being used by me anywhere another developer might encounter it. You could say that using PYTHONPATH isn't part of my "public API" . APIs that expose unconventional interfaces make me worry about the implementation. (hello half the stdlib...much of which is weird but works fine)

And now you've made my expose my laziness on my public API, thanks a lot!

FYI, you shouldn't use case conventions because they're written in a PEP, you should use them because they're conventions and conventions are important stuff! PEP8 just codified things that were already convention. In the days prior to PEP8 it was not unreasonable to do a double-take at code using camelCase.

There's many things that aren't written in a PEP but are conventions you should avoid breaking without good reason anyway.

It's already so hard to work on code. Developers should do a things to signal competence. Increasing the confidence your coworkers can have in your code is valuable.



QuarkJets posted:

and the answer boiled down to "well it isn't, really, but I prefer it".

I mean, in some sense everything you want the world to be is only that way because you prefer it. I prefer it because I think it makes the world work better.


QuarkJets posted:

Then they came back around and compared using PYTHONPATH to violating PEP8,

Like all analogies the comparison only goes so far and I feel like you should've read it more charitably.


I probably won't post about PYTHONPATH again, but the utility of best practices and conventions are near and dear to my heart and I can talk about them a lot. Many of my favorite problems nowadays are less technical and more about improving the stuff I talked about in this post.

pmchem
Jan 22, 2010


Thermopyle posted:

It's already so hard to work on code. Developers should do a things to signal competence.

Sometimes when you’re writing a small set of scripts for a very specialized scientific application that only a limited number of humans will ever use, and that needs to run on computers with restricted access and weird paths and python configurations, you just tell people to set the PYTHONPATH and move on the the more important problems you’re trying to solve that actually matter.

It has nothing to do with competence.

Zoracle Zed
Jul 10, 2001

Genuinely good post.

At risk of completely ignoring all of the social context you're discussing and reverting to purely technical solutions: recently I had to do some unfortunate pythonpath wrangling and discovered that, if you're in a conda env,

Sh code:
conda env var set "PYTHONPATH=$(pwd):$PYTHONPATH"
will add the cwd to pythonpath only when the venv is active. (And conda'll print a little warning message when you activate the venv.) It's a nice intermediate between setting an envvar temporarily (and forgetting to redo so every time you restart your console), and stuffing it in your bashrc and forgetting to tell other people about it when you share code. Actually there's even support for it by conda-dev for putting it in your environment.yml file so it's explicitly specified.

Dominoes
Sep 20, 2007

Love it Therm.

Adbot
ADBOT LOVES YOU

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
I work in a huge-rear end company and have more particular reasons to avoid PYTHONPATH. Generally, these reasons come down to people being inexperienced with environment variables and/or Python. This unfortunately means that using virtual environments is also an uphill battle. So some stuff with PYTHONPATH:

1. You risk people mishandling that environment variable when they have more than one package from different people that both need to be in PYTHONPATH. If you just tell them to set PYTHONPATH=blablabla then somebody's gonna lose.
2. A place with a PYTHONPATH culture that also installs packages will wind up with a situations where the application is installed as a package while PYTHONPATH is pointing to somewhere else. They then report a bug you fixed a year ago despite having installing the honest-to-god latest.
3. People create deployment systems that set PYTHONPATH to God-knows-what and you don't even realize it's set. It then causes you to import from somewhere you didn't expect.
4. Subversive shell scripters that are using your software behind multiple layers of batch/bash files write over the environment time and time again and you lost track of where your stuff is actually importing.
5. [this one is more about messing with sys.path directly] you're consuming other people's stuff that isn't deployed using a standard so you have to mess with sys.path. Somebody imports your code and discovers you're loving with sys.path and it breaks them. Suddenly you're doing all this try-catch-finally acrobats to undo your manipulation of the path, only to discover what you imported also messed with sys.path so you're suddenly parsing the drat thing to try to keep it straight.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply