Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

Norton the First: Dec 4, 2018; by Fluffdaddy

SurgicalOntologist posted:

code:
df.groupby('Middle Level').sum()
One thing to get used to, if you're coming from Excel, is that you wouldn't put the subtotals "under" the raw data. I mean, you could probably do it, with the above line then assigning new rows, but it would be awkward. Better to use pandas to do the calculations, then if you need a pretty output at some point you would output to html or something.

By the way, I found a solution here to my original question:

code:

test_table = df.groupby(['First Level', 'Second Level']).apply(lambda x: x.pivot_table("Values", index=["Third Level"], columns=["Columns"], margins=True, margins_name="Subtotal"))

Using apply to group things into sub-tables is a neat idea I wouldn't have considered.

# ? Oct 24, 2019 17:06

Adbot: ADBOT LOVES YOU

# ? May 25, 2024 18:55

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Business posted:

Thanks yeah I was overthinking it. Messed something up the first try and based on google got the impression it was more complicated than it actually was

I use selenium a lot and when creating new envs I always:

code:

pip install selenium
conda install -c conda-forge geckodriver

This sorts out the path issue, where the binary is, etc. Makes life breezy

# ? Oct 24, 2019 18:39

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

I was looking for a way to convert a pyproject.toml file used by poetry into a setup.py file and found DepHell which seems to do the job. It manages to include all non-python files that need to be included so that�s cool. It can also go the opposite way and also supports pipfile, flit, and conda.

Just in case anyone else was looking for something like that.

# ? Oct 30, 2019 11:37

NinpoEspiritoSanto: Oct 22, 2013

Boris Galerkin posted:

I was looking for a way to convert a pyproject.toml file used by poetry into a setup.py file and found DepHell which seems to do the job. It manages to include all non-python files that need to be included so that’s cool. It can also go the opposite way and also supports pipfile, flit, and conda.

Just in case anyone else was looking for something like that.

Nice thanks for this

# ? Oct 30, 2019 11:57

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

dephell posted:

Why it is better than all other tools:

Now that's some inspiring confidence!

# ? Oct 30, 2019 16:34

Mycroft Holmes: Mar 26, 2010; by Azathoth

no matter what I input, i'm tripping the first if statement about seconds.

code:

time = input("Enter time [hh:mm:ss]: ")
number_colon = 0

if time[2] != ':' or time[5] != ':':
    print("Must separate hour, minute and second with colons.")
    quit()

second = time[6:7]
length = len(second)
num_second = int(second)
if length != 2 or second.isdigit() is False:
    print("Second must be a 2-digit number.")
    quit()
if num_second < 0 or num_second > 59:
    print("Second must be between 0 and 59.")
    quit()

minute = time[6:7]
length = len(minute)
num_minute = int(minute)
if length != 2 or minute.isdigit() is False:
    print("Minute must be a 2-digit number.")
    quit()
if num_minute < 0 or num_minute > 59:
    print("Minute must be between 0 and 59.")
    quit()

hour = time[6:7]
length = len(hour)
num_hour = int(hour)
if length != 2 or hour.isdigit() is False:
    print("Hour must be a 2-digit number.")
    quit()
if num_hour < 0 or num_hour > 23:
    print("Hour must be between 0 and 23.")
    quit()

time_list = time.split(':')
print("Time with colons removed: ",time_list[0],time_list[1],time_list[2])

# ? Nov 3, 2019 04:24

Dominoes: Sep 20, 2007

Mycroft Holmes posted:

no matter what I input, i'm tripping the first if statement about seconds.

Your indexes are wrong. Should be

code:

second = time[6:]
minute = time[3:5]
hour = time[0:2]

This is easy to troubleshoot by inserting breakpoint() before the error, or using print() statements on the variables in question.

# ? Nov 3, 2019 04:33

Solumin: Jan 11, 2013

Why not call time.split first instead of last?

code:

if time[2] != ':' or time[5] != ':':
    print("Must separate hour, minute and second with colons.")
    quit()
hour, minute, second = time.split(':')
if len(second) != 2 or not second.isdigit():
  print("Second must be a two-digit number.")
  quit()
# I'm not sure if you can use the walrus operator here.
num_seconds = int(second)
if num_seconds < 0 or num_seconds > 59:
  # and so on.

(Of course, you can simplify this even further with regex.)

# ? Nov 3, 2019 05:27

Dominoes: Sep 20, 2007

Or:

Python code:

from datetime import datetime

time = input("Enter time [hh:mm:ss]: ")
parsed = datetime.strptime(time, "%H:%M:%S")

print(f"Time with colons removed: {parsed.hour} {parsed.minute} {parsed.second}")

# ? Nov 3, 2019 06:10

a foolish pianist: May 6, 2007; (bi)cyclic mutation

Mycroft Holmes posted:

code:

second = time[6:7]

Python indices like this are inclusive:exclusive. That's going to give you all the characters starting at index 6 before index 7.

code:

>>> x = "012345678"
>>> x[5:6]
'5'
>>> x[5:7]
'56'

# ? Nov 4, 2019 19:52

Mycroft Holmes: Mar 26, 2010; by Azathoth

new problem. My output to a file is incorrect. It isn't printing to a new line when I put in /n

code:

i = 0
output_file = open('water.txt', 'w')

while i < 4:
    account = input("Enter account number")
    while not account.isdigit():
        print("Invalid account number")
        account = input("Enter account number")
    custType = input("Enter R for residential or B for business: ")
    while custType != 'R' and custType != 'B':
        print("Invalid customer type")
        custType = input("Enter R for residential or B for business: ")
    gallons = input("Enter number of gallons")
    while not gallons.isdigit():
        print("Invalid number of gallons")
        gallons = input("Enter number of gallons")
    output_string = account+' '+custType+' '+gallons+'/n'
    output_file.write(output_string)
    i = i + 1

output_file.close()

output

code:

3147 R 1000/n2168 R 6002/n5984 B 2000/n7086 B 8002/n

# ? Nov 4, 2019 19:58

a foolish pianist: May 6, 2007; (bi)cyclic mutation

# ? Nov 4, 2019 20:01

Solumin: Jan 11, 2013

Newlines are '\n', not '/n'.

# ? Nov 4, 2019 20:01

Hemingway To Go!: Nov 10, 2008; im stupider then dog shit, i dont give a shit, and i dont give a fuck, and i will never shut the fuck up, and i'll always Respect my enemys.
- ernest hemingway

So recently trying to install multiple versions of python on ubuntu 18.04 and then uninstalling python3.6.8 completely broke my installation.
I installed python 3.6.8 when latest python3 was already installed and software updates and the terminal broke. So I uninstalled python3.6.8 and absolutely everything broke, on next book it could not even load graphics.

As you can tell I'm not very good at linux.

I do now have a new installation of ubuntu 19 now, and "python" isn't even a recognized command, but I'm spooked enough that I'd like to ask for advice about installing multiple python installations on ubuntu. python 3.6.8 is specifically needed for certain projects - like I want to play with evennia as it involves all the things I'm trying to learn right now, but some of its packages need 3.6.8 and nothing later specifically, and there's some other things like that like a Code for Boston project. I could just install python 3.6.8 but I'm afraid I'll need the latest python at some point, and I no longer trust whatever instructions I was following.

# ? Nov 4, 2019 20:16

necrotic: Aug 2, 2005; I owe my brother big time for this!

Use pyenv or anaconda to use different python versions.

# ? Nov 4, 2019 20:27

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

necrotic posted:

Use pyenv or anaconda to use different python versions.

This.

I have no less than 5 python versions in different conda envs right now.

To make a new one, conda will handle all the installation and everything, just type:

code:

conda create -n "name" python=3.x

# ? Nov 4, 2019 21:42

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

This is a decent explanation on why you'd want to use pyenv or conda and how to use pyenv.

https://realpython.com/intro-to-pyenv/

It's worth noting that pyenv and conda have slightly different mechanics in their ideomatic usage.

pyenv installs different versions of python and then you can select which ones to use...but they're kind of global. You can't just be using one of the versions and install packages all willy-nilly and expect them to be isolated from another project using the same version. For virtual environments, you use...virtual environments just like you would if you only had one version of python installed.

(of course, most people also use pyenv-virtualenv which merges the two concepts into one)

Conda wraps environments and python versions into one tool.

Thermopyle fucked around with this message at 23:57 on Nov 4, 2019

# ? Nov 4, 2019 23:48

Dominoes: Sep 20, 2007

Obligatory spam of the project I've been posting about, since it's specifically designed to prevent problems like you describe.

You'd run pyflow init in your project folder, enter the Py version you want when it asks you. If that version's not installed, it'll download and set it up in an isolated path. To switch, run pyflow switch 3.8 as required. The concept of environments is abstracted out in favor of projects. Your existing system config and what you have installed should have no impact on how pyflow behaves.

Dominoes fucked around with this message at 06:48 on Nov 5, 2019

# ? Nov 5, 2019 03:29

mbt: Aug 13, 2012

Dominoes posted:

What happens if you replace the contents of `__init__.py` with `from .cycle_events import *` ?

only a month late but I remembered! this works! thank you!

# ? Nov 5, 2019 20:04

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Obligatory spam of the project I've been posting about, since it's specifically designed to prevent problems like you describe.

I've meant to say this a couple of times, but this reminds me of something...

README.md posted:

conda...If not, it requires falling back to Pip, which means using two separate package managers.

This is why I never, ever use conda to install packages when I'm using conda. Conda creates the python environment and then I use pip to install everything. Always one package manager that way.

# ? Nov 5, 2019 21:24

Dominoes: Sep 20, 2007

Thermopyle posted:

This is why I never, ever use conda to install packages when I'm using conda. Conda creates the python environment and then I use pip to install everything. Always one package manager that way.

This might come in handy in a moment... Using Conda to install Fenics, but it's jamming up when attempting to install Matplotlib with the conda command. Seems to work by installing Fenics with conda, and MPL with pip in the Conda env. Haven't done enough troubleshooting to figure out if this is a conflict between Fenics' dependencies and MPL's, or something else.

# ? Nov 5, 2019 21:33

cinci zoo sniper: Mar 15, 2013

Thermopyle posted:

I've meant to say this a couple of times, but this reminds me of something...

This is why I never, ever use conda to install packages when I'm using conda. Conda creates the python environment and then I use pip to install everything. Always one package manager that way.

Same, except that one time I tried installing something esoteric (honestly don�t remember what) when pip failed but conda prevailed. That was in my company�s environment so I would not be surprised to learn that it was not a pip problem.

# ? Nov 5, 2019 21:37

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Thermopyle posted:

I've meant to say this a couple of times, but this reminds me of something...

This is why I never, ever use conda to install packages when I'm using conda. Conda creates the python environment and then I use pip to install everything. Always one package manager that way.

What about stuff that depends on binaries, PATH etc that pip doesn�t handle?

# ? Nov 6, 2019 00:01

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

CarForumPoster posted:

What about stuff that depends on binaries, PATH etc that pip doesn�t handle?

Like?

It depends on the exact package, the reasons it's not pip-installable, and the other packages in the project.

# ? Nov 6, 2019 00:18

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Thermopyle posted:

Like?

It depends on the exact package, the reasons it's not pip-installable, and the other packages in the project.

Selenium/geckodriver on windows is the one i am most annoyed with so I conda install instead of pip.

Tensorflow gpu w/cuda is another one that likes to have a lot of steps but is easier conda installed.

# ? Nov 6, 2019 01:23

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

CarForumPoster posted:

Selenium/geckodriver on windows is the one i am most annoyed with so I conda install instead of pip.

Tensorflow gpu w/cuda is another one that likes to have a lot of steps but is easier conda installed.

Geckodriver and the like is best thought of as a separate piece of software like anything else that isn't python, but that python can interact with (postgres, redis, Excel, whatever). You download it, put it somewhere accessible, and you tell your projects where it's at. After you've done that, you just pip install selenium like normal.

If I'm doing a project with tensorflow gpu, and all of my projects dependencies are available via conda, then I'll use conda instead of pip.

If I'm doing a project with tensorflow gpu, all of my projects dependencies are NOT available via conda, then I'll pick either pip or conda depending on how difficult the manual install of packages not automatically installable on each are.

Of course, the whole part where you're knowing how difficult the manual install is going to be, and having the foresight to make a decent guess at which packages you're going to need going forward, is easier said then done. It requires experience.

If you don't have the requisite experience, then you might be better off using two package managers to manage your project.

Python isn't great at this whole thing...but I'm not sure a perfect story is achievable here. We want to interact with too many different technologies.

(I keep saying tensorflow, but the concept is really applicable to any difficult-to-install package.)
(I keep saying pip, but I really mean poetry, pipenv, or whatever is going to best centralize my package management)

# ? Nov 6, 2019 02:00

Dominoes: Sep 20, 2007

Thermopyle posted:

Like?

It depends on the exact package, the reasons it's not pip-installable, and the other packages in the project.

The fenics package I posted about is one example, where the easiest way to install is via Conda. There's a wheel on `pypi`, but it appears broken, and isn't documented anywhere.

# ? Nov 6, 2019 04:45

QuarkJets: Sep 8, 2008

Thermopyle posted:

I've meant to say this a couple of times, but this reminds me of something...

This is why I never, ever use conda to install packages when I'm using conda. Conda creates the python environment and then I use pip to install everything. Always one package manager that way.

I have the exact opposite position; I use conda for everything, because anything not available in the Anaconda repos is on conda-forge or some other channel.

I dislike pip because it tends to link against specific versions of libraries, so if you upgrade a library you may be breaking a pip package. This is why you should never use a system-wide pip, and it's why pip inside a conda environment creates such fragile environments. The solution to this is not to use pip for everything; your conda environment will still break easily, it will just do so when you upgrade system libraries instead of conda libraries. Using conda for everything means that your environment is always consistent, because while it also links against specific library versions it also provides those libraries.

# ? Nov 6, 2019 05:26

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

QuarkJets posted:

I have the exact opposite position; I use conda for everything, because anything not available in the Anaconda repos is on conda-forge or some other channel.

I dislike pip because it tends to link against specific versions of libraries, so if you upgrade a library you may be breaking a pip package. This is why you should never use a system-wide pip, and it's why pip inside a conda environment creates such fragile environments. The solution to this is not to use pip for everything; your conda environment will still break easily, it will just do so when you upgrade system libraries instead of conda libraries. Using conda for everything means that your environment is always consistent, because while it also links against specific library versions it also provides those libraries.

I can't say I've ever had this problem.

But then again, I pretty much gave up on conda long ago and now only use it when forced to.

# ? Nov 6, 2019 05:53

QuarkJets: Sep 8, 2008

Thermopyle posted:

I can't say I've ever had this problem.

Probably because you haven't updated any system packages that your pip-installed packages are linked against. And for most operating systems, that makes sense; whereas conda packages are constantly receiving updates, system packages tend to have a much longer update cycle. And naturally pure python packages simply won't break, since they don't link against anything

# ? Nov 6, 2019 06:03

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Dominoes posted:

The fenics package I posted about is one example, where the easiest way to install is via Conda. There's a wheel on `pypi`, but it appears broken, and isn't documented anywhere.

I wouldn't really consider FEniCS a Python package, though. It's a FEM library written in C++ that just happens to have a Python interface which is just there to translate code into C++. That said, I have never had any issues with install FEniCS via conda-forge with matplotlib. Did you ever manage to figure out what the issue was?

# ? Nov 10, 2019 13:55

SurgicalOntologist: Jun 17, 2004

I'm with QuarkJets here. At work we use the system I was previously using for personal projects: an environment.yml file, relying on conda-forge when necessary for all dependencies python or otherwise, with only the most obscure packages using pip. A nearly identical Dockerfile for everything that is basically just "conda env create". If anything goes wrong with a dev environment (usually someone who didn't understand the system or didn't read the README) it's easy to start over. And production and testing environments are built as needed so we never have to upgrade anything in place.

For an extra layer of comfort you could even delete and recreate the conda environment every time you change the dependencies instead of trying to update something. Never found this necessary though.

# ? Nov 10, 2019 14:10

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

What's people's favorite config parser that allows for env var override?

I'm spoiled by spring boots config engine which is p nice for yaml

# ? Nov 10, 2019 18:08

Dominoes: Sep 20, 2007

Boris Galerkin posted:

I wouldn't really consider FEniCS a Python package, though. It's a FEM library written in C++ that just happens to have a Python interface which is just there to translate code into C++. That said, I have never had any issues with install FEniCS via conda-forge with matplotlib. Did you ever manage to figure out what the issue was?

Good point. Didn't figure out the issue. Could troubleshoot by installing various combos of FEniCS and MPL (change the order, conda official/forge/pip), but haven't bothered. I'm suspicious it's a collision between forge FEniCS and normal-conda MPL, but that's speculation.

Tangent: I'm curious how easy it'd be to compile FEniCS as a wheel, ie installable with pip or unzipping the wheel. It's easy to think of a wheel as python code, maybe with some compiled modules, yet it can have an arbitrary ratio of compiled to python code. Eg, I've built wheels with no python code, that point to a compiled entry point/script... and can be hosted on pypi and installed with pip.

Dominoes fucked around with this message at 21:32 on Nov 10, 2019

# ? Nov 10, 2019 20:33

QuarkJets: Sep 8, 2008

Dominoes posted:

Good point. Didn't figure out the issue. Could troubleshoot by installing various combos of FEniCS and MPL (change the order, conda official/forge/pip), but haven't bothered. I'm suspicious it's a collision between forge FEniCS and normal-conda MPL, but that's speculation.

Tangent: I'm curious how easy it'd be to compile FEniCS as a wheel, ie installable with pip or unzipping the wheel.

If you install different versions of dependencies than what some package actually needs then you're going to have a bad time. Ideally the people managing these packages are supposed to specify these dependencies and their versions, but often they fail to do it correctly

In my experience the official Anaconda repos are very good about getting the versioning just right, they actually test poo poo unlike 420WeedLordGoku's conda channel. That makes the official Anaconda repos a good place to start when debugging versioning issues in a mixed-manager environment; install only the Anaconda versions of all of your packages, then test if possible, then add whatever byzantine poo poo you need from conda-forge or god-forbid pip, then test again.

Even OS distributions, which you would expect to be very carefully tested since they get released so infrequently, can produce poorly-configured environments. RHEL6 (and thus CentOS6) had a long-standing version mismatch between the version of numpy that their h5py rpm was compiled against, producing very annoying warnings and difficult to debug issues

QuarkJets fucked around with this message at 21:41 on Nov 10, 2019

# ? Nov 10, 2019 21:36

Dominoes: Sep 20, 2007

Makes sense - I bet it would be a lot of trouble if feasible at all. The manylinux specs are supposed to partly address this, where you build on a standard docker image (That runs on Centos 6, since you're more likely to have forward compatibility than backward), but I'm suspicious even that would cause troubles. This might be why that wheel I found doesn't work for me. I've encountered issues where there appears to be an OpenSSL incompatibility wall between Centos/RH etc, and newer distros, where between newer ones, there's fwd-compatibility.

I wonder what the limfac is on fenics compiled for Windows.

Dominoes fucked around with this message at 05:35 on Nov 11, 2019

# ? Nov 11, 2019 05:13

Hed: Mar 31, 2004; Fun Shoe

Malcolm XML posted:

What's people's favorite config parser that allows for env var override?

I'm spoiled by spring boots config engine which is p nice for yaml

This isn't exactly what you're talking about but for Django configs I like this one: https://github.com/joke2k/django-environ so I would look at one of the ones it is based on like envparse

# ? Nov 11, 2019 05:24

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Dominoes posted:

Makes sense - I bet it would be a lot of trouble if feasible at all. The manylinux specs are supposed to partly address this, where you build on a standard docker image (That runs on Centos 6, since you're more likely to have forward compatibility than backward), but I'm suspicious even that would cause troubles. This might be why that wheel I found doesn't work for me. I've encountered issues where there appears to be an OpenSSL incompatibility wall between Centos/RH etc, and newer distros, where between newer ones, there's fwd-compatibility.

I wonder what the limfac is on fenics compiled for Windows.

The limiting factor (limfac?) of getting fenics running on Windows is that approximately 0 out of the 7.5 billion people on the planet would use it :v:

.

Seriously though, fenics depends on a hodgepodge of scientific/numerical libraries written in C/C++/Fortran: petsc, scalapack, scotch/metis, blas/atlas/mkl, openmpi/mpich, etc. I'm sure these could all be compiled and linked together for Windows, but there is seriously no point as none of the computers that would run fenics code is running Windows and nobody wants to waste time trying to figure this out.

e: If you're trying to use this on windows, just save your sanity and use the docker container. It might sound weird or slow to do numerical computations in a docker container, but the more weird/slow part is trying to do it on windows.

Boris Galerkin fucked around with this message at 08:47 on Nov 11, 2019

# ? Nov 11, 2019 08:40