Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

Jose Cuervo: Aug 25, 2004

Bad Munki posted:

Logically, it may be clearer to find the offset of your epoch from the standard epoch and add that number of seconds to the times you have, and then convert THAT to a standard datetime object, just because then you're not calculating a known-to-be-wrong datetime at any point, but tomato tomato.

Yep, that makes more sense. Will do that.

# ? Sep 28, 2020 18:10

Adbot: ADBOT LOVES YOU

# ? May 18, 2024 05:27

NinpoEspiritoSanto: Oct 22, 2013

The pendulum library can help you with this, phone posting but the docs are pretty good.

# ? Sep 28, 2020 20:57

Dominoes: Sep 20, 2007

Pendulum uses the Monent trap of equating all types as datetimes. Your date-only arithmetic will be exposed to sneaky errors caused by manipulating times that shouldn't be involved.

# ? Sep 28, 2020 21:11

QuarkJets: Sep 8, 2008

Jose Cuervo posted:

I have a list of timestamps which represent the number of seconds since January 1st 2008. I would like to convert these into a pandas series of date times (e.g., 2020-09-27 22:13:56).

Looking through stack overflow I have found this question which uses the datetime library to convert a timestamp in seconds to a "readable" timestamp. However the datetime.datetime.fromtimestamp() function seems to be defined as seconds since January 1st 1970.

Is converting my timestamp using the datetime.datetime.fromtimestamp() as is, and then adding on the number of days between Jan 1st 1970 and Jan 1st 2008 (via pandas.DateOffset(days=num_days) where num_days is the number of days between the two dates) the best way to go about this?

Pandas can generate timestamps from a series of integers, using whatever epoch you want, interpreting them in whatever units you want (the default is nanoseconds)

Python code:

# The read-in array is "sec_from_epoch"
# Warning: I assume seconds from 00:00, but plenty of epochs use 12:00
timestamps = pd.to_datetime(sec_from_epoch, unit='s', origin="2008-01-01")

Is this what you mean? For instance passing in [10, 20] returns a DatetimeIndex(['2008-01-01 00:00:10', '2008-01-01 00:00:20'])

# ? Sep 28, 2020 21:59

Jose Cuervo: Aug 25, 2004

QuarkJets posted:

Pandas can generate timestamps from a series of integers, using whatever epoch you want, interpreting them in whatever units you want (the default is nanoseconds)
Python code:
# The read-in array is "sec_from_epoch"
# Warning: I assume seconds from 00:00, but plenty of epochs use 12:00
timestamps = pd.to_datetime(sec_from_epoch, unit='s', origin="2008-01-01")
Is this what you mean? For instance passing in [10, 20] returns a DatetimeIndex(['2008-01-01 00:00:10', '2008-01-01 00:00:20'])

Ah, excellent. That is even simpler than what I was proposing to do. Thanks for pointing this out - I will change this part of my code from what I was doing.

# ? Oct 1, 2020 14:51

dragon enthusiast: Jan 1, 2010

I'm guessing the answer to this is no, but is there a way to define a function in such a way that args are treated as kwargs instead? For example, creating a function where func(foo,bar) would cause 'foo' and 'bar' to be printed while func(peepee,doodoo,heisabadpresident) would instead print those three arguments. The closest I can think of is doing something like func(foo=_, bar=_) combined with kwargs.

# ? Oct 1, 2020 20:33

Macichne Leainig: Jul 26, 2012; by VG

I'm sure you already know about defining a function with **kwargs, but there is also *args. Those are the only ways to define variable length function parameters I know of. Not sure if plain *args covers your use case though.

# ? Oct 1, 2020 20:40

dragon enthusiast: Jan 1, 2010

Wouldn't using *args require that foo and bar be named and assigned to outside of the function?

I guess in my specific use case is an object that could contain properties obj.foo and obj.bar. I'm trying to figure out if there's any syntactic sugar to express something like obj.action(foo,bar,piss) as concisely as possible. R supports that kind of syntax for example.

dragon enthusiast fucked around with this message at 21:05 on Oct 1, 2020

# ? Oct 1, 2020 21:03

Macichne Leainig: Jul 26, 2012; by VG

Yeah, it does absolutely require that foo and bar are existing objects. You could probably fudge it with the "foo=_" kwarg thing with some kind of helper function that has *args, and pass in strings? That's the best idea I have with those constraints, sorry.

# ? Oct 1, 2020 21:20

NinpoEspiritoSanto: Oct 22, 2013

I hate to be that guy but what problem are you solving where the best solution is some loving awful function definition?

# ? Oct 2, 2020 00:37

OnceIWasAnOstrich: Jul 22, 2006

dragon enthusiast posted:

Wouldn't using *args require that foo and bar be named and assigned to outside of the function?

I guess in my specific use case is an object that could contain properties obj.foo and obj.bar. I'm trying to figure out if there's any syntactic sugar to express something like obj.action(foo,bar,piss) as concisely as possible. R supports that kind of syntax for example.

You can just iterate over the results args list and access the property from the list item without knowing its original variable name, assuming I'm understanding what you are very non-specifically trying to do.

Python code:

def fn(*args):
  for arg in args:
    print(arg.foo)

edit: Oh you want the name of the variable that you pass in as a non-keyword argument to be available in the function's scope. That is definitely not the way anyone would expect a Python function to work so you should probably use the suggestion to pass in a dictionary which is intended for that sort of name-value relationship.

OnceIWasAnOstrich fucked around with this message at 18:28 on Oct 2, 2020

# ? Oct 2, 2020 18:08

SurgicalOntologist: Jun 17, 2004

It sounds like you want different behavior depending on the name of the variables that get passed in? Don't do that, pass a dict.

# ? Oct 2, 2020 18:22

dragon enthusiast: Jan 1, 2010

for what its worth i was trying to figure out whether the people responsible for pandas picked the most cumbersome syntax possible to make people suffer or if they just didn�t bother to research usability.

Like i would love to be able to type out something like people.filter(age < 20, height > 61) instead of what i understand to be the standard of people[(people.age < 20) & (people.height > 61)]

# ? Oct 6, 2020 00:38

KICK BAMA KICK: Mar 2, 2009

dragon enthusiast posted:

for what its worth i was trying to figure out whether the people responsible for pandas picked the most cumbersome syntax possible to make people suffer or if they just didn�t bother to research usability.

Like i would love to be able to type out something like people.filter(age < 20, height > 61) instead of what i understand to be the standard of people[(people.age < 20) & (people.height > 61)]

Fair enough complaint if you can't include as many arguments as you want in a single operation but, how would they make it work so age and height by themselves would be valid references to something in-scope? In Django's ORM queries use kwargs naming fields with suffixes for the operations like People.objects.filter(age__lt=20, height__gt=61).

# ? Oct 6, 2020 00:48

QuarkJets: Sep 8, 2008

dragon enthusiast posted:

for what its worth i was trying to figure out whether the people responsible for pandas picked the most cumbersome syntax possible to make people suffer or if they just didn�t bother to research usability.

Like i would love to be able to type out something like people.filter(age < 20, height > 61) instead of what i understand to be the standard of people[(people.age < 20) & (people.height > 61)]

I don't think that's a choice made by pandas, it's how python syntax works more generally; `age` and `height` are attributes, they only become variables if you declare them as such

That said you can do what you want to do with the `query` method, you just have to use a string:

Python code:

people.query('age < 20 & height > 61')

# ? Oct 6, 2020 01:01

Dominoes: Sep 20, 2007

Assuming native Python, and people being a collection of classes that have age and height methods:

Python code:

people.filter(lambda p: p.age < 20 and p.height > 61)

The lambda syntax is notably more verbose than it could be... Perhaps on purpose!

Python code:

(p for p in people if p.age < 20 and p.height > 61)

Pandas' API's... not ideal. And it's very slow compared to the numpy arrays it wraps.

# ? Oct 6, 2020 02:27

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Dominoes posted:

Assuming native Python, and people being a collection of classes that have age and height methods:
Python code:
people.filter(lambda p: p.age < 20 and p.height > 61)
The lambda syntax is notably more verbose than it could be... Perhaps on purpose!
Python code:
(p for p in people if p.age < 20 and p.height > 61)
Pandas' API's... not ideal. And it's very slow compared to the numpy arrays it wraps.

It�s often �slow� compared to numpy but some is well optimized, for example .isin()

# ? Oct 6, 2020 04:03

dragon enthusiast: Jan 1, 2010

I�ve seen one library attempt to tackle it by creating a global variable X that contained properties of every instanced object of the class, so youd write it out as people.filter(X.age < 20, X.height > 60) but that�s disgusting.

# ? Oct 6, 2020 06:49

SurgicalOntologist: Jun 17, 2004

It's annoying but R's execution model or whatever you call it (that solves this problem) is super confusing. In R I never have it clear when or in what scope my expressions are evaluated.

# ? Oct 6, 2020 07:21

Asleep Style: Oct 20, 2010

I'm running into an issue with pytest and vscode. I have a project where I can run pytest successfully from the command line and all tests pass just fine. When I try to do test discovery from within vscode I get module import errors.

I have to set my PYTHONPATH to my projects src/ directory to get pytest to run successfully in the terminal, so I assume that's what my problems are related to. Setting PYTHONPATH in the vscode terminal doesn't seem to help.

I've verified that pytest is configured to use my tests/ folder as it's root directory within vscode.

Any suggestions? I'd like to figure this out because I have a way easier time debugging my tests from within vscode

Edit: this seems to be working now. I added a .envs file to my workspace where I set the PYTHONPATH and then added a "python.envFile" setting to my settings.json

Asleep Style fucked around with this message at 20:32 on Oct 8, 2020

# ? Oct 8, 2020 19:55

SurgicalOntologist: Jun 17, 2004

The root dir should not be tests/, it should be one level above that. That might solve your problem.

The more robust solution is to install your code (with pip or some fancy new tool I don't know) and test against the installed version of your code.

# ? Oct 8, 2020 20:57

Hollow Talk: Feb 2, 2014

Via pip install -e ., your current code gets "installed" via soft-links, which should make setting PYTHONPATH manually obsolete, while it still automatically always gets you all changes without repeated manual installs after every change.

# ? Oct 9, 2020 10:56

SurgicalOntologist: Jun 17, 2004

Getting into the weeds here, but it's more robust to test without -e, because that can hide packaging problems.

I mean, when developing an editable install is great, but it's good practice to use a regular install when doing automated testing in CI.

# ? Oct 9, 2020 17:17

Dominoes: Sep 20, 2007

The PYTHONPATH system blows. This isn't the place for hidden state. We need a new Python interpreter that doesn't use it, nor venvs, and doesn't have an assumption that Linux OSes rely on having a specific version installed. Given how long people resisted Python 3, I'm not optimistic. It's both a Linux problem and a Python problem.

I love the Python language, but groan when installation instructions for a program include Download this python project/script... Almost as much as Compile this C code. Rarely works due to assumptions about system state.

Dominoes fucked around with this message at 01:03 on Oct 10, 2020

# ? Oct 10, 2020 00:58

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Dominoes posted:

The PYTHONPATH system blows. This isn't the place for hidden state. We need a new Python interpreter that doesn't use it, nor venvs, and doesn't have an assumption that Linux OSes rely on having a specific version installed. Given how long people resisted Python 3, I'm not optimistic. It's both a Linux problem and a Python problem.

I love the Python language, but groan when installation instructions for a program include Download this python project/script... Almost as much as Compile this C code. Rarely works due to assumptions about system state.

Amen.

# ? Oct 10, 2020 01:05

NinpoEspiritoSanto: Oct 22, 2013

Yeah I love the language but gently caress me distributing remains an unsolved problem in 2020. Some interesting things are happening there still though.

# ? Oct 10, 2020 01:54

Dominoes: Sep 20, 2007

One minor, tangental* step: I'm going to move the python process manager I wrote towards only supporting the latest python minor version. I've been neglecting this project for want of spare time, but this change may simplify the codebase and API.

With that in mind: If you're using 3.6, 3.7 etc, why? Would switching to 3.8 break anything for you?

* Relevant in that it reduces a system state degree-of-freedom.

Dominoes fucked around with this message at 02:19 on Oct 10, 2020

# ? Oct 10, 2020 02:09

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Bundy posted:

Yeah I love the language but gently caress me distributing remains an unsolved problem in 2020. Some interesting things are happening there still though.

No joke I use web apps as my way of distributing any python thing.

# ? Oct 10, 2020 03:07

QuarkJets: Sep 8, 2008

Dominoes posted:

The PYTHONPATH system blows. This isn't the place for hidden state. We need a new Python interpreter that doesn't use it, nor venvs, and doesn't have an assumption that Linux OSes rely on having a specific version installed. Given how long people resisted Python 3, I'm not optimistic. It's both a Linux problem and a Python problem.

I love the Python language, but groan when installation instructions for a program include Download this python project/script... Almost as much as Compile this C code. Rarely works due to assumptions about system state.

PYTHONPATH purely exists to provide you greater, system-independent flexibility. Upgrading Python versions is trivial compared to, say, glibc, and PYTHONPATH is partly why

# ? Oct 10, 2020 04:35

OnceIWasAnOstrich: Jul 22, 2006

Dominoes posted:

One minor, tangental* step: I'm going to move the python process manager I wrote towards only supporting the latest python minor version. I've been neglecting this project for want of spare time, but this change may simplify the codebase and API.

With that in mind: If you're using 3.6, 3.7 etc, why? Would switching to 3.8 break anything for you?

* Relevant in that it reduces a system state degree-of-freedom.

Python 3 is now old enough scientific libraries have started accumulating dumb poo poo that only works with 3.5 but not newer because of dependency hell. I'm thinking of the difficulty of getting PyQT4 working with 3.6+ to use with visualization libraries that only work with PyQT4 but not PyQT5. With the exception of legacy C++ codebases like that there isn't much reason that I've run into.

# ? Oct 10, 2020 05:31

Macichne Leainig: Jul 26, 2012; by VG

I know I have TF1.15 in some projects and the wheel straight up did not exist in 3.7 or higher at the time.

Really should just upgrade to TF2 and rip that bandaid off.

# ? Oct 10, 2020 14:55

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

Dominoes posted:

One minor, tangental* step: I'm going to move the python process manager I wrote towards only supporting the latest python minor version. I've been neglecting this project for want of spare time, but this change may simplify the codebase and API.

With that in mind: If you're using 3.6, 3.7 etc, why? Would switching to 3.8 break anything for you?

* Relevant in that it reduces a system state degree-of-freedom.

I work on a project that (and obviously this isn't great) runs a robot but also is installable from pypi for local simulations of actions. While I can bump the python version on the robot all I want, I also want to keep it backwards compatible with a reasonable set of python 3 variants (where backwards compatible mostly deals with asyncio utils and typing).

# ? Oct 10, 2020 15:38

Dominoes: Sep 20, 2007

Duly noted! I'm now leaning against this idea.

Another tangent: Have any of y'all dug into the cyphon codebase? The tool I referenced above is essentially an API wrapper. I wonder how tough it would be to modify Python to use better behavior natively.

Dominoes fucked around with this message at 19:00 on Oct 10, 2020

# ? Oct 10, 2020 18:46

QuarkJets: Sep 8, 2008

Protocol7 posted:

I know I have TF1.15 in some projects and the wheel straight up did not exist in 3.7 or higher at the time.

Really should just upgrade to TF2 and rip that bandaid off.

Good news! The compat module should trivialize your transition to TF2 but yeah it's still nonzero effort. There are some significant performance and design improvements in TF2, I recommend it

# ? Oct 10, 2020 20:49

Wallet: Jun 19, 2006

Dominoes posted:

With that in mind: If you're using 3.6, 3.7 etc, why? Would switching to 3.8 break anything for you?

I'm now addicted to the walrus. I don't know if I could go back.

# ? Oct 10, 2020 22:40

Dominoes: Sep 20, 2007

Are you serious?

Let's go for a drive!

# ? Oct 10, 2020 22:43

Wallet: Jun 19, 2006

Dominoes posted:

Are you serious?

About the walrus? Always. I didn't think it would come up that much and now I use it all the time. I don't know how to Python without it anymore.

# ? Oct 10, 2020 22:45

QuarkJets: Sep 8, 2008

The walrus operator seems cool and good

# ? Oct 11, 2020 02:33

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

I want to have my cake and eat it too with pathlib. Is there some way to universally accept Windows or POSIX paths as inputs and have them properly transformed for the local platform? All of these paths are relative so I don't have to deal with drive letters. I couldn't figure out how to do it without some conditionals and actively testing the strings. I was kind of hoping for a pathlib one-or-two-liner kind of thing. I'm hoping I just missed something.

I'm doing something like this right now:

code:

        if '\\' in raw_path_str:
            p = pathlib.PureWindowsPath(raw_path_str)
        else:
            p = pathlib.PurePosixPath(raw_path_str)

        if os.path.sep == '/':
            return p.as_posix()
        else:
            return str(p)

It's not that big at all but I still think the code just kind of sucks.

# ? Oct 12, 2020 18:04

Adbot: ADBOT LOVES YOU

# ? May 18, 2024 05:27

Wallet: Jun 19, 2006

Rocko Bonaparte posted:

I want to have my cake and eat it too with pathlib. Is there some way to universally accept Windows or POSIX paths as inputs and have them properly transformed for the local platform?

I could be misremembering but I'm pretty sure pathlib handles this on its own. I think the only real reason to use PureWindowsPath or PurePosixPath is if you are trying to gently caress around with a Windows path on a POSIX machine or vice versa.

# ? Oct 12, 2020 18:21

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »