Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
TheKingofSprings
Oct 9, 2012

QuarkJets posted:

Yeah what's Python being used for here? Just launching a subprocess and nothing else? Also, there are many ways to create a subprocess, which are you using and with what options?

Python is remotely sshing into a raspberry pi and launching half a GStreamer pipeline which sends a video stream to the computer running python. It then opens up a receiver pipeline using subprocess.Popen which is where the latency is getting measured and printed to stdout. The only option I’m using is shell=True right now.

I think it’s the receiver stuff which is causing this steady increase in latency but I need to confirm by running my transmission pipeline outside python while using subprocess for the receiver still.

TheKingofSprings fucked around with this message at 01:55 on May 6, 2020

Adbot
ADBOT LOVES YOU

susan b buffering
Nov 14, 2016

TheKingofSprings posted:

Python is remotely sshing into a raspberry pi and launching half a GStreamer pipeline which sends a video stream to the computer running python. It then opens up a receiver pipeline using subprocess.Popen which is where the latency is getting measured and printed to stdout. The only option I’m using is shell=True right now.

I think it’s the receiver stuff which is causing this steady increase in latency but I need to confirm by running my transmission pipeline outside python while using subprocess for the receiver still.

It'd be helpful if you could post the code in question, or a minimal, working example. There are a lot of ways that the subprocessing module can be used that can result in deadlocking.

QuarkJets
Sep 8, 2008

TheKingofSprings posted:

Python is remotely sshing into a raspberry pi and launching half a GStreamer pipeline which sends a video stream to the computer running python. It then opens up a receiver pipeline using subprocess.Popen which is where the latency is getting measured and printed to stdout. The only option I’m using is shell=True right now.

I think it’s the receiver stuff which is causing this steady increase in latency but I need to confirm by running my transmission pipeline outside python while using subprocess for the receiver still.

Are you just invoking communicate() or wait() on one or both ends, or are you reading from Popen's stdout or stderr or piping anything from Popen to anywhere else?

Processes created with Popen do not receive fewer resources than any other process, they're exactly as though you had opened a terminal and invoked the command yourself. With shell=True it's identical to running "sh <your command here>".

QuarkJets fucked around with this message at 13:54 on May 6, 2020

Zoracle Zed
Jul 10, 2001
Anyone know if there's a library that does shell style $(foobar) string interpolation? There's so many goddamn articles about python string formatting it's impossible to search for.

necrotic
Aug 2, 2005
I owe my brother big time for this!
Python 3 has that with f strings.


code:

f'some {var}'

QuarkJets
Sep 8, 2008

necrotic posted:

Python 3 has that with f strings.


code:

f'some {var}'

e;fb!

f strings own and I don't use any other kind of string formatting unless I have a bunch of repeating args

KICK BAMA KICK
Mar 2, 2009

f-strings have cleaned up my code more than anything except maybe the walrus (I love the walrus and use it at every opportunity)

Zoracle Zed
Jul 10, 2001
hah no fstrings are great I just literally need to interpolate $ strings. yes, it is for a very stupid reason

QuarkJets
Sep 8, 2008

Zoracle Zed posted:

hah no fstrings are great I just literally need to interpolate $ strings. yes, it is for a very stupid reason

Can you elaborate? Do you mean that you have a bunch of bash-style strings in a bash script and you want Python to run those string interpolations but don't just want to run a bash subprocess to do it?

Zoracle Zed
Jul 10, 2001
I'm using htcondor on a compute cluster. It lets you submit jobs with some bespoke syntax like "input = input_$(Process).txt output=output_$(Process).txt queue 100" and it will launch 100 cpus and feed them the corresponding input and output files. There are very nice python bindings for interacting with the master condor daemon but because they're thin wrappers around the C library they're pretty low level and I'm picking away at a more pleasant high level abstraction. A problem is that all of the variable expansion is done inside the library and as far as I can tell it doesn't expose that functionality anywhere. In theory since you know most of the variable inputs and the format strings you should be able to reconstruct the expansion pretty easily but then I thought about handling nested parens and barfed a little

e: to clarify the variables that would be expanded are interior condor variables, not environmental -- so shelling out to bash wouldn't help

Zoracle Zed fucked around with this message at 05:21 on May 11, 2020

punished milkman
Dec 5, 2018

would have won

KICK BAMA KICK posted:

f-strings have cleaned up my code more than anything except maybe the walrus (I love the walrus and use it at every opportunity)

Can you show me a time where you used the walrus? I hate and resent the walrus but am willing to keep an open mind

QuarkJets
Sep 8, 2008

Zoracle Zed posted:

I'm using htcondor on a compute cluster. It lets you submit jobs with some bespoke syntax like "input = input_$(Process).txt output=output_$(Process).txt queue 100" and it will launch 100 cpus and feed them the corresponding input and output files. There are very nice python bindings for interacting with the master condor daemon but because they're thin wrappers around the C library they're pretty low level and I'm picking away at a more pleasant high level abstraction. A problem is that all of the variable expansion is done inside the library and as far as I can tell it doesn't expose that functionality anywhere. In theory since you know most of the variable inputs and the format strings you should be able to reconstruct the expansion pretty easily but then I thought about handling nested parens and barfed a little

e: to clarify the variables that would be expanded are interior condor variables, not environmental -- so shelling out to bash wouldn't help

I would be hesitant to try and duplicate the variable expansion myself in Python, if apparently it's being done inside the library; can you design your high-level interface to pass through strings without expansion and let the library do them as-necessary? I know you said that the library doesn't expose that functionality, what I mean is that you don't deal with the expanded variables at all, your high-level interface could just take care of any boilerplate issues with using the low-level wrappers.

What does htcondor do for those variable expansions? Is it all custom-coded crap or is it a commonly-used feature of the library that just isn't being exposed?

Wallet
Jun 19, 2006

punished milkman posted:

Can you show me a time where you used the walrus? I hate and resent the walrus but am willing to keep an open mind

I have come around on the walrus despite my initial apprehension though it is kind of narrow. I'd love to see more clever uses.

This is the pattern where I find myself using it most often:
Python code:
    if not (session_token := request.headers.get('Open-Session')):
        return APIError.GLOBAL_SESSION_REQUIRED
    if not (session := Sessions.verify_token(session_token)):
        return APIError.GLOBAL_INVALID_SESSION
It can also save a bunch of lines when defining things at the beginning of functions, though I'm sure some people find it less readable:
Python code:
account = create_account(client, token := verify(client, active_user := test_pool.add_user()))

Wallet fucked around with this message at 15:41 on May 11, 2020

KICK BAMA KICK
Mar 2, 2009

punished milkman posted:

Can you show me a time where you used the walrus? I hate and resent the walrus but am willing to keep an open mind
The two patterns that come to mind for me are either
Python code:

while data := read_from_an_api():
or
Python code:

if match := re.match(pattern, text):
    return match.group('foo')
when you're assigning a name to something and then checking that it's non-empty/null before working with it.

Data Graham
Dec 28, 2009

📈📊🍪😋



Every time I find myself writing

Python code:
important_shit = some_dict.get('important_shit')
if important_shit:
    do_stuff_with(important_shit)
I say to myself "boy I wish Python had a walrus operator ... oh wait"

NinpoEspiritoSanto
Oct 22, 2013




Data Graham posted:

Every time I find myself writing

Python code:
important_shit = some_dict.get('important_shit')
if important_shit:
    do_stuff_with(important_shit)
I say to myself "boy I wish Python had a walrus operator ... oh wait"

Better than me, I wrote code like that the other day and happily moved on forgetting all about the walrus

Zoracle Zed
Jul 10, 2001

QuarkJets posted:

I would be hesitant to try and duplicate the variable expansion myself in Python, if apparently it's being done inside the library; can you design your high-level interface to pass through strings without expansion and let the library do them as-necessary? I know you said that the library doesn't expose that functionality, what I mean is that you don't deal with the expanded variables at all, your high-level interface could just take care of any boilerplate issues with using the low-level wrappers.

What does htcondor do for those variable expansions? Is it all custom-coded crap or is it a commonly-used feature of the library that just isn't being exposed?

we have some function that takes too long:
Python code:
def poop(n):
    return Butt(n).poo poo()
and some input data:
Python code:
n_turds = (randint(1e6) for _ in range(100))
it'd be real convenient if we could do
Python code:
shits = map_async_somehow(poop, n_turds)
shits.wait_all()
n_total_shits = sum(s.result() for s in shits) 
what you have to do instead is
Python code:
submit_to_condor(
    executable='python -c "print(poop($n_turd))"',
    data=[{'n_turd': n, 'my_id': i} for (i, n_turd) in enumerate(n_turds)], 
    log_file='log_$(my_id).txt',
    output='output_$(my_id).txt',
)

actual_log_files = [f'log_{i}.txt' for i in range(len(n_turds))]
while any(status(f) == 'RUNNING' for f in actual_log_files ):
    time.sleep(1)

actual_output_files = [f'output_{i}.txt' for i in range(len(n_turds))]
n_total_shits = sum(int(open(f).readline()) for f in actual_output_files )
And part of abstracting away all that poo poo is doing the log_files and output files expansion. Most of the time you're just doing simple substitution so I could stop giving a poo poo and do a str.replace but in theory you can do more complicated substitutions and I figured if there's already a library to do it I might as well use it.

IAmKale
Jun 7, 2007

やらないか

Fun Shoe

KICK BAMA KICK posted:

Python code:
if match := re.match(pattern, text):
    return match.group('foo')
Huh, I've been dabbling in Swift lately and they often make use of this pattern:
code:
if let needsToBeDefined = possiblyNil {
  // Do something with needsToBeDefined knowing that it's not null
}
This means that the walrus operator makes this same pattern possible in Python? Now I don't know what the fuss is all about, I think it's pretty useful...

KICK BAMA KICK
Mar 2, 2009

Don't see an active general algorithms type thread so I'll try this here, point me in another direction if need be. Trying to keep it simple cause I'm guessing someone who knows CS or statistics will instantly recognize this as a known type of thing and the answer is just "that's a ____ problem, (you can easily solve it with ___ OR there's no good solution.)"

I have some code in the shape of
Python code:
def solution(input) -> int:
    values = [expensive_computation(input, data) for data in a_few_thousand_pieces]
    return max(values)
and I want to figure out how to use some other information I have in many cases to realize when I've already seen the maximum value and short-circuit the computations.
  • I've got ~1000 instances of the problem I know the solution to, and ~100 for which I've recorded the maximum value and second-best to quickly get a sense of the distribution of the values; can get more data along those lines if that helps
  • I'm treating the expensive_computation like a black box; there's probably something worth understanding about how it works but that's over my head right now
  • The scale for the maximum value, for all the inputs I've seen, is not very wide -- like 30s to 300s
  • The maximum value is often like 3x higher than the second-highest, but there are cases where they're very close
  • Here's the key: for a significant portion of the inputs, there's another property they have that I can use to quickly sort the data I'm comparing them against; in over half the cases I know the answer to, the correct answer is the first piece in the sorted data, and for even more it's among the first ~10, and that's just with a crude technique for finding the key to sort by that I know I can improve
  • but there are instances where the correct answer is near the end of the sorted data
  • It's no catastrophe if the solution comes out wrong but doing it the hard way is 99.9% accurate, and it's just not useful if it gets much less accurate
I figure there's some technique I can use to, for those cases with the property I can sort by, where I sort the data, run the computation against only the first ~N pieces of data, and then notice if there's a spike in the values that is probably the maximum but I do not know how to translate those words into code.

IAmKale posted:

This means that the walrus operator makes this same pattern possible in Python? Now I don't know what the fuss is all about, I think it's pretty useful...
Yeah, exactly

Wallet posted:

Python code:
account = create_account(client, token := verify(client, active_user := test_pool.add_user()))
OK this might be a bridge too far

a foolish pianist
May 6, 2007

(bi)cyclic mutation

KICK BAMA KICK posted:

Don't see an active general algorithms type thread so I'll try this here, point me in another direction if need be. Trying to keep it simple cause I'm guessing someone who knows CS or statistics will instantly recognize this as a known type of thing and the answer is just "that's a ____ problem, (you can easily solve it with ___ OR there's no good solution.)"

I have some code in the shape of
Python code:
def solution(input) -> int:
    values = [expensive_computation(input, data) for data in a_few_thousand_pieces]
    return max(values)
and I want to figure out how to use some other information I have in many cases to realize when I've already seen the maximum value and short-circuit the computations.
  • I've got ~1000 instances of the problem I know the solution to, and ~100 for which I've recorded the maximum value and second-best to quickly get a sense of the distribution of the values; can get more data along those lines if that helps
  • I'm treating the expensive_computation like a black box; there's probably something worth understanding about how it works but that's over my head right now
  • The scale for the maximum value, for all the inputs I've seen, is not very wide -- like 30s to 300s
  • The maximum value is often like 3x higher than the second-highest, but there are cases where they're very close
  • Here's the key: for a significant portion of the inputs, there's another property they have that I can use to quickly sort the data I'm comparing them against; in over half the cases I know the answer to, the correct answer is the first piece in the sorted data, and for even more it's among the first ~10, and that's just with a crude technique for finding the key to sort by that I know I can improve
  • but there are instances where the correct answer is near the end of the sorted data
  • It's no catastrophe if the solution comes out wrong but doing it the hard way is 99.9% accurate, and it's just not useful if it gets much less accurate
I figure there's some technique I can use to, for those cases with the property I can sort by, where I sort the data, run the computation against only the first ~N pieces of data, and then notice if there's a spike in the values that is probably the maximum but I do not know how to translate those words into code.

There isn't a simple way to skip your computations and always get the result correct. Without knowing more about the properties of the output data, here's what I would try:
  • Do your sorting thing, then compute the values of the first N elements of your sorted list. If you see one result that's X% larger than the mean of the results, return it. X and N will be tuning parameters for your system.
  • From here, you could go one of two ways:
    • If there's no element in the first N that's X% above the mean, continue computing through your list, computing the running mean as you go. If you see a result Y% larger than the running mean, return it. If you hit the end of the list, just return max(results)
    • If there wasn't an obvious max in the first N, just suck it up and compute the rest of the list and return the real max.

You'll have to experiment a fair bit and see whether there are values for N, X, and Y that give you acceptable results.

a foolish pianist fucked around with this message at 17:05 on May 12, 2020

Tayter Swift
Nov 18, 2002

Pillbug
I don't do a whole lot of OOP these days, so please bear with me here.

I've made a thin wrapper API around DataFrame, let's call it SDataFrame. All's well and good until I want to call a method that returns a DataFrame, such as df.copy() or df.drop(). This of course returns a DataFrame, but I want it to return an SDataFrame instead. How can I do this? I tired various flavors of overriding the calls with super() but no dice.

Macichne Leainig
Jul 26, 2012

by VG

Tayter Swift posted:

I don't do a whole lot of OOP these days, so please bear with me here.

I've made a thin wrapper API around DataFrame, let's call it SDataFrame. All's well and good until I want to call a method that returns a DataFrame, such as df.copy() or df.drop(). This of course returns a DataFrame, but I want it to return an SDataFrame instead. How can I do this? I tired various flavors of overriding the calls with super() but no dice.

I think you can pass a DataFrame into the data kwarg of the DataFrame constructor, so you might be able to just make a new instance of your subclass:

Python code:
df = SDataFrame()

# ... do stuff to df

df_as_SDataFrame = SDataFrame(data=df.copy())

Tayter Swift
Nov 18, 2002

Pillbug

Protocol7 posted:

I think you can pass a DataFrame into the data kwarg of the DataFrame constructor, so you might be able to just make a new instance of your subclass:

Python code:
df = SDataFrame()

# ... do stuff to df

df_as_SDataFrame = SDataFrame(data=df.copy())

Yeah I've essentially been doing that, but I was wondering if there's a more transparent way to do it in the class definition itself. Ideally I'd like
Python code:
df = SDataFrame()
df2 = df.copy()
To just go ahead and create an SDataFrame instance so I'm not having to keep track of which functions I have to wrap around.

KICK BAMA KICK
Mar 2, 2009

Tayter Swift posted:

To just go ahead and create an SDataFrame instance so I'm not having to keep track of which functions I have to wrap around.
Class decorator implemented kinda like the way unittest.mock.patch is?

a foolish pianist posted:

There isn't a simple way to skip your computations and always get the result correct. Without knowing more about the properties of the output data, here's what I would try:
Thanks, I'll start playing around with that.

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
I'm having some strange Windows environment thing going on. There's a few systems here that misbehave strangely with some code I recently had installed on them. It looks like the code itself is up-to-date. So I slapped PyCharm on one and tried to run it. The problem immediately went away. I then went back to the shell and tried the command again. It no longer reproduced there either. This happened after it failing over and over again.

My speculation is that PyCharm purged some cache. However, the problem showed up on a different machine where we completely removed the Python installation. I mean that we ran the uninstaller, deleted the renments of the Python folder, and then installed from fresh. My understanding in Python 3.6 is that __pycache__ should go under the directories of the modules being compiled. So that big nuke should have purged them. Is there somewhere else that this would get cached?

I'll take other wild ideas about what might have happened too.

CarForumPoster
Jun 26, 2013

⚡POWER⚡

Rocko Bonaparte posted:

I'm having some strange Windows environment thing going on.

What happens and what is supposed to happen?

unpacked robinhood
Feb 18, 2013

by Fluffdaddy
I came up with a convoluted solution to a simple task.
I'd like to gather items from the list at each key in the following structure, and add them to a new list up to a given goal size.
For example, a goal size of 10 turns this:
Python code:
articles_groups = {
    "favorite_letters": ["a","f","c","u","d","r","g"], 
    "sound":["loud","quiet"],
    "hair":[12,16,17],
}
into this:
code:
['loud', 'quiet', 12, 16, 17, 'a', 'f', 'c', 'u', 'd']
It doesn't matter if the goal size can't be met but members of each group (favorite_letters, sound and hair) must be present in the result.
Who's got a more elegant and readable solution ?.

Dominoes
Sep 20, 2007

I don't have a good grasp on the problem, but what's the format of the raw data? Is it the first code excerpt? More to the point: Your desired form is an odd use of a list, given you're mixing items that mean different things.

Dominoes fucked around with this message at 14:28 on May 27, 2020

unpacked robinhood
Feb 18, 2013

by Fluffdaddy
Key values are actually urls and the items are namedtuples instead of ints and strs but it works exactly the same.

Each url is the domain to a news website and items represent articles.

I don't keep track of the key values because they're an attribute inside the namedtuples/Article object

unpacked robinhood fucked around with this message at 14:41 on May 27, 2020

ynohtna
Feb 16, 2007

backwoods compatible
Illegal Hen

unpacked robinhood posted:

I came up with a convoluted solution to a simple task.
I'd like to gather items from the list at each key in the following structure, and add them to a new list up to a given goal size.
For example, a goal size of 10 turns this:
Python code:
articles_groups = {
    "favorite_letters": ["a","f","c","u","d","r","g"], 
    "sound":["loud","quiet"],
    "hair":[12,16,17],
}
into this:
code:
['loud', 'quiet', 12, 16, 17, 'a', 'f', 'c', 'u', 'd']
It doesn't matter if the goal size can't be met but members of each group (favorite_letters, sound and hair) must be present in the result.
Who's got a more elegant and readable solution ?.

I've probably misunderstood your needs (having a very muddled day), but maybe something like this gets you close?

Python code:
from more_itertools import take, interleave_longest

results = take(goal_size, interleave_longest(*articles_groups.values())
e: this does interleave the elements from your groups, though.

unpacked robinhood
Feb 18, 2013

by Fluffdaddy

ynohtna posted:

I've probably misunderstood your needs (having a very muddled day), but maybe something like this gets you close?
e: this does interleave the elements from your groups, though.

That seems perfect, thank you !

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!

CarForumPoster posted:

What happens and what is supposed to happen?

Oh sorry, I didn't really expect a discussion on it. I have some people that launch scripts directly from their file name on Windows. The Python launcher associates the extension to the interpreter. Somewhere in this, it is eating some of the command-line arguments. I saw sys.argv was usually missing the last flag pairing. That is a whole --this that combo, not just one element. If you invoke Python explicitly then there is no problem.

I personally hate that launcher and with Python 2 and 3 on these machines, it is a complete nightmare. So I just tell people to call out the interpreter on the shell, or preferably use something like virtualenv when juggling interpreters. However, I have to handle that case by case as they come in and tell me my code is not of the work halp. I was looking to see if later 3.6 releases had a launcher that happened to fix it, but there apparently aren't any Windows releases past 3.8.

It went away with PyCharm because it invokes the interpreter properly, and the problem stopped afterwards because I was pretending the interpreter just out of habit.

Edit: I've since started moving on to 3.8.3 and will be trying to see if the problem persists there. I kind of count on other people's environments because I'm personally not as familiar with using the Python Launcher due to my contempt for it. I can't count on personally replicating it.

QuarkJets
Sep 8, 2008

Use argparse. Check that any wrappers you've written are properly formatted, and that you're not accidentally dropping arguments. Avoid using a wrapper at all, if you can

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!

QuarkJets posted:

Use argparse. Check that any wrappers you've written are properly formatted, and that you're not accidentally dropping arguments. Avoid using a wrapper at all, if you can

I could demonstrate it with a script that just printed sys.argv without doing anything else. If I ran "script.py" it would get butchered. If I ran "python script.py" then it was fine.

QuarkJets
Sep 8, 2008

That implies a difference in the executable being used; what happens in each case if you print sys.executable? Does that match what's in your shebang? Do both of those match the output of which python ?

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!

QuarkJets posted:

That implies a difference in the executable being used; what happens in each case if you print sys.executable? Does that match what's in your shebang? Do both of those match the output of which python ?

They did match, believe it or not. It happens so much here that people mix 2.7 and 3.6 that I had a blurb in __init__.py for my code that prints a huge hate mail on the console when it determined the path to the __init__.py wasn't shared by the interpreter launching it. I figured that had happened again and my detection was broken (so I aggressively debugged that one).

Bonus points when double-clicking the file in explorer launches a completely different interpreter.

Rocko Bonaparte fucked around with this message at 04:55 on May 29, 2020

bobmarleysghost
Mar 7, 2006



Hey dudes, I'm having this dumb issue with alembic and I know I'm doing something dumb but I can't see it, hope you guys can point it out.

This is my current folder structure:


When I try to run alembic (with alembic stamp head) I get a module not found error:
code:
File "alembic/env.py", line 12, in <module>
    from application.db.models import Base
ModuleNotFoundError: No module named 'application'
and here is the relevant env.py code:
code:
from logging.config import fileConfig
from sqlalchemy import engine_from_config
from sqlalchemy import pool
from sqlalchemy import create_engine
from alembic import context

from application.db.models import Base
from application.config import Config as app_config
Even if I try import db.models or just models, I get a module not found error for each of those.

What am I doing wrong?

Cosa Nostra Aetate
Jan 1, 2019
How are you invoking? If you're setting PYTHONPATH=. and running from the root folder you show I'd think that would work.

bobmarleysghost
Mar 7, 2006



:doh: drat, that did it. thanks!

Adbot
ADBOT LOVES YOU

NinpoEspiritoSanto
Oct 22, 2013




Don't gently caress with PYTHONPATH, how are you invoking this code and where's the entry or dunder main?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply