|
QuarkJets posted:Yeah what's Python being used for here? Just launching a subprocess and nothing else? Also, there are many ways to create a subprocess, which are you using and with what options? Python is remotely sshing into a raspberry pi and launching half a GStreamer pipeline which sends a video stream to the computer running python. It then opens up a receiver pipeline using subprocess.Popen which is where the latency is getting measured and printed to stdout. The only option I’m using is shell=True right now. I think it’s the receiver stuff which is causing this steady increase in latency but I need to confirm by running my transmission pipeline outside python while using subprocess for the receiver still. TheKingofSprings fucked around with this message at 01:55 on May 6, 2020 |
# ? May 6, 2020 01:14 |
|
|
# ? May 15, 2024 03:11 |
|
TheKingofSprings posted:Python is remotely sshing into a raspberry pi and launching half a GStreamer pipeline which sends a video stream to the computer running python. It then opens up a receiver pipeline using subprocess.Popen which is where the latency is getting measured and printed to stdout. The only option Im using is shell=True right now. It'd be helpful if you could post the code in question, or a minimal, working example. There are a lot of ways that the subprocessing module can be used that can result in deadlocking.
|
# ? May 6, 2020 05:08 |
|
TheKingofSprings posted:Python is remotely sshing into a raspberry pi and launching half a GStreamer pipeline which sends a video stream to the computer running python. It then opens up a receiver pipeline using subprocess.Popen which is where the latency is getting measured and printed to stdout. The only option I’m using is shell=True right now. Are you just invoking communicate() or wait() on one or both ends, or are you reading from Popen's stdout or stderr or piping anything from Popen to anywhere else? Processes created with Popen do not receive fewer resources than any other process, they're exactly as though you had opened a terminal and invoked the command yourself. With shell=True it's identical to running "sh <your command here>". QuarkJets fucked around with this message at 13:54 on May 6, 2020 |
# ? May 6, 2020 13:50 |
|
Anyone know if there's a library that does shell style $(foobar) string interpolation? There's so many goddamn articles about python string formatting it's impossible to search for.
|
# ? May 11, 2020 03:23 |
|
Python 3 has that with f strings.code:
|
# ? May 11, 2020 03:26 |
|
necrotic posted:Python 3 has that with f strings. e;fb! f strings own and I don't use any other kind of string formatting unless I have a bunch of repeating args
|
# ? May 11, 2020 03:33 |
|
f-strings have cleaned up my code more than anything except maybe the walrus (I love the walrus and use it at every opportunity)
|
# ? May 11, 2020 03:35 |
|
hah no fstrings are great I just literally need to interpolate $ strings. yes, it is for a very stupid reason
|
# ? May 11, 2020 03:45 |
|
Zoracle Zed posted:hah no fstrings are great I just literally need to interpolate $ strings. yes, it is for a very stupid reason Can you elaborate? Do you mean that you have a bunch of bash-style strings in a bash script and you want Python to run those string interpolations but don't just want to run a bash subprocess to do it?
|
# ? May 11, 2020 05:01 |
|
I'm using htcondor on a compute cluster. It lets you submit jobs with some bespoke syntax like "input = input_$(Process).txt output=output_$(Process).txt queue 100" and it will launch 100 cpus and feed them the corresponding input and output files. There are very nice python bindings for interacting with the master condor daemon but because they're thin wrappers around the C library they're pretty low level and I'm picking away at a more pleasant high level abstraction. A problem is that all of the variable expansion is done inside the library and as far as I can tell it doesn't expose that functionality anywhere. In theory since you know most of the variable inputs and the format strings you should be able to reconstruct the expansion pretty easily but then I thought about handling nested parens and barfed a little e: to clarify the variables that would be expanded are interior condor variables, not environmental -- so shelling out to bash wouldn't help Zoracle Zed fucked around with this message at 05:21 on May 11, 2020 |
# ? May 11, 2020 05:17 |
|
KICK BAMA KICK posted:f-strings have cleaned up my code more than anything except maybe the walrus (I love the walrus and use it at every opportunity) Can you show me a time where you used the walrus? I hate and resent the walrus but am willing to keep an open mind
|
# ? May 11, 2020 05:19 |
|
Zoracle Zed posted:I'm using htcondor on a compute cluster. It lets you submit jobs with some bespoke syntax like "input = input_$(Process).txt output=output_$(Process).txt queue 100" and it will launch 100 cpus and feed them the corresponding input and output files. There are very nice python bindings for interacting with the master condor daemon but because they're thin wrappers around the C library they're pretty low level and I'm picking away at a more pleasant high level abstraction. A problem is that all of the variable expansion is done inside the library and as far as I can tell it doesn't expose that functionality anywhere. In theory since you know most of the variable inputs and the format strings you should be able to reconstruct the expansion pretty easily but then I thought about handling nested parens and barfed a little I would be hesitant to try and duplicate the variable expansion myself in Python, if apparently it's being done inside the library; can you design your high-level interface to pass through strings without expansion and let the library do them as-necessary? I know you said that the library doesn't expose that functionality, what I mean is that you don't deal with the expanded variables at all, your high-level interface could just take care of any boilerplate issues with using the low-level wrappers. What does htcondor do for those variable expansions? Is it all custom-coded crap or is it a commonly-used feature of the library that just isn't being exposed?
|
# ? May 11, 2020 05:33 |
|
punished milkman posted:Can you show me a time where you used the walrus? I hate and resent the walrus but am willing to keep an open mind I have come around on the walrus despite my initial apprehension though it is kind of narrow. I'd love to see more clever uses. This is the pattern where I find myself using it most often: Python code:
Python code:
Wallet fucked around with this message at 15:41 on May 11, 2020 |
# ? May 11, 2020 13:39 |
|
punished milkman posted:Can you show me a time where you used the walrus? I hate and resent the walrus but am willing to keep an open mind Python code:
Python code:
|
# ? May 11, 2020 14:15 |
Every time I find myself writingPython code:
|
|
# ? May 11, 2020 15:11 |
Data Graham posted:Every time I find myself writing Better than me, I wrote code like that the other day and happily moved on forgetting all about the walrus
|
|
# ? May 11, 2020 15:52 |
|
QuarkJets posted:I would be hesitant to try and duplicate the variable expansion myself in Python, if apparently it's being done inside the library; can you design your high-level interface to pass through strings without expansion and let the library do them as-necessary? I know you said that the library doesn't expose that functionality, what I mean is that you don't deal with the expanded variables at all, your high-level interface could just take care of any boilerplate issues with using the low-level wrappers. we have some function that takes too long: Python code:
Python code:
Python code:
Python code:
|
# ? May 11, 2020 16:19 |
|
KICK BAMA KICK posted:
code:
|
# ? May 11, 2020 19:56 |
|
Don't see an active general algorithms type thread so I'll try this here, point me in another direction if need be. Trying to keep it simple cause I'm guessing someone who knows CS or statistics will instantly recognize this as a known type of thing and the answer is just "that's a ____ problem, (you can easily solve it with ___ OR there's no good solution.)" I have some code in the shape of Python code:
IAmKale posted:This means that the walrus operator makes this same pattern possible in Python? Now I don't know what the fuss is all about, I think it's pretty useful... Wallet posted:
|
# ? May 12, 2020 02:57 |
KICK BAMA KICK posted:Don't see an active general algorithms type thread so I'll try this here, point me in another direction if need be. Trying to keep it simple cause I'm guessing someone who knows CS or statistics will instantly recognize this as a known type of thing and the answer is just "that's a ____ problem, (you can easily solve it with ___ OR there's no good solution.)" There isn't a simple way to skip your computations and always get the result correct. Without knowing more about the properties of the output data, here's what I would try:
You'll have to experiment a fair bit and see whether there are values for N, X, and Y that give you acceptable results. a foolish pianist fucked around with this message at 17:05 on May 12, 2020 |
|
# ? May 12, 2020 14:29 |
|
I don't do a whole lot of OOP these days, so please bear with me here. I've made a thin wrapper API around DataFrame, let's call it SDataFrame. All's well and good until I want to call a method that returns a DataFrame, such as df.copy() or df.drop(). This of course returns a DataFrame, but I want it to return an SDataFrame instead. How can I do this? I tired various flavors of overriding the calls with super() but no dice.
|
# ? May 12, 2020 21:17 |
|
Tayter Swift posted:I don't do a whole lot of OOP these days, so please bear with me here. I think you can pass a DataFrame into the data kwarg of the DataFrame constructor, so you might be able to just make a new instance of your subclass: Python code:
|
# ? May 12, 2020 21:50 |
|
Protocol7 posted:I think you can pass a DataFrame into the data kwarg of the DataFrame constructor, so you might be able to just make a new instance of your subclass: Yeah I've essentially been doing that, but I was wondering if there's a more transparent way to do it in the class definition itself. Ideally I'd like Python code:
|
# ? May 12, 2020 22:50 |
|
Tayter Swift posted:To just go ahead and create an SDataFrame instance so I'm not having to keep track of which functions I have to wrap around. a foolish pianist posted:There isn't a simple way to skip your computations and always get the result correct. Without knowing more about the properties of the output data, here's what I would try:
|
# ? May 13, 2020 00:34 |
|
I'm having some strange Windows environment thing going on. There's a few systems here that misbehave strangely with some code I recently had installed on them. It looks like the code itself is up-to-date. So I slapped PyCharm on one and tried to run it. The problem immediately went away. I then went back to the shell and tried the command again. It no longer reproduced there either. This happened after it failing over and over again. My speculation is that PyCharm purged some cache. However, the problem showed up on a different machine where we completely removed the Python installation. I mean that we ran the uninstaller, deleted the renments of the Python folder, and then installed from fresh. My understanding in Python 3.6 is that __pycache__ should go under the directories of the modules being compiled. So that big nuke should have purged them. Is there somewhere else that this would get cached? I'll take other wild ideas about what might have happened too.
|
# ? May 15, 2020 20:06 |
|
Rocko Bonaparte posted:I'm having some strange Windows environment thing going on. What happens and what is supposed to happen?
|
# ? May 16, 2020 02:13 |
|
I came up with a convoluted solution to a simple task. I'd like to gather items from the list at each key in the following structure, and add them to a new list up to a given goal size. For example, a goal size of 10 turns this: Python code:
code:
Who's got a more elegant and readable solution ?.
|
# ? May 27, 2020 14:23 |
|
I don't have a good grasp on the problem, but what's the format of the raw data? Is it the first code excerpt? More to the point: Your desired form is an odd use of a list, given you're mixing items that mean different things.
Dominoes fucked around with this message at 14:28 on May 27, 2020 |
# ? May 27, 2020 14:26 |
|
Key values are actually urls and the items are namedtuples instead of ints and strs but it works exactly the same. Each url is the domain to a news website and items represent articles. I don't keep track of the key values because they're an attribute inside the namedtuples/Article object unpacked robinhood fucked around with this message at 14:41 on May 27, 2020 |
# ? May 27, 2020 14:30 |
|
unpacked robinhood posted:I came up with a convoluted solution to a simple task. I've probably misunderstood your needs (having a very muddled day), but maybe something like this gets you close? Python code:
|
# ? May 27, 2020 14:49 |
|
ynohtna posted:I've probably misunderstood your needs (having a very muddled day), but maybe something like this gets you close? That seems perfect, thank you !
|
# ? May 27, 2020 15:00 |
|
CarForumPoster posted:What happens and what is supposed to happen? Oh sorry, I didn't really expect a discussion on it. I have some people that launch scripts directly from their file name on Windows. The Python launcher associates the extension to the interpreter. Somewhere in this, it is eating some of the command-line arguments. I saw sys.argv was usually missing the last flag pairing. That is a whole --this that combo, not just one element. If you invoke Python explicitly then there is no problem. I personally hate that launcher and with Python 2 and 3 on these machines, it is a complete nightmare. So I just tell people to call out the interpreter on the shell, or preferably use something like virtualenv when juggling interpreters. However, I have to handle that case by case as they come in and tell me my code is not of the work halp. I was looking to see if later 3.6 releases had a launcher that happened to fix it, but there apparently aren't any Windows releases past 3.8. It went away with PyCharm because it invokes the interpreter properly, and the problem stopped afterwards because I was pretending the interpreter just out of habit. Edit: I've since started moving on to 3.8.3 and will be trying to see if the problem persists there. I kind of count on other people's environments because I'm personally not as familiar with using the Python Launcher due to my contempt for it. I can't count on personally replicating it.
|
# ? May 28, 2020 05:13 |
|
Use argparse. Check that any wrappers you've written are properly formatted, and that you're not accidentally dropping arguments. Avoid using a wrapper at all, if you can
|
# ? May 28, 2020 06:16 |
|
QuarkJets posted:Use argparse. Check that any wrappers you've written are properly formatted, and that you're not accidentally dropping arguments. Avoid using a wrapper at all, if you can I could demonstrate it with a script that just printed sys.argv without doing anything else. If I ran "script.py" it would get butchered. If I ran "python script.py" then it was fine.
|
# ? May 28, 2020 07:33 |
|
That implies a difference in the executable being used; what happens in each case if you print sys.executable? Does that match what's in your shebang? Do both of those match the output of which python ?
|
# ? May 28, 2020 22:56 |
|
QuarkJets posted:That implies a difference in the executable being used; what happens in each case if you print sys.executable? Does that match what's in your shebang? Do both of those match the output of which python ? They did match, believe it or not. It happens so much here that people mix 2.7 and 3.6 that I had a blurb in __init__.py for my code that prints a huge hate mail on the console when it determined the path to the __init__.py wasn't shared by the interpreter launching it. I figured that had happened again and my detection was broken (so I aggressively debugged that one). Bonus points when double-clicking the file in explorer launches a completely different interpreter. Rocko Bonaparte fucked around with this message at 04:55 on May 29, 2020 |
# ? May 29, 2020 04:51 |
|
Hey dudes, I'm having this dumb issue with alembic and I know I'm doing something dumb but I can't see it, hope you guys can point it out. This is my current folder structure: When I try to run alembic (with alembic stamp head) I get a module not found error: code:
code:
What am I doing wrong?
|
# ? May 29, 2020 15:26 |
How are you invoking? If you're setting PYTHONPATH=. and running from the root folder you show I'd think that would work.
|
|
# ? May 29, 2020 15:36 |
|
drat, that did it. thanks!
|
# ? May 29, 2020 16:22 |
|
|
# ? May 15, 2024 03:11 |
Don't gently caress with PYTHONPATH, how are you invoking this code and where's the entry or dunder main?
|
|
# ? May 29, 2020 23:32 |