Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
lazerwolf
Dec 22, 2009

Orange and Black

hbag posted:

man
im really not sure if this should go in this thread or the HTML thread but uhh

i'm making a REST API in flask, ive got some of this poo poo set up:
Python code:
@app.route('/flags/user/<string:search_user>', methods=['GET','POST'])
def userflags(search_user):
    return userflags_actual(search_user)
works great, but ive got a search form on one of the pages, and i want it to navigate to the appropriate page when they submit, obviously
so for example if someone were to search 'hbag', it'd take them to '/flags/users/hbag'

but since it's a HTML form, i have no idea how to get around that, since adding an 'onsubmit' function seems to just run that function and THEN submit the form regularly anyway

Can you post your html?

Generally buttons within forms will try to submit the form on click in addition to any action you tell it to do onClick as you've stated.

If you give the button type="button" it won't submit the form
code:
<button type="button" onClick=do-my-on-click-REST-call>I don't submit forms</button>

Adbot
ADBOT LOVES YOU

Mr. Bubbles
Jul 19, 2012
I'd like to get some ideas on how I would go about creating a python application that handles a fantasy football style vacation lottery draft. Any recommendations of librarys or example scripts that accomplish similar tasks? I'm not finding much available and having trouble brainstorming how to go about with the fundamentals of how the program would work.

The basics I'm imagining is a list of people in a specific order. Person 1 through 50 take turns selecting a day or set of days for vacation with some limitations. There are a cap of # of vacation slots available per calendar day and the program keeps track of availability, shows a calendar with days either green, yellow, or red depending on # slots still available.

Any thoughts on where I would start?

Deadite
Aug 30, 2003

A fat guy, a watermelon, and a stack of magazines?
Family.
I'm having a problem figuring out a regex and hopefully someone can point to my mistake. Here's the test case:

code:
import re
re.findall(r"\bX_[RR|FF]?[_]?D", 'X_FF_D X_RR_D X_D')
My desired result is a list with all of the words in the string (['X_FF_D', 'X_RR_D', 'X_D']) but I can't figure out how to structure the or for RR|FF to return what I want. It doesn't seem to like multiple characters the way I have it now.

The code works when it's just one character:
code:
import re
re.findall(r"\bX_[R|F]?[_]?D", 'X_F_D X_R_D X_D')

HappyHippo
Nov 19, 2003
Do you have an Air Miles Card?
I think your issue is you're using [ ] instead of (). [ ] indicates a set of characters. Like [abc] matches a or b or c. So you're asking for "R or R or | or F or F" with that regex.
https://docs.python.org/3/library/re.html

Macichne Leainig
Jul 26, 2012

by VG
Someone much better at regex than I should hopefully be able to come up with a better solution, but this works:


Python code:
>>> import re
>>> [x for x, _ in re.findall(r"\b(X_(RR|FF)?[_]?D)", 'X_FF_D X_RR_D X_D')]
['X_FF_D', 'X_RR_D', 'X_D']

Deadite
Aug 30, 2003

A fat guy, a watermelon, and a stack of magazines?
Family.
Thanks to you both, I'm not great with groups and didn't realize I needed an outer group along with the inner group to return what I wanted

Wallet
Jun 19, 2006

Protocol7 posted:

Someone much better at regex than I should hopefully be able to come up with a better solution, but this works:


Python code:
>>> import re
>>> [x for x, _ in re.findall(r"\b(X_(RR|FF)?[_]?D)", 'X_FF_D X_RR_D X_D')]
['X_FF_D', 'X_RR_D', 'X_D']

Groups are how you bound alternation in regex—there's nothing wrong with this. If the posted example is actually the full group of things they want to capture then (RR_|FF_) is more precise (since it would otherwise match X__D etc). That aside there's no reason I'm aware of to have an underscore on its own in a character set.

Mr. Bubbles posted:

I'd like to get some ideas on how I would go about creating a python application that handles a fantasy football style vacation lottery draft. Any recommendations of librarys or example scripts that accomplish similar tasks? I'm not finding much available and having trouble brainstorming how to go about with the fundamentals of how the program would work.

The basics I'm imagining is a list of people in a specific order. Person 1 through 50 take turns selecting a day or set of days for vacation with some limitations. There are a cap of # of vacation slots available per calendar day and the program keeps track of availability, shows a calendar with days either green, yellow, or red depending on # slots still available.

Any thoughts on where I would start?

I think you probably aren't getting much response to this because it's extremely open ended. Are you asking for help on figuring out what framework to use for a UI for this to run as a program on a single computer because all of these people are literally going to take turns with the same keyboard/mouse or are you trying to make a website to do this or?

Wallet fucked around with this message at 23:30 on Dec 8, 2021

Mr. Bubbles
Jul 19, 2012
Yeah, that's ok. Not looking for code per say, but a framework of some ideas on how to organize it. Or even an existing program that has some similar features so I can get ideas.

Starting with a single program and may change to a web app in the future.

It's ok if no one has ideas, I'll just start coding part by part and figure it out!

For instance, one way I was thinking was having an array of days of the year, 1-365. In the array I could include data from a library like time board that contains day of week / month info in order to facilitate display in a month calendar view. Also the array would contain number of "slots" remaining.

I would iterate through a loop to emulate the draft and order choice. An array of users would contain a user ID# and a list of dates they selected. When selected, the slot number would decrease by one.

Something like that, but also interested in other ways to organize the program / design the methods to store this data.

Mr. Bubbles fucked around with this message at 01:29 on Dec 9, 2021

Wallet
Jun 19, 2006

Mr. Bubbles posted:

Something like that, but also interested in other ways to organize the program / design the methods to store this data.

If the days and the users are simple objects you can assign the user objects to the slots on the days instead of just counting which will let you access the information from either end and probably make your life easier.

Fruits of the sea
Dec 1, 2010

Any suggestions for resources if I want to get proficient with Pandas? I’ve mostly been following the official documentation so far, but it seems more useful for looking up methods as opposed to actually implementing them in a smart way.

cinci zoo sniper
Mar 15, 2013




Fruits of the sea posted:

Any suggestions for resources if I want to get proficient with Pandas? I’ve mostly been following the official documentation so far, but it seems more useful for looking up methods as opposed to actually implementing them in a smart way.

Just time in the trenches. Though it won’t hurt to read this blog from cover to cover - https://tomaugspurger.github.io/ - especially the Modern Pandas series.

Deadite
Aug 30, 2003

A fat guy, a watermelon, and a stack of magazines?
Family.
Following along with this video helped me immensely when I was starting out with pandas

https://youtu.be/5JnMutdy6Fw

a dingus
Mar 22, 2008

Rhetorical questions only
Fun Shoe
Datacamp helped me with pandas quite a bit years ago. It's like free code camp style learning but it costs like $30 a month.

Fruits of the sea
Dec 1, 2010

Thanks for all the suggestions, it’ll give me something to work on over the holidays.

Twerk from Home
Jan 17, 2009

This avatar brought to you by the 'save our dead gay forums' foundation.
I could use a hand figuring out where setuptools gets default configuration from. I am failing to pip install uwsgi on an Apple mac in a Python 3.9.9 venv.

I have an Apple Silicon Mac that has conda installed for some projects, but others are managed with pip and venv. I am able to create a venv entirely outside of anaconda by calling a non-anaconda python executable that was installed with pyenv to create the venv, which I'm then using pip to install libraries with.

The problem that I'm having for some packages is that for some reason, setuptools within the venv is trying to link against the Anaconda library path, which doesn't have arm64 libraries, like this:

pre:
/opt/anaconda3/lib/libz.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
When I see the pip log, everything is running in the venv appropriately except for the link step, which is calling the linker with the below, but I have no idea where any configuration would be that's causing Python to look in the anaconda path for this, because everything should be either in the virtual environment or linking against system libraries.

It's invoking the linker with:
pre:
-L/Users/me/opt/anaconda3/lib

QuarkJets
Sep 8, 2008

When you say "installed with pyenv", could that have come from an anaconda environment?

I don't know much about Mac but I know that it's *nix-y. In *nix your LD_LIBRARY_PATH dictates what libraries are visible to you. Likewise, PATH shows access to directories with executables in them. Check your LD_LIBRARY_PATH and PATH and see if you have an anaconda3 path in there. Then recursively grep for anaconda3 in the directory that holds your venv and see what comes up.

Or just skip doing any of that poo poo and instead grab a mambaforge installer and create a conda environment wherever you need for whatever packages you want, abandoning whatever janky workflow has led to this situation.

QuarkJets
Sep 8, 2008

PSA: This is not directed at the last poster, I just want people to be aware of mamba. Stop using conda, use mamba instead.

SurgicalOntologist
Jun 17, 2004

That's awesome, how did I miss mamba for so long! My team will be very pleased to have faster builds.

QuarkJets
Sep 8, 2008

Some of my environments mamba sets up in seconds whereas conda can't even resolve them. Coworkers will get frustrated with their environment build time and I'll be like "you used conda, what is this 2018???"

Macichne Leainig
Jul 26, 2012

by VG
Thank you friendly Python goons for introducing me to yet another tool my apparently archaic rear end hasn't even heard of yet.

Twerk from Home
Jan 17, 2009

This avatar brought to you by the 'save our dead gay forums' foundation.

QuarkJets posted:

When you say "installed with pyenv", could that have come from an anaconda environment?

I don't know much about Mac but I know that it's *nix-y. In *nix your LD_LIBRARY_PATH dictates what libraries are visible to you. Likewise, PATH shows access to directories with executables in them. Check your LD_LIBRARY_PATH and PATH and see if you have an anaconda3 path in there. Then recursively grep for anaconda3 in the directory that holds your venv and see what comes up.

Or just skip doing any of that poo poo and instead grab a mambaforge installer and create a conda environment wherever you need for whatever packages you want, abandoning whatever janky workflow has led to this situation.

My continued debugging efforts are below, but I've got a quick question too: How do people do local devlopment of a celery application given that celery is a separate process? I've got a local devleopment start script, but this feels pretty janky and I bet there's a better way to do this:

pre:
#!/bin/sh

export FLASK_ENV=development

celery -A app.celery worker &
flask run

I really appreciate the suggestion and am checking out Mamba, but I'm still stuck at this same failure state. Should I just give up at using pip and only use packages offered through the conda infrastructure? I'm only using pip for this one because I'm much more familiar with doing CI / deployment of web services using pip with requirements.txt.

My full setup is:
  • pyenv installed from brew
  • conda installed with anaconda desktop distribution initially, now trying mamba
  • Conda environments used for stuff the scientists give me
  • Python versions installed with pyenv used for things I want to

To create a virtual environment, I'm directly invoking a Python version installed with pyenv like this:
pre:
 ~/.pyenv/versions/3.9.9/bin/python -m venv env
LD_LIBRARY_PATH is not set at all. I think that this isn't a problem with LD_LIBRARY_PATH because when I run pip install uwsgi or pip install -r requirements.txt, I see the below, which is why I came to the python thread. Pip trying to install it results in it compiling from source, where we see clang specifically being called with: -L/Users/me/opt/anaconda3/lib

I have no idea where that is coming from, but I feel like it must be coming from the Python setuptools, because if the command were run

pre:
    *** uWSGI linking ***
    clang -o /Users/me/src/me/slack-deleter-pro/env/bin/uwsgi  core/utils.o core/protocol.o core/socket.o core/logging.o core/master.o core/master_utils.o core/emperor.o core/notify.o core/mule.o core/subscription.o core/stats.o core/sendfile.o core/async.o core/master_checks.o core/fifo.o core/offload.o core/io.o core/static.o core/websockets.o core/spooler.o core/snmp.o core/exceptions.o core/config.o core/setup_utils.o core/clock.o core/init.o core/buffer.o core/reader.o core/writer.o core/alarm.o core/cron.o core/hooks.o core/plugins.o core/lock.o core/cache.o core/daemons.o core/errors.o core/hash.o core/master_events.o core/chunked.o core/queue.o core/event.o core/signal.o core/strings.o core/progress.o core/timebomb.o core/ini.o core/fsmon.o core/mount.o core/metrics.o core/plugins_builder.o core/sharedarea.o core/rpc.o core/gateway.o core/loop.o core/cookie.o core/querystring.o core/rb_timers.o core/transformations.o core/uwsgi.o proto/base.o proto/uwsgi.o proto/http.o proto/fastcgi.o proto/scgi.o proto/puwsgi.o core/zlib.o core/regexp.o core/routing.o core/yaml.o core/xmlconf.o core/dot_h.o core/config_py.o plugins/python/python_plugin.o plugins/python/pyutils.o plugins/python/pyloader.o plugins/python/wsgi_handlers.o plugins/python/wsgi_headers.o plugins/python/wsgi_subhandler.o plugins/python/web3_subhandler.o plugins/python/pump_subhandler.o plugins/python/gil.o plugins/python/uwsgi_pymodule.o plugins/python/profiler.o plugins/python/symimporter.o plugins/python/tracebacker.o plugins/python/raw.o plugins/gevent/gevent.o plugins/gevent/hooks.o plugins/ping/ping_plugin.o plugins/cache/cache.o plugins/nagios/nagios.o plugins/rrdtool/rrdtool.o plugins/carbon/carbon.o plugins/rpc/rpc_plugin.o plugins/corerouter/cr_common.o plugins/corerouter/cr_map.o plugins/corerouter/corerouter.o plugins/fastrouter/fastrouter.o plugins/http/http.o plugins/http/keepalive.o plugins/http/https.o plugins/http/spdy3.o plugins/signal/signal_plugin.o plugins/syslog/syslog_plugin.o plugins/rsyslog/rsyslog_plugin.o plugins/logsocket/logsocket_plugin.o plugins/router_uwsgi/router_uwsgi.o plugins/router_redirect/router_redirect.o plugins/router_basicauth/router_basicauth.o plugins/zergpool/zergpool.o plugins/redislog/redislog_plugin.o plugins/mongodblog/mongodblog_plugin.o plugins/router_rewrite/router_rewrite.o plugins/router_http/router_http.o plugins/logfile/logfile.o plugins/router_cache/router_cache.o plugins/rawrouter/rawrouter.o plugins/router_static/router_static.o plugins/sslrouter/sslrouter.o plugins/spooler/spooler_plugin.o plugins/cheaper_busyness/cheaper_busyness.o plugins/symcall/symcall_plugin.o plugins/transformation_tofile/tofile.o plugins/transformation_gzip/gzip.o plugins/transformation_chunked/chunked.o plugins/transformation_offload/offload.o plugins/router_memcached/router_memcached.o plugins/router_redis/router_redis.o plugins/router_hash/router_hash.o plugins/router_expires/expires.o plugins/router_metrics/plugin.o plugins/transformation_template/tt.o plugins/stats_pusher_socket/plugin.o -lpthread -lm -lz -L/Users/me/opt/anaconda3/lib -lpcre -lexpat -lintl -ldl -framework CoreFoundation /Users/me/.pyenv/versions/3.9.9/lib/python3.9/config-3.9-darwin/libpython3.9.a
    ld: warning: ignoring file /Users/me/opt/anaconda3/lib/libz.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    ld: warning: ignoring file /Users/me/opt/anaconda3/lib/libpcre.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    ld: warning: ignoring file /Users/me/opt/anaconda3/lib/libexpat.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    ld: warning: ignoring file /Users/me/opt/anaconda3/lib/libintl.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    Undefined symbols for architecture arm64:
      "_XML_ErrorString", referenced from:
          _uwsgi_xml_config in xmlconf.o
      "_XML_GetCurrentLineNumber", referenced from:
          _uwsgi_xml_config in xmlconf.o

I feel like I'm getting so, so much closer. The libraries that I need are in /opt/homebrew/lib, because I've installed them or something that uses them with Homebrew at some point. I ended up working around this by installing uwsgi from homebrew instead of using pip, but I am worried this issue will crop up again in the future whenever a package needs to link against native libraries again.

This would be solved if anaconda would distribute arm64 libraries or universal libraries instead of x86 only, then it would be able to link against the libraries that anaconda has.

necrotic
Aug 2, 2005
I owe my brother big time for this!
Foreman, or one of the ports if you don’t want to use the ruby version, is great for running multiple processes locally easily.

https://github.com/ddollar/foreman

12 rats tied together
Sep 7, 2006

the usual python answer here tends to be supervisord, in my experience, which might also be a port of foreman, i have never checked

for my personal projects i have some tasks defined in vs code for starting or bumping services, in production i just use systemd though

QuarkJets
Sep 8, 2008

Twerk from Home posted:

I really appreciate the suggestion and am checking out Mamba, but I'm still stuck at this same failure state. Should I just give up at using pip and only use packages offered through the conda infrastructure? I'm only using pip for this one because I'm much more familiar with doing CI / deployment of web services using pip with requirements.txt.

My full setup is:
  • pyenv installed from brew
  • conda installed with anaconda desktop distribution initially, now trying mamba
  • Conda environments used for stuff the scientists give me
  • Python versions installed with pyenv used for things I want to

To create a virtual environment, I'm directly invoking a Python version installed with pyenv like this:
pre:
 ~/.pyenv/versions/3.9.9/bin/python -m venv env

I think that you should try using mamba to create the environment instead of pyenv. Then activate the environment and use "mamba install X" or "pip install Y" to install things. You should not touch any other version of pip or conda than what exists from within your environment

Or define a yaml file defining the specific packages and version requirements, and create the entire environment with "mamba env create" while specifying the yaml file. You can put a pip block in conda yaml files, anything within the block will be pip installed, everything else will be mamba installed.

I think doing the above should solve this problem and let you pip install that package, but the specific problem was that pip couldn't build a wheel, right? Conda/mamba use prebuilts, so just using mamba to install whatever package you're stuck on is also something you can try. Personally I don't use pip for anything unless I absolutely have to

Twerk from Home
Jan 17, 2009

This avatar brought to you by the 'save our dead gay forums' foundation.

QuarkJets posted:

I think that you should try using mamba to create the environment instead of pyenv. Then activate the environment and use "mamba install X" or "pip install Y" to install things. You should not touch any other version of pip or conda than what exists from within your environment

Or define a yaml file defining the specific packages and version requirements, and create the entire environment with "mamba env create" while specifying the yaml file. You can put a pip block in conda yaml files, anything within the block will be pip installed, everything else will be mamba installed.

I think doing the above should solve this problem and let you pip install that package, but the specific problem was that pip couldn't build a wheel, right? Conda/mamba use prebuilts, so just using mamba to install whatever package you're stuck on is also something you can try. Personally I don't use pip for anything unless I absolutely have to

Thanks, that was actually the first thing that I tried, mamba / conda only install x86 binaries / libraries, and I was encountering another problem at runtime rather than build-time because it was trying to link conda-managed x86 binaries against system-managed arm64 libraries, which was failing too.

I was using pip because pip is able to load / link arm64 binaries, libraries, and python runtimes, which conda / mamba don't seem to do at all. I'll try just using uwsgi from conda-forge again: https://anaconda.org/conda-forge/uwsgi

former glory
Jul 11, 2011

Twerk from Home posted:

Thanks, that was actually the first thing that I tried, mamba / conda only install x86 binaries / libraries, and I was encountering another problem at runtime rather than build-time because it was trying to link conda-managed x86 binaries against system-managed arm64 libraries, which was failing too.

I was using pip because pip is able to load / link arm64 binaries, libraries, and python runtimes, which conda / mamba don't seem to do at all. I'll try just using uwsgi from conda-forge again: https://anaconda.org/conda-forge/uwsgi

I'm not familiar with all the stuff in your OP, but if it helps, I used Miniforge to get tensorflow working on my m1, and I was able to subsequently get matplotlib and CV2 installed no problem via pip within that miniforge env, which was giving me a bunch of issues wrt linking x86 with ARM before. I followed something like this to get it going:

https://betterprogramming.pub/installing-tensorflow-on-apple-m1-with-new-metal-plugin-6d3cb9cb00ca

It solved so many problems that I even pip installed all my flask deps into the same venv without any trouble just to keep it simple, as I don't dev much on this thing.

Macichne Leainig
Jul 26, 2012

by VG
Does Tensorflow actually get accelerated by the M1's hardware in any way, or is utilizing the GPU and Neural Engine still a few months out?

cinci zoo sniper
Mar 15, 2013




Protocol7 posted:

Does Tensorflow actually get accelerated by the M1's hardware in any way, or is utilizing the GPU and Neural Engine still a few months out?

Not out of box, but there's, for example, this.

bolind
Jun 19, 2005



Pillbug
I have a program which basically uses multiprocessing to create a pool of workers (one per CPU core, 36 in this instance.), and each worker calls an external program via subprocess.call(), such that they run in parallel. The external program is an unholy command of the form:

code:
program --option >> foo.txt &&
program --anotheroption >> foo.txt &&
program --thirdoption >> foo.txt &&
...
program --makeitstopoption >> foo.txt &&
There can easily be over a dozen of these lines.

The problem is that it runs slow, it's like the completion time for each worker (the tasks should be roughly identical in complexity.) goes up exponentially. And there's no rhyme or reason to which tasks complete fast and which take a long time:

code:
Total number of tasks: 24
Task 000: 0:00:01.744050 
Task 012: 0:00:02.713520
Task 009: 0:00:03.759242
Task 022: 0:00:03.777296
Task 014: 0:00:03.781031
Task 005: 0:00:04.774287
Task 019: 0:00:04.795411
Task 007: 0:00:05.791027
Task 023: 0:00:05.816474
Task 016: 0:00:06.801034
Task 020: 0:00:06.812298
Task 018: 0:00:07.792578
Task 008: 0:00:08.771852
Task 015: 0:00:09.797658
Task 004: 0:00:10.835293
Task 006: 0:00:11.816410
Task 002: 0:00:15.864415
Task 021: 0:00:17.125667
Task 013: 0:00:18.221979
Task 003: 0:00:33.564914
Task 010: 0:00:34.612100
Task 001: 0:01:03.823003
Task 017: 0:01:33.927607
Task 011: 0:02:04.090997
Whereas if I run the code with only one worker thread (effectively making it serial) I get:

code:
Total number of tasks: 24
Task 000: 0:00:00.672847
Task 001: 0:00:01.772496
Task 002: 0:00:01.636722
Task 003: 0:00:01.611853
Task 004: 0:00:01.589885
Task 005: 0:00:01.583887
Task 006: 0:00:01.589106
Task 007: 0:00:01.613979
Task 008: 0:00:01.631233
Task 009: 0:00:01.588971
Task 010: 0:00:01.611877
Task 011: 0:00:01.583438
Task 012: 0:00:01.598294
Task 013: 0:00:01.599282
Task 014: 0:00:01.604587
Task 015: 0:00:01.607439
Task 016: 0:00:01.584491
Task 017: 0:00:01.599547
Task 018: 0:00:01.612187
Task 019: 0:00:01.598965
Task 020: 0:00:01.595185
Task 021: 0:00:01.608082
Task 022: 0:00:01.588849
Task 023: 0:00:01.583208
Any ideas where to start looking? System is not memory exhausted, and the program doesn't generate huge files.

Python 3.8.3 on Rocky Linux 8.5, if it matters.

QuarkJets
Sep 8, 2008

Are all instances writing to the same output file, as described (foo.txt)? That would be a problem. Use temporary working directories to solve that.

Start by wrapping this up in a concurrent.futures.ProcessPoolExecutor context manager, that's currently the proper way to implement multiple worker dispatch on a single node. It is a higher level interface for multiprocessing and is the new hotness. That's the point where you define the number of workers. Define a function that invokes your executable with subprocess.run (call is deprecated). Then map against it. Your N workers will each run the function until the argument map is exhausted.

Also, make sure you're not overloading the CPU. Half of your cores are probably logical, and processor-intensive tasks can't efficiently make use of them.

There is also a possibility that IO incurs a heavy penalty for simultaneous execution. Do the above changes first, then worry about this if you still see exponential overhead growth

QuarkJets fucked around with this message at 20:46 on Dec 17, 2021

QuarkJets
Sep 8, 2008

That pattern of execution is also pretty suspicious, multiple calls to the same program may be interfering with your efforts (e.g. if the first invocation forks a new process that just sits around listening, and the listening is a little too widely scoped such that all open processes are receiving and queuing commands from all other forked processes)

Popete
Oct 6, 2009

This will make sure you don't suggest to the KDz
That he should grow greens instead of crushing on MCs

Grimey Drawer
I created this structure but I have no idea what to call it or if it's even a proper thing. Can anyone help me figure out what this thing is? Obviously it's a 2d array but am I using the {} correctly with []? If so what is the distinction? I must have based this off an example but I can't seem to Google what kind of structure this is called in Python.

code:
sensor_temp_limits = {TempSensors.thermal_zone0: ['thermal_zone0', -35, False, 90, False],
                          TempSensors.thermal_zone1: ['thermal_zone1', -35, False, 90, False],
                          TempSensors.thermal_zone2: ['thermal_zone2', -35, False, 90, False],
                          TempSensors.thermal_zone3: ['thermal_zone3', -35, False, 90, False],
                          TempSensors.thermal_zone4: ['thermal_zone4', -35, False, 90, False],
                          TempSensors.thermal_zone5: ['thermal_zone5', -35, False, 90, False],
                          TempSensors.thermal_zone6: ['thermal_zone6', -35, False, 90, False],
                          TempSensors.thermal_zone7: ['thermal_zone7', -35, False, 90, False],
                          TempSensors.thermal_zone8: ['thermal_zone8', -35, False, 90, False],
                          TempSensors.thermal_zone9: ['thermal_zone9', -35, False, 90, False],
                          TempSensors.thermal_zone10: ['thermal_zone10', -35, False, 90, False],
                          TempSensors.thermal_zone11: ['thermal_zone11', -35, False, 90, False]}
I'm using an IntEnum called TempSensors to index into the 2d array but I'm not sure if that's entirely necessary.

QuarkJets
Sep 8, 2008

Popete posted:

I created this structure but I have no idea what to call it or if it's even a proper thing. Can anyone help me figure out what this thing is? Obviously it's a 2d array but am I using the {} correctly with []? If so what is the distinction? I must have based this off an example but I can't seem to Google what kind of structure this is called in Python.

code:
sensor_temp_limits = {TempSensors.thermal_zone0: ['thermal_zone0', -35, False, 90, False],
                          TempSensors.thermal_zone1: ['thermal_zone1', -35, False, 90, False],
                          TempSensors.thermal_zone2: ['thermal_zone2', -35, False, 90, False],
                          TempSensors.thermal_zone3: ['thermal_zone3', -35, False, 90, False],
                          TempSensors.thermal_zone4: ['thermal_zone4', -35, False, 90, False],
                          TempSensors.thermal_zone5: ['thermal_zone5', -35, False, 90, False],
                          TempSensors.thermal_zone6: ['thermal_zone6', -35, False, 90, False],
                          TempSensors.thermal_zone7: ['thermal_zone7', -35, False, 90, False],
                          TempSensors.thermal_zone8: ['thermal_zone8', -35, False, 90, False],
                          TempSensors.thermal_zone9: ['thermal_zone9', -35, False, 90, False],
                          TempSensors.thermal_zone10: ['thermal_zone10', -35, False, 90, False],
                          TempSensors.thermal_zone11: ['thermal_zone11', -35, False, 90, False]}
I'm using an IntEnum called TempSensors to index into the 2d array but I'm not sure if that's entirely necessary.

Maybe explain what you're trying to do

Macichne Leainig
Jul 26, 2012

by VG
Definitely need some more context. But at a surface level it looks like a normal dict with arrays as values which isn't the weirdest thing.

Popete
Oct 6, 2009

This will make sure you don't suggest to the KDz
That he should grow greens instead of crushing on MCs

Grimey Drawer

QuarkJets posted:

Maybe explain what you're trying to do

I'm writing a daemon that among other things reads out the temperatures from sysfs. I'm using that structure to hold all the thermal point names I want to read as well as the over/under temp thresholds and current fault state, when an over/under temp is detected it reports a fault to a fault log on the system.

I'm a relative newbie to Python and the distinction between arrays/lists/dicts etc. kinda goes over my head.

So I guess my question is, what data structure is it called? What is the distinction between curly braces and square brackets and did I do that correctly? It works, I just want to know if I did it "properly".

Popete fucked around with this message at 21:20 on Dec 17, 2021

bolind
Jun 19, 2005



Pillbug

QuarkJets posted:

Are all instances writing to the same output file, as described (foo.txt)? That would be a problem. Use temporary working directories to solve that.

Start by wrapping this up in a concurrent.futures.ProcessPoolExecutor context manager, that's currently the proper way to implement multiple worker dispatch on a single node. It is a higher level interface for multiprocessing and is the new hotness. That's the point where you define the number of workers. Define a function that invokes your executable with subprocess.run (call is deprecated). Then map against it. Your N workers will each run the function until the argument map is exhausted.

Also, make sure you're not overloading the CPU. Half of your cores are probably logical, and processor-intensive tasks can't efficiently make use of them.

There is also a possibility that IO incurs a heavy penalty for simultaneous execution. Do the above changes first, then worry about this if you still see exponential overhead growth

Thanks for replying!

The programs separated by && all write to the same file (sequentially) but each worker thread writes to its own file.

HyperThreading is in fact turned off on these machines, so one core == one real core.

I’ll try the ProcessPoolExecutor.

Macichne Leainig
Jul 26, 2012

by VG

Popete posted:

I'm a relative newbie to Python and the distinction between arrays/lists/dicts etc. kinda goes over my head.

So I guess my question is, what data structure is it called? What is the distinction between curly braces and square brackets and did I do that correctly? It works, I just want to know if I did it "properly".


So the big thing about lists and dicts is they are both collection types. What this basically means is that they're containers of data. One of the most notable differences between the two is how you access a specific piece of data.

In a standard list, it's index based, so you need to pass a number to access a member like so:

Python code:
>>> my_list
[1, 2, 3]
>>> my_list[0]
1
A dictionary instead is key-based, where the key is a string. The value and keys are paired together. To access a member, you'd use the string that represents the key:

Python code:
>>> my_dict
{'a': 1, 'b': 2, 'c': 3}
>>> my_dict["a"]
1
So let's go back to the dictionary you have defined and dissect it a bit.

Python code:
# Define the dict with a single element
sensor_temp_limits = {TempSensors.thermal_zone0: ['thermal_zone0', -35, False, 90, False]}
This dictionary only has one data item, whose value is the array ['thermal_zone0', -35, False, 90, False], and the key is TempSensors.thermal_zone0 which I presume is a string "thermal_zone0". So overall, I don't think what you have is "bad" but certainly could be improved. Maybe you're looking more for something like this:

Python code:
sensor_temp_limits = {
		TempSensors.thermal_zone0: [-35, False, 90, False],
		TempSensors.thermal_zone1: [-35, False, 90, False],
		TempSensors.thermal_zone2: [-35, False, 90, False],
		TempSensors.thermal_zone3: [-35, False, 90, False],
		TempSensors.thermal_zone4: [-35, False, 90, False],
		TempSensors.thermal_zone5: [-35, False, 90, False],
		TempSensors.thermal_zone6: [-35, False, 90, False],
		TempSensors.thermal_zone7: [-35, False, 90, False],
		TempSensors.thermal_zone8: [-35, False, 90, False],
		TempSensors.thermal_zone9: [-35, False, 90, False],
		TempSensors.thermal_zone10: [-35, False, 90, False],
		TempSensors.thermal_zone11: [-35, False, 90, False]
	}
This assumes that TempSensors.thermal_zone0 resolves to a unique string of course.

Then, to see what sensors you have configured, it's as simple as grabbing the keys. You could even check them programatically like so:

Python code:
temp_sensor_keys = sensor_temp_limits.keys()

for sensor_key in temp_sensor_keys:
	min_temp, is_foo, max_temp, is_bar = sensor_temp_limits[sensor_key]
	
	current_temp = get_current_temp(sensor_key)
	
	if current_temp < min_temp:
		# do something
I think what you have here - I'd call it a dictionary with lists for values - is pretty close to what you want, assuming my understanding of your problem is correct of course.

Macichne Leainig fucked around with this message at 21:36 on Dec 17, 2021

Popete
Oct 6, 2009

This will make sure you don't suggest to the KDz
That he should grow greens instead of crushing on MCs

Grimey Drawer

Protocol7 posted:

Helpful post

This is a great explanation thank you! I think I can improve upon it as you mentioned. I was actually using TempSensors as an IntEnum.

code:
    class TempSensors(IntEnum): 
        thermal_zone0 = 0 
        thermal_zone1 = 1 
        thermal_zone2 = 2 
        thermal_zone3 = 3 
        thermal_zone4 = 4 
        thermal_zone5 = 5 
        thermal_zone6 = 6 
        thermal_zone7 = 7 
        thermal_zone8 = 8 
        thermal_zone9 = 9 
        thermal_zone10 = 10 
        thermal_zone11 = 11 
        all_sensors = 12
So the key really resolved to an int and not a string. It probably makes sense to use the dictionary key as a string and eliminate the name field in the dictionary itself.

Like you showed here.

Python code:
sensor_temp_limits = {
		TempSensors.thermal_zone0: [-35, False, 90, False],
		TempSensors.thermal_zone1: [-35, False, 90, False],
		TempSensors.thermal_zone2: [-35, False, 90, False],
		TempSensors.thermal_zone3: [-35, False, 90, False],
		TempSensors.thermal_zone4: [-35, False, 90, False],
		TempSensors.thermal_zone5: [-35, False, 90, False],
		TempSensors.thermal_zone6: [-35, False, 90, False],
		TempSensors.thermal_zone7: [-35, False, 90, False],
		TempSensors.thermal_zone8: [-35, False, 90, False],
		TempSensors.thermal_zone9: [-35, False, 90, False],
		TempSensors.thermal_zone10: [-35, False, 90, False],
		TempSensors.thermal_zone11: [-35, False, 90, False]
	}

Protocol7 posted:

I think what you have here - I'd call it a dictionary with lists for values - is pretty close to what you want, assuming my understanding of your problem is correct of course.

That sounds correct. Basically I'm trying to use a single structure to track if a particular sensor currently has an over/under temp fault along with the threshold values and it's name.

Popete fucked around with this message at 21:42 on Dec 17, 2021

Macichne Leainig
Jul 26, 2012

by VG
Yeah that should be fine. I'm so used to just using strings with dicts that they're kinda the default key type for me. But as long as the numbers are unique (and it looks like they are) that should work just fine :)

I suppose you'll want to retain the string in your lists if you want to keep the sensor name, but that's a small change.

Adbot
ADBOT LOVES YOU

QuarkJets
Sep 8, 2008

Yeah, for the values you could maybe use a dataclass instead of a list to help label your code better. Seems fine to me though

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply