Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›9 »

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Cyril Sneer posted:

Hey guys. To what extend can these various web frameworks be used to control local hardware?

Let me set up the problem I'm trying to solve. I work in a facility where Technician Bob might want to interface with hardware X, and Technician Sam might want to interface with hardware Y. The way it works right now is someone like me physically accesses Bob's laptop and installs whatever Python Stuff (environment, scripts) is needed for interfacing with hardware X. Then, someone like me gets ahold of Sam's computer and installs whatever Python Stuff is needed for him to interface with hardware Y.

I was thinking it would be really cool if instead Bob, Sam, and whoever else could simply access an internal webapp that provided the necessary functionality. I know that you can control hardware via a web interface but (and I'm going to bungle the phrasing here) where I've seen this, its external hardware connected to a server, and the server provides a remote user access to the local hardware. What I'm thinking is a bit of an inversion of this -- user connects their laptop to the hardware, loads the appropriate site, and then that "remote" site enables control of the local hardware (all of this would be fully internal).

My coding background is primarily in DSP/algorithm development/embedded processing so this webapp stuff is all a bit foreign to me.

If Python code can run the hardware, you can very easily call whatever function you�ve already written.

Computer hardware is cheap compared to technician time so I�m assuming you�re putting a dedicated laptop on the hardware.

There�s a gotcha to this if you�re new which is timeout. You can make long running calls, but they should return something before the timeout. Typical timeouts are 10s-30s. Obvs then you can poll for status or push when the long running action is complete.

Dash is basically tailor made for this. It has some long running process functionality built in

CarForumPoster fucked around with this message at 22:23 on Jul 5, 2023

# ¿ Jul 5, 2023 22:17

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 13:06

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Cyril Sneer posted:

No, there aren't dedicated laptops attached to each machine. Yes, it would be easy if that was the case - hence the question.

To elaborate, these "machines" are really just things like scopes and VNAs. We have scripts for operating them in particular ways (along with collecting/recording data in certain ways). Our engineers have their own laptops and want to be able to use these tools freely.

Could you draw a use diagram that has these components and some idea of what scripts are where?

It sounds like the current state of affairs is this:

And what you want is...??this??

IDGI, provide the specifics. What problem you tryin to solve, what does success look like?

# ¿ Jul 5, 2023 23:57

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

pmchem posted:

what's the easiest way to plot county-by-county geographic data in python these days along with some good data sources?

like say I want to plot rainfall, or average seasonal temperatures, and mosquito density for a county-by-county map of the USA. counties colored by amount. googling this leads down many not entirely useful rabbit holes.

are arcgis and geopandas the current hotness? does arcgis only interface with... arcgis, which I generally understand is a nonfree api?

I haven�t looked in three years but back then county by county was a bitch. I think I ended up doing it by zip assigning certain zip codes to certain counties based on a spreadsheet I found of that. IIRC some zips spam multiple counties but it was good enough for the problem I had then.

# ¿ Jul 10, 2023 23:20

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Falcon2001 posted:

I haven't used PyCharm in a while but in general, I'll at least reassure you that sometimes debugger setups are kind of weird and not well explained and cause more consternation than most new devs are expecting.

Seems like all Python IDEs debugging of .py files are so bad that it inspired someone to make jupyter

# ¿ Aug 17, 2023 04:05

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Falcon2001 posted:

Ehhhh Jupiter is a very different beast and has a lot more to do with cached cell by cell execution, etc.

# ¿ Aug 17, 2023 04:15

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

StumblyWumbly posted:

Just today, I put in the most basic rear end updates into some code I wrote like a year ago. The file already had the most basic testing possible in it, and I saw it and said "sure, I guess I'll check my work" and found like 3 errors.
Because I wrote those tests a year ago, I was able to finish half a bottle of wine before my long weekend started.
So proud of who I was, that man is my hero

At what point in the coding journey do you transition from "there no coder worse than me, 6 months ago" to "old me was my hero"

# ¿ Sep 1, 2023 04:18

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Falcon2001 posted:

This implies a one way path when in reality Previous Me is the best and worst of all possible futures; a superstate composed upon itself and containing all dread and terror and wonder at once. But mostly, he's just a jerk.

Schrodinger's code

its only once the code is deployed to prod do you know whether previous you is good or bad

# ¿ Sep 1, 2023 04:55

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Generic Monk posted:

Good point

https://github.com/poisonwomb/flaskblog

I think that's provided by flask_login

lmao @ being a dude with the github name "poison womb"

# ¿ Sep 2, 2023 21:18

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Chillmatic posted:

That's amazing idk what you're talkin about

Im talking about how good of a github user name it is

a lot to very

# ¿ Sep 2, 2023 21:39

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Selenium raises whenever an element isn�t present in a web page so try except NoSuchElementException is basically if/then.

It�s normal to verify something ISNT there by passing or suppressing.

# ¿ Oct 3, 2023 14:01

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Jose Cuervo posted:

I want to make sure that a string that is passed in has the year-month format 'yyyy-mm'. So for example, '2022-09' would work or even '2022-9', but not '2022 09' or '22-09' etc. Should I be trying to use a regular expression for this?

This seemed like a fun thing to pass to ChatGPT, so I asked for three ways it could be done because two seemed obvious: regex and datetime. It found a third and that other way ended up being the fastest (according to ChatGPT Advanced Data Analytics Plugin, I didnt test it)

code:

#Prompt: Give me three ways in python to make sure that a string that is passed in has the year-month format 'yyyy-mm'.  So for example, '2022-09' would work or even '2022-9', but not '2022 09' or '22-09' etc. 

import re
from datetime import datetime

# Method 1: Using Regular Expression
def is_valid_date1(date_str):
    return bool(re.fullmatch(r'\d{4}-\d{1,2}', date_str))

# Method 2: Using str.split() and integer conversion
def is_valid_date2(date_str):
    parts = date_str.split('-')
    return len(parts) == 2 and parts[0].isdigit() and len(parts[0]) == 4 and parts[1].isdigit()

# Method 3: Using datetime.strptime() for strict format matching
def is_valid_date3(date_str):
    try:
        datetime.strptime(date_str, '%Y-%m')
        return True
    except ValueError:
        return False

# Test the functions
test_strings = ['2022-09', '2022-9', '2022 09', '22-09']
results1 = [is_valid_date1(s) for s in test_strings]
results2 = [is_valid_date2(s) for s in test_strings]
results3 = [is_valid_date3(s) for s in test_strings]

results1, results2, results3

code:


# Prompt: Test which of the three functions is fastest in evaluating 100 example date strings. 

import timeit

# Generate 100 example date strings (50 valid and 50 invalid)
example_dates = ['2022-09', '1999-12', '2022 09', '22-09'] * 25

# Time each function
time1 = timeit.timeit("for s in example_dates: is_valid_date1(s)", globals=globals(), number=1000)
time2 = timeit.timeit("for s in example_dates: is_valid_date2(s)", globals=globals(), number=1000)
time3 = timeit.timeit("for s in example_dates: is_valid_date3(s)", globals=globals(), number=1000)

time1, time2, time3
RESULT
(0.1321420590000031, 0.04751532600000985, 0.7127733419999913)

# ¿ Oct 6, 2023 03:56

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Zugzwang posted:

I might check it out, thanks.

Plotly is really good for a lot of things.

Matplotlib's strength is its weakness; it can do everything. But its API can be cumbersome because of that. I like Seaborn for making the most common types of plots. It's nice because it's a wrapper around Matplotlib, so you can still add extra fancy customizations to its Figure objects if you want to.

ChatGPT with GPT4 does decent with very tailored matplotlib made over several prompts if you end up in that edge case where seaborn and plotly don�t let you customize quite how you want.

I�ve done this when making specifically shaped and annotate histograms for example.

# ¿ Oct 19, 2023 12:10

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

StumblyWumbly posted:

Also it seems like the kind of thing chatgpt would be good at, but I have not tested that

Its better than me, but thats not hard.

# ¿ Oct 21, 2023 13:14

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

TasogareNoKagi posted:

Guess I won't be using Django then.

Why would you use composite keys (idk poo poo about DBs)

# ¿ Nov 4, 2023 11:58

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

His Divine Shadow posted:

Well I wrote my first small python app. I almost broke my brain with Oauth2, problem was microsofts backend, god I hate MS. In the end it was not possible to get imap/smtp working so I had to use microsoft.graph and that worked.

Now I got a little app that searches my inbox, finds a particular mail, processes it as desired and sends it along. A little manual work gone.

As someone who has hated themselves for this: if you have to rotate your app's credentials in Azure AD PUT THAT ON YOUR CALENDAR RIGHT NOW.

# ¿ Nov 22, 2023 05:59

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

onionradish posted:

Speaking of OAuth, is there an easy way or recommended module that will allow a Windows PC to respond to the callback to get a token after passing in the initial key to an arbitrary service?

I just want to authorize access for a simple personal script on my home PC. The OAuth hassle to set up a public server vs using a simple authentication key is pushing a bunch of "I could automate that" projects to "UGH; maybe some other day..."

Im not 100% sure I understand the ? If you need to refresh tokens on some interval on windows just schedule a task to run every n minutes. If this is a PC that is intermittently on such that tokens may expire in that time, you'll need to have some scheduled or always on thing handle that, such as a AWS Lambda and/or a Zapier task.

Lambda + Zapier make easy work of this if you dont wanna gently caress with configuring a Lambda to be able to reach the public internet.

Or is the callback you're referring to in this case requesting a user browse to a web page and log in?

EDIT: I forgot that you prob have to register a URL to make this request, and I am assuming that they won't let you register localhost. Yea, try Zapier they might have something for this use case. Otherwise AWS lambda set up with Zappa should give you an API Gateway that you can register and would cost nothing for infrequent use.

CarForumPoster fucked around with this message at 04:52 on Nov 23, 2023

# ¿ Nov 23, 2023 04:47

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

It looks like zapier might already have a pocket integration in which case you don�t need to worry about any of this.

https://zapier.com/apps/pocket/integrations

Send your stuff wherever you want like a Google sheet, MS Teams, or use webhooks to make your own api.

# ¿ Nov 24, 2023 16:44

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

The Fool posted:

tangentially related, is there a good way to generate a requirements.txt that only has what is being imported in my working directory and not everything in my entire environment

Requirements.txt is usually made by pip freeze or by hand as you install stuff so you can maintain orders and versions. What�s being imported doesn�t necessarily have 1:1 naming with the package name on pip.

So maybe, but there�s likely going to be a lot of issues with that a year or two from now if the environment needs to get reinstalled.

# ¿ Jan 22, 2024 19:48

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

StumblyWumbly posted:

Is using docker to do builds with pyinstaller a dumb idea or a great idea?
It seems like a venv could work just as well but there's always something weird going on

Ive only ever pip installed my requirements in the container's dockerfile but is always for AWS Lambda.

# ¿ Feb 13, 2024 21:31

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Oysters Autobio posted:

Any pandas experts here?

I've got an adhoc spreadsheet I need to clean for ETL (why this is my job as a data analyst, I don't know) but I'm having trouble even articulating the problem decently enough to leverage ChatGPT or Google.

Spreadsheet (.XLSX file) is structured like this
code:
+----------+-------+------------+
| category | names | dates      |
+----------+-------+------------+
| Western  |       |            |
| 1        | Jim   | 2023.02.01 |
| 2        | Greg  | 2013.12.11 |
| 3        | Bob   | 2003.07.11 |
| Eastern  |       |            |
| 1        | Jess  | 2002.02.01 |
| 2        | Bill  | 2001.10.11 |
|          |       |            |
|          |       |            |
+----------+-------+------------+
Repeat for many more categories (i.e. "Southern", "Southwestwern") each with different ranges of numbers following them with matching names/dates.

Reading it in as a data frame with defaults has pandas assigning the "category" column as the index, but despite reading docs and googling I still can't really wrap my head around multi - indexes. Lots of tutorials on creating them, or analysing them, but can't find anything for the reverse (taking a multi-level index and transforming it into a repeating column value). The target model should be something like this:
code:
+---+----------+-------+------------+
|   | category | names | dates      |
+---+----------+-------+------------+
| 1 | Western  | Jim   | 2023.02.01 |
| 2 | Western  | Greg  | 2013.12.11 |
| 3 | Western  | Bob   | 2003.07.11 |
| 1 | Eastern  | Jess  | 2002.02.01 |
| 2 | Eastern  | Bill  | 2001.10.11 |
|   |          |       |            |
+---+----------+-------+------------+
edit: should specify that we'll like receive these types of spreadsheets with different labels and values. So I'd like to write it to be somewhat parameterized and reusable, which is why I'm not just hacking through this for a one-time ETL

Stupid simple option if they�re grouped in order
Get list of your categories
Save it as a CSV
Use the category list to break it into chunks for each category.
Make each chunk df, adding a column for the category
pd.concat

Otherwise I�d try to read the excel spreadsheet such that the index is ordered numerically and get the indexes that contain the categories (eg by filtering to where the other columns have nans) then just map the category to its applicable data rows between each category

# ¿ Feb 22, 2024 05:47

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Oysters Autobio posted:

Just getting into learning flask and my only real question is how people generally start on their HTML templates. Are folks just handwriting these? Or am I missing something here with flask?

Basically I'm looking for any good resources out there for HTML templates to use for scaffolding your initial templates. Tried searching for HTML templates but generally could only find paid products for entire websites, whereas I just want sort of like an HTML component library that has existing HTML. i.e. here's a generic dashboard page html, here's a web form, here's a basic website with top navbar etc.

Bonus points if it already has Jinja too but even if these were just plain HTML it would be awesome.

edit: additionally, has anyone used flask app builder before? The built in auth and admin panels are mighty appealing but I'm worried about locking into a heavily opinionated framework (why I chose flask over say Django)

If your goal is to make dashboards or web apps with a few pages for internal tools I highly recommend Dash. No HTML, Python objects only including a Markdown object which makes things more readable than templates imo. Theres a well supported Bootstrap package called dash bootstrap components which gives you pretty bootstrap things and has a Bootstap navbar. I wouldn�t consider it any more opinionated than Flask, which doesn�t mean none, certainly no where near Django.

No need for Jinja because again, Python only.

CarForumPoster fucked around with this message at 22:11 on Mar 19, 2024

# ¿ Mar 19, 2024 22:08

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Oysters Autobio posted:

Would it be odd to use data frames (pandas, polars etc) for non dataset/table related functions? Like I'm making a generic Flask app and I move and manipulate the data around with pandas data frames, transforms with map-apply style functions etc? Or would this really make it an unreadable mess for others?

That�s fine

# ¿ Mar 23, 2024 23:15

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Fender posted:

Chiming in about how someone gets wormy brains to the point where they use lxml. In short, fintech startup-land.

We had a product that was an API mostly running python scrapers in the backend. I don't know if it was ever explained to us why we used lxml. By default BeautifulSoup uses lxml as its parser, so I think we just cut out the middleman. I always assumed it was just an attempt at resource savings at a large scale.

Two years of that and I'm a super good scraper and I can get a lot done with just a few convenience methods for lxml and some xpath. And I have no idea how to use BeautifulSoup.

I use lxml when needing to iterate over huge lists via xPaths from scraped data. Seems to be the fastest and it ain�t that hard. Selenium is slow at finding elements via xpath when you start needing to find hundreds of individual elements.

Also if you�re using selenium, lxml code can kinda look similar.

I spent multiple years writing and maintaining web scrapers and basically never used BS4.

CarForumPoster fucked around with this message at 10:13 on Apr 5, 2024

# ¿ Apr 5, 2024 10:10

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Falcon2001 posted:

Data classes rule. Use them everywhere.

I�ve met like three functions that should be a class.

# ¿ Apr 26, 2024 20:35

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 13:06

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Chillmatic posted:

I wrote a script to get all the xml files from the show in question, and then iterate through them using the intro marker data in those xml files to decide where to make the edits using ffmpeg, except now I'm running into losing the subtitles from the original files, and converting them is a whole thing because they're PGS format.

Comedy option: Send the edited files to OpenAIs whisper API and let them create new transcripts for you!

# ¿ May 1, 2024 04:14

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›9 »