Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

QuarkJets: Sep 8, 2008

Boris Galerkin posted:

e: If you're trying to use this on windows, just save your sanity and use the docker container. It might sound weird or slow to do numerical computations in a docker container, but the more weird/slow part is trying to do it on windows.

I agree with this; Windows isn't really good at high performance computing, and getting all of that stuff working in Windows is going to take more time than the minor performance hit you take for using a container

It's pretty normal to use containers for high performance academic stuff anyway, the very first thing that tensorflow's user guide tells you about is Docker

# ? Nov 11, 2019 18:51

Adbot: ADBOT LOVES YOU

# ? May 27, 2024 16:13

Dominoes: Sep 20, 2007

M. Night Skymall posted:

Look, we can't all be the financial industry and run our critical services on *checks notes* giant excel spreadsheets.

Or the DoD pay system on *checks notes* millions of lines of undocumented COBOL.

I'm about to switch from Linux to Windows, and plan to use WSL+Conda in this case. There's also a Ubuntu package avail.

Dominoes fucked around with this message at 19:19 on Nov 11, 2019

# ? Nov 11, 2019 19:13

VictualSquid: Feb 29, 2012; Gently enveloping the target with indiscriminate love.

M. Night Skymall posted:

Look, we can't all be the financial industry and run our critical services on *checks notes* giant excel spreadsheets.

Hey, some academic software is based on matlab controlled excel spreadsheets.
Or excel based GPIB drivers for strange hardware.

# ? Nov 11, 2019 19:29

Hollow Talk: Feb 2, 2014

Malcolm XML posted:

What's people's favorite config parser that allows for env var override?

I'm spoiled by spring boots config engine which is p nice for yaml

I use oslo.config in a bunch of projects. It's originally part of Openstack, but it combines ENV, config file, and command-line parsing and is reasonably intuitive. I'm not overly keen on its use of a global config variable, but that makes sense in its original context and can be circumvented, and I still have trouble getting its cli config generator to work, since that also seems to assume an Openstack directory structure (and also thanks to python's stellar name spacing). Overall, I'm quite happy with it, though.

https://docs.openstack.org/oslo.config/latest/

# ? Nov 11, 2019 19:31

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

VictualSquid posted:

Or excel based GPIB drivers for strange hardware.

The excel files are actually csv files delimited with tabs.

# ? Nov 11, 2019 19:49

mbt: Aug 13, 2012

Boris Galerkin posted:

The excel files are actually csv files delimited with tabs.

Tab delimited comma separated values

# ? Nov 11, 2019 20:23

QuarkJets: Sep 8, 2008

Dominoes posted:

I'm about to switch from Linux to Windows

Yikes

# ? Nov 11, 2019 22:20

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

Hollow Talk posted:

I use oslo.config in a bunch of projects. It's originally part of Openstack, but it combines ENV, config file, and command-line parsing and is reasonably intuitive. I'm not overly keen on its use of a global config variable, but that makes sense in its original context and can be circumvented, and I still have trouble getting its cli config generator to work, since that also seems to assume an Openstack directory structure (and also thanks to python's stellar name spacing). Overall, I'm quite happy with it, though.

https://docs.openstack.org/oslo.config/latest/

Ended up going with ConfigArgParse which subclasses argparse parsers and adds env var and Yaml config overrides, which is good enough for me

# ? Nov 11, 2019 22:29

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

QuarkJets posted:

Yikes

I spend a lot of time developing on Windows and Linux so I can say something relevant about this:

It's pretty fine nowadays from the can-i-install-packages-and-stuff-without-futzing-around-with-Windows-specific-stuff stand point.

# ? Nov 12, 2019 01:57

Dominoes: Sep 20, 2007

Anecdotally, most software (Exceptions including the one we discussed) works the same on both. Advantage of Windows: I can play Age of Empires 2 without closing everything and rebooting. Also, this Linux adventure's quirks include crashing/rebooting or something whenever I'm away for a few hours. The proposed fix on the Ubuntu forums was Don't use non-LTS releases.🤷

Dominoes fucked around with this message at 04:19 on Nov 12, 2019

# ? Nov 12, 2019 04:17

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

If your Linux box is randomly crashing and rebooting then I think you have a bigger underlying problem with the hardware�

# ? Nov 12, 2019 06:25

Dominoes: Sep 20, 2007

Negative dawg! Don't gaslight me.

# ? Nov 12, 2019 18:50

cinci zoo sniper: Mar 15, 2013

They are right tho, even with non-LTS stable branch Ubuntu any crashes means either hardware fault or a PEBCAK issue. :colbert:

# ? Nov 12, 2019 19:30

Dominoes: Sep 20, 2007

Bro, sometimes you don't want to analyse, and just recognize that if things don't work, you move on. Life's too short.

# ? Nov 12, 2019 19:39

cinci zoo sniper: Mar 15, 2013

Dominoes posted:

Bro, sometimes you don't want to analyse, and just recognize that if things don't work, you move on. Life's too short.

I agree there, so far that I have a Mac as a non-video game home computer, but it still ought to be remarked, in my opinion, that system runtime stability is one of the distinctive features of your typical Linux distribution, specifically due to Unix philosophy with everything being a file and so on.

# ? Nov 12, 2019 20:17

DearSirXNORMadam: Aug 1, 2009

I was trying to do something dumb, but then reached a minimal case that confuses me

a = lambda: 1+1

is fine

a = lambda: pass

is not fine. What's up with that? Shouldn't I just be able to create an anonymous function that does nothing?

Edit:
And a followup question, it appears the issue is that pass is a statement that doesn't actually return None, which is what you would have happen with a function that only contains pass. Del is also a statement, rather than being a full fledged function. Can you have del(whatever) inside a lambda and have it work?

Edit2: Apparently lambdas just can't contain statements, question withdrawn but left up for posterity and to document to future generations my googling prowess

DearSirXNORMadam fucked around with this message at 21:08 on Nov 12, 2019

# ? Nov 12, 2019 20:59

susan b buffering: Nov 14, 2016

Lambdas must evaluate to a value, which is why that doesn�t work.

lambda: None would probably work.

# ? Nov 12, 2019 21:14

KICK BAMA KICK: Mar 2, 2009

Mostly a Docker question: a $5 Linode seems to be completely sufficient for my Django project (no real traffic, just stats/control panel for some code it already runs fine) but there's one CPU-bound task that runs like 5-10x faster on my new laptop without taxing it too hard or anything. So running the django-rq worker process for that task on my laptop when it's available and on the Linode when it's off is completely doable by setting up a Docker swarm, right? The Linode serves as the manager and all the other services -- nginx, gunicorn, the database, the django-rq scheduler, the other workers -- are configured to only run on it; the worker process for the CPU-bound-task is configured to run one instance on either node but prefer my laptop when it's up. Do I have that right?

# ? Nov 18, 2019 02:52

Sylink: Apr 17, 2004

How do I get intellisense/autocomplete working properly in VS Code with additional modules ? It seems to work generally, I launch via shell (this is on Linux Mint) in directory with my venv activated -> code . and its in the right environment.

But for example I use some additional non-standard lib like pygame or whatever, and it doesn't show the available functions properly in autocomplete.

I can't tell if its just broken or I'm missing some configuration step.

# ? Nov 18, 2019 19:43

Mr. Crow: May 22, 2008; Snap City mayor for life

I'm working on some selenium project to automate downloading transactions from my banks because I'm sick of everyone else's poo poo support for it; I've got it working for one bank, however not being familiar with selenium I wanted to sanity check that it's not going to be leaking secrets everywhere. It should be no different than using my browser correct? It's not actually doing anything outside the behest if the browser/web driver right (e.g. random unexpected requests)?

Also I created a FirefoxProfile to, what I thought, handle 2FA more effectively by saving cookies but it doesn't appear to load cookies? I had to create a map to answer security questions and it just runs through that each time which I'd like to avoid. How do I persist cookies?

Mr. Crow fucked around with this message at 00:05 on Nov 19, 2019

# ? Nov 18, 2019 23:59

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Mr. Crow posted:

I'm working on some selenium project to automate downloading transactions from my banks because I'm sick of everyone else's poo poo support for it; I've got it working for one bank, however not being familiar with selenium I wanted to sanity check that it's not going to be leaking secrets everywhere. It should be no different than using my browser correct? It's not actually doing anything outside the behest if the browser/web driver right (e.g. random unexpected requests)?

Also I created a FirefoxProfile to, what I thought, handle 2FA more effectively by saving cookies but it doesn't appear to load cookies? I had to create a map to answer security questions and it just runs through that each time which I'd like to avoid. How do I persist cookies?

I have not done any auditing of selenium but I believe you're correct. As an FYI I use AWS Secrets Manager so as not to hard code any secrets into my code. Took about 5 minutes of coding to get Secrets Manager set up and working. Even has a python example.

You should be able to get a list of cookies with cookies = driver.get_cookies(), save the cookies to a text file then load the list when needed and
driver.add_cookie(cookie) for cookie in cookies

CarForumPoster fucked around with this message at 00:29 on Nov 19, 2019

# ? Nov 19, 2019 00:27

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Yay, PyCharm now supports TypedDict!

The next big typing feature I'm waiting for is Literal support. That's coming in 2019.3.

# ? Nov 20, 2019 03:34

Dominoes: Sep 20, 2007

Have y'all been using TypedDict? What do you think? I've been using Dataclass all over my code, and it seems like mixing them in the same codebase might be confusing, due to how much they overlap. It seems like one notable difference is Dataclass's in-order constructor vs TypedDict's explicitly-keyed one. I wish you could modify DataClass to have a keyed default constructor, although without the quotes.

The TypedDict PEP cites json-serialization as something missing in DataClass, and mentions the dataclasses-json module. Pydantic also seems to be a great alternative, although I think it mainly uses its own struct-like.

Dominoes fucked around with this message at 08:43 on Nov 20, 2019

# ? Nov 20, 2019 08:33

unpacked robinhood: Feb 18, 2013; by Fluffdaddy

Is there a python adjacent way to do end to end testing for web services ?
At this point I'd just need something that takes a request and an expected result.

e: I forgot an important detail: once it's done my thing requests an url, It's the request data I'd like to check.
What happens is I receive a request on url A, immediately send an ack, run a job, and envtually send an execution report to url B

Basically

punished milkman posted:

I'm a bad programmer

unpacked robinhood fucked around with this message at 15:24 on Nov 20, 2019

# ? Nov 20, 2019 11:59

punished milkman: Dec 5, 2018; would have won

I'm a bad programmer so I use dataclasses absolutely everywhere for everything. I don't even understand why I would bother writing a regular class again when I can just get all of that sweet sweet boilerplate for free.

# ? Nov 20, 2019 14:58

Dominoes: Sep 20, 2007

punished milkman posted:

I'm a bad programmer so I use dataclasses absolutely everywhere for everything. I don't even understand why I would bother writing a regular class again when I can just get all of that sweet sweet boilerplate for free.

Amen. Haven't used a reg class since 3.7.

# ? Nov 20, 2019 15:25

mbt: Aug 13, 2012

Why would that make you a bad programmer

Input turns into output, your job is done

Most people probably use classes as loose collections of variables which data classes solves in a fantastic way

mbt fucked around with this message at 15:56 on Nov 20, 2019

# ? Nov 20, 2019 15:52

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I use typeddict to type dicts that come from elsewhere or that are going elsewhere, and this includes dicts that come from or are going to JSON APIs.

A typeddict is just a more natural fit for a dict.

I use dataclasses where a class is more appropriate.

It's the same reason regular dicts and regular classes exist in python.

# ? Nov 20, 2019 17:43

Dominoes: Sep 20, 2007

This is entirely subjective, but I associate a dict as something that has an arbitrary number of items (Where all keys have the same type, as do all vals), and a class/struct has a fixed number of items, each of a specific type. TypedDict falls into the latter category, with syntax of the former. It's an interesting hybrid. JS's Objects are another hybrid, and have historically been used for both cases, although now Map makes more sense for the former.

Dominoes fucked around with this message at 22:56 on Nov 20, 2019

# ? Nov 20, 2019 22:51

Hollow Talk: Feb 2, 2014

I use Lists of Tuples for everything unless I need a specific key, in which case I use dicts. I tend to avoid classes (any classes) as much as possible, though realistically, that's not really an option in Python. I do data engineering, so the ability to quickly destructure and restructure lists of tuples and dicts is a major boon, especially in comprehensions. I don't originally have a Python-background, why do you ask? :v:

For me, dicts are just arbitrary mappings, so any type is fine, really. One of the few places where I use actual classes are for config, since inheritance is a lot nicer than copy/pasting dicts for every environment. Essentially, I write a super class BaseConfig, and then have subclasses for different environments that simply override inherited values where necessary. I suppose for config stuff, direct access is also nicer than pulling things out of a dict. Then again, NamedTuples would offer that as well, though they are weird classes either way.

That said...avoiding classes isn't a thing in Python, I suppose.

code:

>>> type(''.__str__().__str__().__str__().__str__().__str__())
<class 'str'>

Hollow Talk fucked around with this message at 00:01 on Nov 21, 2019

# ? Nov 20, 2019 23:58

Dominoes: Sep 20, 2007

Lists of tuples sounds like trouble, since you're relying on indexes, which have no semantic meaning.

I've found organizing my projects are dataclasses and enums is nice - it provides a backbone where I think of my project in terms of how the data's organized. I with the enums had auto set by default, since adding integers manually implies meaning where there is none.

Eg from the docs:

Python code:

class Ordinal(Enum):
    NORTH = auto()
    SOUTH = auto()
    EAST = auto()
    WEST = auto()

Is more verbose than needed (ie Enums could have their own name without inheriting a class, and auto for each item is noise), but isn't too bad.

Dominoes fucked around with this message at 00:15 on Nov 21, 2019

# ? Nov 21, 2019 00:11

Hollow Talk: Feb 2, 2014

Dominoes posted:

Lists of tuples sounds like trouble, since you're relying on indexes, which have no semantic meaning.

That's why I said convert to dicts when you need to access a specific value.

code:

from itertools import groupby

data = [[('a', 1), ('b', 'foo')], [('a', 3), ('b', 'bar')], [('a', 2), ('b', 'baz')], [('a', 3), ('b', 'bat')]]

data_transformed = [[(k, v if isinstance(v, str) else str(v)) for k, v in x] for x in data]
data_sorted = sorted(data_transformed, key=lambda x: x[0][1])
data_grouped = [(k, len(g), g) for k, g in [(k, list(g)) for k, g in groupby(data_sorted, key=lambda x: x[0][1])]]

duplicate_keys = [k for k, l, g in data_grouped if l > 1]

They are great for certain things, and if I need specific keys, well...

code:

data = [[('a', 1), ('b', 'foo')], [('a', 3), ('b', 'bar')], [('a', 2), ('b', 'baz')], [('a', 3), ('b', 'bat')]]
data_dicts = [dict(x) for x in data]
data_tuples = [list(x.items()) for x in data_dicts]
assert data == data_tuples

All of this can be trivially changed to evaluate lazily.

edit: as background: https://norvig.com/python-lisp.html

Hollow Talk fucked around with this message at 00:55 on Nov 21, 2019

# ? Nov 21, 2019 00:47

susan b buffering: Nov 14, 2016

I tend to use NamedTuples from the typing module for data containers, and DataClasses for when I need a bit more than that.

I�ll probably use typed dicts more with APIs. Haven�t really had a chance to mess with them yet.

Dominoes posted:

Lists of tuples sounds like trouble, since you're relying on indexes, which have no semantic meaning.

I've found organizing my projects are dataclasses and enums is nice - it provides a backbone where I think of my project in terms of how the data's organized. I with the enums had auto set by default, since adding integers manually implies meaning where there is none.

Eg from the docs:
Python code:
class Ordinal(Enum):
    NORTH = auto()
    SOUTH = auto()
    EAST = auto()
    WEST = auto()
Is more verbose than needed (ie Enums could have their own name without inheriting a class, and auto for each item is noise), but isn't too bad.

You can use the functional API to achieve what you want. The below is equivalent to what you posted. I chose the space-delimited form for the second argument, but a sequence or mapping can be used instead.

Python code:

Ordinal = Enum("Ordinal", "NORTH SOUTH EAST WEST")

susan b buffering fucked around with this message at 02:00 on Nov 21, 2019

# ? Nov 21, 2019 01:51

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

Hollow Talk posted:

I use Lists of Tuples for everything unless I need a specific key, in which case I use dicts. I tend to avoid classes (any classes) as much as possible, though realistically, that's not really an option in Python. I do data engineering, so the ability to quickly destructure and restructure lists of tuples and dicts is a major boon, especially in comprehensions. I don't originally have a Python-background, why do you ask?

For me, dicts are just arbitrary mappings, so any type is fine, really. One of the few places where I use actual classes are for config, since inheritance is a lot nicer than copy/pasting dicts for every environment. Essentially, I write a super class BaseConfig, and then have subclasses for different environments that simply override inherited values where necessary. I suppose for config stuff, direct access is also nicer than pulling things out of a dict. Then again, NamedTuples would offer that as well, though they are weird classes either way.

That said...avoiding classes isn't a thing in Python, I suppose.
code:
>>> type(''.__str__().__str__().__str__().__str__().__str__())
<class 'str'>

Let me tell you about namedtuples,

# ? Nov 21, 2019 02:16

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

The functional api for things like namedtuples or enums is an insane code smell and a stain on the language imo. I mean come on

# ? Nov 21, 2019 02:48

NinpoEspiritoSanto: Oct 22, 2013

NamedTuples are shite use attrs

Use attrs anyway.

# ? Nov 21, 2019 03:19

Solumin: Jan 11, 2013

Bundy posted:

NamedTuples are shite use attrs

Use attrs anyway.

Attrs is superseded by dataclasses.

Phobeste posted:

The functional api for things like namedtuples or enums is an insane code smell and a stain on the language imo. I mean come on

If you're using the functional api for namedtuples, please upgrade to Python 3 and use the class api.

# ? Nov 21, 2019 05:14

QuarkJets: Sep 8, 2008

PyCharm is great in how easily it lets me convert .format strings into f-strings

Does anyone know how I can reformat an entire file, or an entire project, in this way? Basically replacing any explicit string with a .format on the end of it into an f-string?

# ? Nov 21, 2019 07:39

NinpoEspiritoSanto: Oct 22, 2013

Solumin posted:

Attrs is superseded by dataclasses.

Ummm... No it isn't.

https://www.attrs.org/en/stable/why.html#data-classes

NamedTuples continue to suck in py3 too:

https://www.attrs.org/en/stable/why.html#namedtuples

Obviously if a dataclass covers what you need great, but they no more supersede attrs than datetime supercedes pendulum or urllib supercedes requests.

NinpoEspiritoSanto fucked around with this message at 10:37 on Nov 21, 2019

# ? Nov 21, 2019 10:34

Adbot: ADBOT LOVES YOU

# ? May 27, 2024 16:13

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

TLDR: Is it possible to use the string keys in name_map to pass argument values from the def save_row(args) to my Series?

Context:
I want to update a Google Sheet row based on a bunch of HTML inputs and clicking a button using my Dash (Flask for Dashboards) app.

I have a pandas Series, df, that contains the initial values of a google sheet row. wks.update_row() passes a list. name_map maps the arg value passed from State to the pandas Series index name.

I want to do something like this: (I have 3x the variable names, this is just shortened)

code:

@app.callback(
    Output('save-gsheet-container', 'children'),
    [Input('save-gsheet', 'n_clicks')],
    [State('company_name', 'company_name_input')],
    [State('fn', 'fn_input')],
    [State('ln', 'ln_input')],
    [State('a1', 'a1_input')],)
def save_row(n_clicks, company_name_input, fn_input, ln_input, a1_input):
    name_map = {
                "company_name": "company_name",
                "fn": "First Name",
                "ln" : "Last Name",
                "a1": "Address 1"}
        for k, v in name_map.items():
            df[v] = [THE KEYWORD PASSED FROM State THAT MAPS TO k IN name_map]
        row_update = df.to_list()
        wks.update_row(row_update)

EDIT:

CarForumPoster posted:

TLDR: Is it possible to use the string keys in name_map to pass argument values from the def save_row(args) to my Series?

So this: exec(f"df[v] = {k}_input") works

but it seems like I am not supposed to use exec for things...is there a better way that I'm gonna feel dumb for not realizing in a minute?

CarForumPoster fucked around with this message at 16:32 on Nov 21, 2019

# ? Nov 21, 2019 15:33

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »