Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Is the smtplib module not thread-safe or something? I have a little monitor script set up to monitor some SQL servers and send out an e-mail if there's a long-running query that's taking up too much time on the server. Everything works fine, but when I added smtplib into the mix, everything went all wrong. As soon as the monitor thread sends an e-mail, the script just seems to stop execution shortly after the server.quit(). It sends that e-mail fine (with some formatting issues, but I'll fix that poo poo later) and I see all the debug output on the terminal from set_debuglevel(), and I see it print "E-mail successful" as well (because I added that after the server.quit() to see if it was hanging on waiting for the connection to time out or something), but it never exits the enclosing elif. It's not using any CPU time or anything... it just doesn't do anything. Here's the relevant code:

code:

for k in new_current_pid_dict:
    if new_current_pid_dict[k][2]:
        pass
    elif (new_current_pid_dict[k][1] - new_current_pid_dict[k][0]) > 15: #3h
        from_addr = "pgmonitor@blah"
        to_addr = "me@blah"
        subject = "Database process",k
        warning_text = "Database process",k,"has been running for more than three hours.\r\n \
        Please review this process and terminate it if necessary with 'kill -SIGINT",k,"."
        msg = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n%s" % (from_addr,to_addr,subject,warning_text)

        print "SENDING EMAIL",k
        server = smtplib.SMTP('localhost')
        server.set_debuglevel(1)
        server.sendmail(from_addr,to_addr,msg)
        server.quit()
        print "E-mail successful"
        new_current_pid_dict[k][2] = True

If I just have it print a string or something, no problem, the script continues as normal. smtplib's presence here is loving things up in some way.

# ¿ Apr 9, 2008 16:27

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 23:14

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

m0nk3yz posted:

Chutwig Regarding smtplib - did you look at this thread on c.l.p?

http://groups.google.com/group/comp...a1de3ed29133546

After I posted, I went back and googled some more and found that discussion. I'll probably just write another script external to this one to handle e-mailing and invoke it through subprocess, or just invoke sendmail directly, or something.

# ¿ Apr 9, 2008 21:09

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

fletcher posted:

Hmmm I think it's because this csv file was actually latin1 encoded even though the dude said it was UTF-8.
code:
>>> 'C�te d\'Ivoire'.encode('utf8')
b"C\xc3\xb4te d'Ivoire"
>>> 'C�te d\'Ivoire'.encode('latin1')
b"C\xf4te d'Ivoire"
How come this command says the file is utf8 even though it isn't actually utf8?
code:
$ file -bi countries.csv 
text/plain; charset=utf-8

Maybe it starts with a UTF-8 BOM but is really ASCII? If you hexdump it and it starts with EFBBBF, file might see that and just stop there.

# ¿ Jun 30, 2015 03:47

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

superstepa posted:

What I want is to have one main class that only defines a few very basic methods and variables and then have a large bulk of extra methods/variables appended to an instance of that class post initialization. I know you can just use setattr but I was wondering if there is an "official" way to do that.

Sorry, I think I'm just making a really simple question sound really convoluted.

This sounds a little bit like you want to create an abstract base class and then create subclasses from that. What's the underlying problem you're trying to solve?

# ¿ Jul 19, 2015 23:41

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Fergus Mac Roich posted:

So, Python mavens, is this the most elegant way to do this?
code:
def countit(arg):
    count_sub1, count_sub2 = Counter(), Counter()
    with open(arg) as dataf:
        sub1, sub2 = tee(DictReader(dataf, fieldnames=fieldnames))
        sub1 = filter(partial(validate, func=is_sub1), sub1)
        sub2 = filter(partial(validate, func=is_sub2), sub2)
        for item in sub1:
            count_sub1[item[a_key_from_fieldnames]] += 1
        for item in sub2:
            count_sub2[item[a_key_from_fieldnames]] += 1
    return count_sub1, count_sub2
fieldnames is a list of strings that comes from outside the function. validate looks a bit like this, in case it's not clear what I'm doing:
code:
def validate(item, func):
    item = {key: item[key].strip() for key in item}
 #maybe some things have whitespace; the functions called below do string comparisons
    return meets_common_criteria(item) and func(item)
For some reason it feels like I could improve this, but I just can't get my head around how. (The variable names are made up)

It seems like you wrote countit() the way you did so that you could use tee() to not read the file twice, but with the consequence that now everything else in the function is doubled up. If you expect to re-use the file contents over and over, read it into a buffer outside the function and pass that in. There are also some things being accessed inside countit() that look like globals, e.g., fieldnames.

Do the counts need to be done together like that? Is there a reason why they can't be done one at a time? If the state of one counter has no bearing on the other, then I would advise doing them separately instead of doubling up. Generally speaking you will want to deal with having 0 things, 1 thing, or more than 1 thing. Right now you're dealing with 2 things as 2 things, instead of dealing with them as 1 thing twice. Changing countit() to handle one counter as a time is a more generalizable solution.

# ¿ Sep 4, 2015 00:36

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Fergus Mac Roich posted:

What do you think of this?
Python code:
#calling code
fieldnames = #some strings
mykey = #a key from fieldnames
with open(myfile) as f:
    mydata = list(DictReader(f, fieldnames=fieldnames))
sub1 = countit(*mydata, validator=partial(validate, is_sub1), key=mykey)
sub2 = countit(*mydata, validator=partial(validate, is_sub2), key=mykey)

def countit(*args, validator, key):
    count = Counter()
    items = [item for item in args if validator(item)]
    for item in items:
        count[item[key]] += 1
This script actually does have some globals that are never modified, and yeah, that includes fieldnames. I've deemed it "okay" for this since it's just a script that runs for a second, writes all the counts to a file, and dies. However, I understand that you're right that as far as elegance and best practices, globals are generally bad design.

I think that is a lot more reasonable, yeah. Obviously there's such a thing as quick hacky scripts, but I've been burned by wackity global variable scope too much in the past and prefer to pass arguments around explicitly.

# ¿ Sep 4, 2015 00:54

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Emacs Headroom posted:

Flask might feel a little underpowered.

Flask intentionally deals with a much smaller subset of the problem domain than Django does. As you said, Django is a large framework, and it provides a lot of different things that Flask by design does not. You can do anything in Flask that you can do in Django, but you'll need to do a lot more of your own plumbing. Some people prefer it that way. I've published Real Applications� in Django and tinkered with Flask, and I like the lightweight feeling of Flask. Django's out-of-the-box admin site is still one of its huge selling points, though, and Flask won't ever provide that.

Also, to your point about Angular/React/Ember stuff, there's no reason why you can't have the frontend of your site written using one of those while the backend is Django/Flask/Rails/something else that emits JSON to the frontend framework. It's not really about the site itself being written "in Python", since that's all on the backend and you're not going to be able to do anything dynamic without JS.

# ¿ Sep 7, 2015 02:00

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Thermopyle posted:

PEP-484 makes tooling (like PyCharm) much better and is now in 3.5 I've been using it on Python 3.4 with PyCharm for a year or so and it's only gotten better.

I really hope that at some point the interpreter will have a mode to enforce the type signatures if present, like a -t flag or something to that effect.

Cingulate posted:

Is there (gonna be) a from __future__ import type_annotations thing? Also why do people continue to use python 2

For Python 2? It's not getting anything new at this point and adding annotations to 2 would require a lot of work, the type signature stuff builds on the generic annotations that 3 introduced.

As for why 2.x is still used... though most OpenStack stuff has�to pass gates at both 2.7 and 3.4 now so it may be usable with 3.x by Mitaka, Twisted seems to be coming along (and a lot of what it does is subsumed into 3.5 anyway with all the async/await functionality), and Mercurial will probably eventually be clubbed into submission by the core Python developers.

# ¿ Sep 14, 2015 02:29

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Thermopyle posted:

You can use MyPy to check types for you.

I know MyPy exists, but it'd be even nicer to have something blessed in CPython that says This Is An Okay Thing. Python's obviously survived a quarter century (jfc) without it, but it'd be a nice-to-have thing.

# ¿ Sep 15, 2015 01:06

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Are you working in Python 2 or 3? You're manually inheriting from object which implies Python 2, but calling super() without arguments which implies (really requires) Python 3.

My best guess is there's some weird inheritance diamond stuff going on. What happens if you change the super call to super(Inventory, self).__init__()?

# ¿ Sep 20, 2015 21:13

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

hooah posted:

When I run this, I get a TypeError on the last line: "randBinList() missing 1 required positional argument: 'n'". I get that randBinList isn't getting its argument, but I don't understand why. I thought the initRepeat should take care of that?

It's because when you supply n=IND_SIZE, that's going to get passed as an argument to tools.initRepeat and not to randBinList. If that register were turned into a method invocation, it would look like

Python code:

tools.initRepeat(creator.Individual, toolbox.attr_bin, n=IND_SIZE)

The callable that you supply to tools.initRepeat looks like it needs to take no arguments or supply defaults for any arguments. If you register randBinList with an argument for n, it works as expected:

Python code:

toolbox.register("attr_bin", randBinList, n=IND_SIZE)

# ¿ Sep 23, 2015 02:26

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Suspicious Dish posted:

Python code:
def is_palindrome(x):
    return x == x[::-1]
Is what I usually see for idiomatic Python.

the magnificent burn of being ultra code golfed

# ¿ Oct 14, 2015 23:31

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

QuarkJets posted:

Numpy is Python

Probably what he means is "can be done with the Python standard library without needing to install anything".

# ¿ Oct 15, 2015 23:48

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

What's the right way to express a complex type hint for a dictionary of the following format:

code:

{'command': 'some string', 'coord1': (0,0), 'coord2': (1,1)}

I tried rendering it as Dict[[str, str, str], [str, Tuple[int, int], Tuple[int, int]]], but the interpreter yakks on this with the error TypeError: Parameters to generic types must be types. Got [<class 'str'>, <class 'str'>, <class 'str'>]. PEP 0483/0484 are not clear on this (the only dict examples they give are a single key-value set) and using type hints is not widespread enough to turn anything up on Google.

# ¿ Dec 7, 2015 00:44

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Ah, I understand now. I re-rendered it as Tuple[str, Tuple[int,int], Tuple[int,int]], since the names weren't important.

# ¿ Dec 7, 2015 02:16

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

QuarkJets posted:

But you're basically creating a bunch of these tuples and then magic-indexing into them to get coord1 and coord2, right? You should really, really consider using a class instead; it'll be so, so much cleaner and easier to understand

It's a one-off script for advent of code, not something that anyone but me will ever look at. I know it would be cleaner to create an actual class and pass that around, but for this use case a tuple with magic indexes is fine.

# ¿ Dec 7, 2015 19:24

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

Hammerite posted:

Why do you need to use type hinting then?

Just practicing expressing things in it. Does there need to be a reason? It's different syntax from what I'm used to with Java/Go, so I figured I'd try to formulate the appropriate Python type hints.

# ¿ Dec 8, 2015 00:58

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

baka kaba posted:

From looking around, this is possible but it's pretty hacky, which makes it seem like you're not really meant to do this kind of thing. What exactly are you doing? If you lay out the general plan then people might have some more, uh, pythonic suggestions

It's not hacky, it's just taking advantage of first-class objects, and you'd treat it as any other callable.

# ¿ Oct 7, 2016 18:00

Adbot: ADBOT LOVES YOU

# ¿ May 14, 2024 23:14

chutwig: May 28, 2001; BURLAP SATCHEL OF CRACKERJACKS

huhu posted:

code:
import configparser

config = configparser.ConfigParser()
config.read('config.ini')

some_value = config['DEFAULT']['some_value']

some_function(some_value)
Is there a more elegant way to get values from a config file and then use them in a Python script or is this it?

You could think about using oslo.config, which is a configuration parser library used by OpenStack that is reasonably nice to use (I've used it in some projects). The downside is that it will drag in a number of other dependencies - not too many by OpenStack standards, but maybe more than you want to deal with: http://git.openstack.org/cgit/openstack/oslo.config/tree/requirements.txt

# ¿ Feb 26, 2017 16:58

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.