Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Quebec Bagnet: Apr 28, 2009; mess with the honk
you get the bonk; Lipstick Apathy

So I need to generate some docx files. The choices for libraries seem to be paradocx (which doesn't look all that advanced, unless I'm missing something), python-docx and its forks (some dead, some alive), pyth (which technically generates RTF files, but those would probably be suitable too), COM interop (Windows-only) or comedy option .NET interop to Word itself using IronPython. Anyone familiar with any of these approaches?

# ? Jan 12, 2012 08:02

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 17:47

FoiledAgain: May 6, 2007

Is there any easy to way to determine if a list with many layers of sublists contains some element *anywhere*? Something like this:

multi_list = ['a', ['b', 'c'], ['d', ['e', 'f']], 'g'] ideal_function(multi_list, 'f') >>> True

In practice, the embedding probably won't be deeper than about 5, [[[[['a']]]]] , if that makes any difference (although in principle any number is possible).

# ? Jan 12, 2012 08:36

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Flatten it then search it?

# ? Jan 12, 2012 08:50

No Safe Word: Feb 26, 2005

i barely GNU her! posted:

So I need to generate some docx files. The choices for libraries seem to be paradocx (which doesn't look all that advanced, unless I'm missing something), python-docx and its forks (some dead, some alive), pyth (which technically generates RTF files, but those would probably be suitable too), COM interop (Windows-only) or comedy option .NET interop to Word itself using IronPython. Anyone familiar with any of these approaches?

If RTF would be suitable, how about HTML?

# ? Jan 12, 2012 08:51

The Gripper: Sep 14, 2004; i am winner

FoiledAgain posted:

Is there any easy to way to determine if a list with many layers of sublists contains some element *anywhere*? Something like this:

multi_list = ['a', ['b', 'c'], ['d', ['e', 'f']], 'g'] ideal_function(multi_list, 'f') >>> True

In practice, the embedding probably won't be deeper than about 5, [[[[['a']]]]] , if that makes any difference (although in principle any number is possible).

Like BeefofAges said, flatten and search is the easiest way. I use the following code to flatten:

code:

def flatten(my_list):
    for i in my_list:
        if type(i) == type([]):
            for s in flatten(i):
                yield s
        else:
            yield i

which you can use to generate a flattened list and use the regular if item in list search:

code:

multi_list = ['a', ['b', 'c'], ['d', ['e', 'f']], 'g']
final = list(flatten(multi_list))

print repr(final)

if 'a' in final: print "Found 'a'"

"""Output:
['a', 'b', 'c', 'd', 'e', 'f', 'g']
Found 'a'
"""

I assume this would eat it and die pretty horribly if you had a large enough nested list as it'd just eat up your stack space, but that would only be if you had a list with thousands of levels to recurse through.

The Gripper fucked around with this message at 10:34 on Jan 12, 2012

# ? Jan 12, 2012 10:03

Quebec Bagnet: Apr 28, 2009; mess with the honk
you get the bonk; Lipstick Apathy

No Safe Word posted:

If RTF would be suitable, how about HTML?

I went and double-checked the requirements, and unfortunately, it turns out it has to be DOCX or RTF. RTF is something I could probably generate myself, but I'd like to avoid reinventing any wheels I don't need to.

# ? Jan 12, 2012 20:17

FoiledAgain: May 6, 2007

Thanks Beefofages and The Gripper. That's exactly what I was looking for!

# ? Jan 13, 2012 01:17

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:24 on Mar 13, 2017

# ? Jan 15, 2012 01:48

The Gripper: Sep 14, 2004; i am winner

GregNorc posted:

So I modified this code from stack overflow to try and automate sending some emails for a project that needs automated.

Eventually the script will pull all these things (subject, adressees, etc) from command line arguments, which are supplied by another program. It'll make my life a lot easier to automate sending certain emails. But I can't get the script to work.

I need BCC functionality, and so I tried to add it in (without much success)...

code:

from smtplib import SMTP
import datetime

debuglevel = 1

smtp = SMTP()
smtp.set_debuglevel(debuglevel)
smtp.connect('mailrelay.iu.edu', 465)
smtp.login('removed', 'removed')

from_addr = "Greg Norcie <gnorcie@indiana.edu>"
to_addr = "test@norcie.com"
cc_addr = "cc@norcie.com"
bcc_addr = "bcc@norcie.com"

subj = "hello"
date = datetime.datetime.now().strftime( "%d/%m/%Y %H:%M" )

message_text = "JUST A TEST MOVE ALONG PEOPLE"
msg = "From: %s\nTo: %s\n CC: %s\nBCC: %s\nSubject: %s\nDate: %s\n\n%s" % ( from_addr, to_addr, cc_addr, bcc_addr,subj, date, message_text)

smtp.sendmail(from_addr, to_addr, msg)
smtp.quit()
[\code]

When I run the above code (with my username and password added in) I get the following error:

[code]
140-182-219-186:code gregnorc$ python ./emailer.py 
connect: ('mail-relay.iu.edu', 465)
connect: (465, 'mail-relay.iu.edu')
Traceback (most recent call last):
  File "./emailer.py", line 8, in <module>
    smtp.connect('mail-relay.iu.edu', 465)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 296, in connect
    (code, msg) = self.getreply()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/smtplib.py", line 337, in getreply
    line = self.file.readline()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/socket.py", line 404, in readline
    data = self._sock.recv(self._rbufsize)
socket.error: [Errno 54] Connection reset by peer

I'm pretty stumped, does anyone have any ideas how to fix this? I need to be able to send emails with the recipients bcced.

Does it work at all without your BCC added in? For BCC in a work application we just have to_addr = to_addr + bcc_addr, and omit the BCC completely from headers since the SMTP app doesn't care about them anyway (and I believe this is how it should be done as per spec).

So I guess change your code to:

code:

...
message_text = "JUST A TEST MOVE ALONG PEOPLE"
msg = "From: %s\nTo: %s\n CC: %s\nSubject: %s\nDate: %s\n\n%s" % ( from_addr, to_addr, cc_addr, subj, date, message_text)

to_addr = to_addr + bcc_addr

smtp.sendmail(from_addr, to_addr, msg)
...

That said, this shouldn't have caused a connection reset by peer error so there may be something else that isn't working as expected.

# ? Jan 15, 2012 02:47

Jonnty: Aug 2, 2007; The enemy has become a flaming star!

The Gripper posted:

Like BeefofAges said, flatten and search is the easiest way. I use the following code to flatten:
code:
def flatten(my_list):
    for i in my_list:
        if type(i) == type([]):
            for s in flatten(i):
                yield s
        else:
            yield i
which you can use to generate a flattened list and use the regular if item in list search:
code:
multi_list = ['a', ['b', 'c'], ['d', ['e', 'f']], 'g']
final = list(flatten(multi_list))

print repr(final)

if 'a' in final: print "Found 'a'"

"""Output:
['a', 'b', 'c', 'd', 'e', 'f', 'g']
Found 'a'
"""
I assume this would eat it and die pretty horribly if you had a large enough nested list as it'd just eat up your stack space, but that would only be if you had a list with thousands of levels to recurse through.

Instead of using type(), you should probably just put each attempt to look through an element into a try...catch block to cover every iterable object. (Of course this "breaks" if you have any infinite lists, so, uh, watch out for that haha.)

# ? Jan 15, 2012 11:59

Opinion Haver: Apr 9, 2007

Jonnty posted:

Instead of using type(), you should probably just put each attempt to look through an element into a try...catch block to cover every iterable object. (Of course this "breaks" if you have any infinite lists, so, uh, watch out for that haha.)

Just make sure your lists are finite before you iterate over them :v:

# ? Jan 15, 2012 12:42

duck monster: Dec 15, 2004

The Gripper posted:

That said, this shouldn't have caused a connection reset by peer error so there may be something else that isn't working as expected.

Not necessarily. A lot of MTAs will plain out hangup if they whiff anything unusual as an anti spam measure.

If you have access to the MTAs logs, I suggest having a read of them for insight.

# ? Jan 15, 2012 14:39

PotatoJudge: May 22, 2004; Tell me about the rabbits, George

I'm not a Python user, but I'm trying to make use of someone else's script.

https://github.com/wellsoliver/py-retrosheet/blob/master/retrosheet.py

It downloads baseball stats and uses sqlalchemy to shove the data into a database. When I run the script it tells me it can't connect to the database.

I've got python2, and the python2 version of sqlalchemy installed and I wrote a little hello world that reported the sqlalchemy version and it worked just fine. The database is configured and works as well, I use it all the time for other projects.

What do I need to change in the

code:


try:
    db = sqlalchemy.create_engine(string)
    conn = db.connect()
except:
    print 'Cannot connect to database'
    raise SystemExit

to get it to give me a useful error message so I can figure out why it isn't connecting.

The script gets the connection information from a db.ini file.

https://github.com/wellsoliver/py-retrosheet/blob/master/db.ini

I'm using postgres, I've tried using localhost and localhost:5432 in the host section, and I've tried it with the user and pass sections commented and uncommented.

# ? Jan 15, 2012 21:28

Reformed Pissboy: Nov 6, 2003

PotatoJudge posted:

What do I need to change in the
code:
try:
    db = sqlalchemy.create_engine(string)
    conn = db.connect()
except:
    print 'Cannot connect to database'
    raise SystemExit
to get it to give me a useful error message so I can figure out why it isn't connecting.

If you want the details of why the try: block is failing, you can simply remove the try... except and let the code throw an exception:

code:

db = sqlalchemy.create_engine(string)
conn = db.connect()

Right now what it's doing is catching any error that happens, printing a generic "cannot connect" message, and re-raising a generic error. (This is generally considered bad practice.) Without the try:, whatever error is happening won't get handled and it'll print an ugly, but informative, traceback.

# ? Jan 15, 2012 22:37

Scaevolus: Apr 16, 2007

PotatoJudge posted:

I'm not a Python user, but I'm trying to make use of someone else's script.

https://github.com/wellsoliver/py-retrosheet/blob/master/retrosheet.py

It downloads baseball stats and uses sqlalchemy to shove the data into a database. When I run the script it tells me it can't connect to the database.

I've got python2, and the python2 version of sqlalchemy installed and I wrote a little hello world that reported the sqlalchemy version and it worked just fine. The database is configured and works as well, I use it all the time for other projects.

What do I need to change in the
code:
try:
    db = sqlalchemy.create_engine(string)
    conn = db.connect()
except:
    print 'Cannot connect to database'
    raise SystemExit
to get it to give me a useful error message so I can figure out why it isn't connecting.

The script gets the connection information from a db.ini file.

https://github.com/wellsoliver/py-retrosheet/blob/master/db.ini

I'm using postgres, I've tried using localhost and localhost:5432 in the host section, and I've tried it with the user and pass sections commented and uncommented.

SQLite should work fine for this (it's a database for yourself, right?), try changing the db.ini to use it instead of postgres. http://www.sqlalchemy.org/docs/core/engines.html

# ? Jan 15, 2012 22:41

PotatoJudge: May 22, 2004; Tell me about the rabbits, George

Harm Barn Gumshoe posted:

If you want the details of why the try: block is failing, you can simply remove the try... except and let the code throw an exception:
code:
db = sqlalchemy.create_engine(string)
conn = db.connect()
Right now what it's doing is catching any error that happens, printing a generic "cannot connect" message, and re-raising a generic error. (This is generally considered bad practice.) Without the try:, whatever error is happening won't get handled and it'll print an ugly, but informative, traceback.

Thanks! If I read the error message right sqlalchemy called to another module, psycopg2. Installed it and now the script is off and running.

# ? Jan 15, 2012 22:52

duck monster: Dec 15, 2004

Its why i don't use try except for pretty much anything ever, except for cleaning up database vomit if something goes wrong.

If it wants to crash, it *should* crash.

# ? Jan 15, 2012 23:35

maskenfreiheit: Dec 30, 2004

Edit: doublepost

maskenfreiheit fucked around with this message at 01:24 on Mar 13, 2017

# ? Jan 16, 2012 04:07

The Gripper: Sep 14, 2004; i am winner

Oh.

Just noticed you're using the secure SMTP port, but the regular SMTP class. So it's not authenticating because it's expecting it to be SSL secured. Try this:

code:

from smtplib import SMTP_SSL
import datetime

debuglevel = 1

smtp = SMTP_SSL()
smtp.set_debuglevel(debuglevel)
smtp.connect('mail.tpg.com.au')
smtp.login('user, 'pw')

from_addr = "User Name <user@abc.net.au>"
to_addr = ['to@abc.net.au']
cc_addr = ['cc@abc.net.au']
bcc_addr = ['bcc@other.net.au']

subj = "hello"
date = datetime.datetime.now().strftime( "%d/%m/%Y %H:%M" )

message_text = "JUST A TEST MOVE ALONG PEOPLE"
msg = "From: %s\nTo: %s\r\nCC: %s\r\nSubject: %s\r\nDate: %s\r\n%s" % ( from_addr, ",".join(to_addr), ",".join(cc_addr), subj, date, message_text)

to_addr = to_addr + cc_addr + bcc_addr

smtp.sendmail(from_addr, to_addr, msg)
smtp.quit()

That works for me, message accepted for delivery and received by all recipients, with BCC recipient hidden from view. Without using SMTP_SSL I was getting the same error as you, and if you don't need it secured you can use regular SMTP again but don't specify the port (or specify port 25).

The Gripper fucked around with this message at 06:31 on Jan 16, 2012

# ? Jan 16, 2012 06:09

thepedestrian: Dec 13, 2004; hey lady, you call him dr. jones!

edit: should have reloaded page first, disregard.

# ? Jan 16, 2012 15:49

Hed: Mar 31, 2004; Fun Shoe

I am connecting to a socket service to query for data and I want to cache answers especially while I'm verifying everything is working, but even when this thing is finished. My first thought was to write query responses to a sqlite db on answer and add an "expires-on" datetime in the future. Does this make sense or should I be doing something lighter weight still?

# ? Jan 16, 2012 18:09

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

Do you want this cache to persist after your application exits, to use the same data on subsequent runs?

If not, :love:

functools.lru_cache

# ? Jan 16, 2012 18:21

M.C. McMic: Nov 8, 2008; The Weight room
Is your friend

I was previously learning C#/ASP.NET, but I changed jobs, and now I think it might be more useful to learn Python in my new position (not *necessary*, but it would definitely be looked upon favorably). Anyway, I'm excited about learning it regardless.

Should I stick to the online tutorials and documentation to start out? Any books I need to have as a reference? I read the OP, but it's over 3 years old now (edit: but it has probably been updated since then, durr). :downs:

# ? Jan 16, 2012 19:31

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

M.C. McMic posted:

I was previously learning C#/ASP.NET, but I changed jobs, and now I think it might be more useful to learn Python in my new position (not *necessary*, but it would definitely be looked upon favorably). Anyway, I'm excited about learning it regardless.

Should I stick to the online tutorials and documentation to start out? Any books I need to have as a reference? I read the OP, but it's over 3 years old now (edit: but it has probably been updated since then, durr).

I learned programming by learning Python and I just used online resources.

The docs + How To Think Like a Computer Scientist + Google + doing some projects was pretty much all I used. I think people are recommending Learn Python The Hard Way nowadays....

# ? Jan 16, 2012 20:59

ufarn: May 30, 2009

Do a project that interests you and which you would like to finish.

Join #python on freenode and use the requests instead of crappy urllib2.

# ? Jan 16, 2012 21:09

maskenfreiheit: Dec 30, 2004

Edit: Double Post

maskenfreiheit fucked around with this message at 20:36 on Mar 13, 2017

# ? Jan 16, 2012 22:20

ynohtna: Feb 16, 2007; backwoods compatible; Illegal Hen

One the servers along the way is failing to route the domain name of one of the addresses you're supplying. Google "smtp 5.1.2" for a better explanation.

# ? Jan 16, 2012 22:35

maskenfreiheit: Dec 30, 2004

Edit: Double Post

maskenfreiheit fucked around with this message at 20:36 on Mar 13, 2017

# ? Jan 16, 2012 22:40

ynohtna: Feb 16, 2007; backwoods compatible; Illegal Hen

Ah, off the top of my head it looks like issues with Python strings being Unicode whereas SMTP is an old 7-bit protocol and interpreting the embedded zero bytes as terminators.

Try applying .encode("utf8") to your strings.

http://docs.python.org/howto/unicode.html

Sorry for the vague speculative response - should really be working on the bugs in my own Python code!

# ? Jan 16, 2012 23:08

Hed: Mar 31, 2004; Fun Shoe

Lysidas posted:

Do you want this cache to persist after your application exits, to use the same data on subsequent runs?

If not, functools.lru_cache

Wow that is a really neat tool! Yes actually I would like it to persist between runs. This is a great example though because my next question was going to be asking about using a decorator to extend this ability to a few classes of the same type. I will look at shelve and trying to do something like this.

# ? Jan 17, 2012 03:34

The Gripper: Sep 14, 2004; i am winner

GregNorc posted:

I updated my post with some more details... I think the error is on my end.

Edit: I'll just C/P the relevant stuff here for easier reading:

I tried messing around a bit. If I only put something in the "to" field, this is what happens:
http://i.imgur.com/gRrpq.png

Look how it tried to mail each _characted_ of the email to me? so g@mailserver.com, n@mailserver.com... and so on and so forth?

I attached the current version of my code here (with my username/password omitted): http://pastebin.com/VtQK0mT7

I'm not too familiar with python's syntax, and actually inherited this particular program, so I'm a bit lost on how to make this work... I'm guessing it's concatinating the string in some weird way? I think if I can fix that concatenation, I'll have the app working...

Yeah I think it's just the concatenation. In my code I had each of the to_addr, cc_addr and bcc_addr as lists e.g. to_addr = ['address1@domain.com', 'address2@domain.com'] so the line to_addr = to_addr + cc_addr + bcc_addr was adding lists together. Yours is concatenating the strings to toaddress@domain.comccaddress@domain.com (which is why one of your domains is whatever@indiana.educc as the cc is from your cc_addr recipient).

Try just changing it to this:

code:

to_addr = ["gnorcie@indiana.edu"]
cc_addr = [""]
bcc_addr = [""]

That should sort the problem out.

vv Twice!

The Gripper fucked around with this message at 05:25 on Jan 17, 2012

# ? Jan 17, 2012 05:15

todesschaf: Nov 6, 2006; Sheep of Death!

GregNorc posted:

I updated my post with some more details... I think the error is on my end.

Edit: I'll just C/P the relevant stuff here for easier reading:

I tried messing around a bit. If I only put something in the "to" field, this is what happens:
http://i.imgur.com/gRrpq.png

Look how it tried to mail each _characted_ of the email to me? so g@mailserver.com, n@mailserver.com... and so on and so forth?

I attached the current version of my code here (with my username/password omitted): http://pastebin.com/VtQK0mT7

I'm not too familiar with python's syntax, and actually inherited this particular program, so I'm a bit lost on how to make this work... I'm guessing it's concatinating the string in some weird way? I think if I can fix that concatenation, I'll have the app working...

Make your to_addr, etc lists of strings instead of bare strings, that's what smtplib seems to expect. The reason it tries each letter is it just assumes if the input is iterable then it must be a proper list. Too bad strings also happen to be iterable.

edit: beaten, dammit.

# ? Jan 17, 2012 05:18

the: Jul 18, 2004; by Cowcaster

Just informed by a research professor that I'll be using python for astrophysics simulations, so I guess I'll be posting in here a lot

# ? Jan 17, 2012 18:50

maskenfreiheit: Dec 30, 2004

Edit: Double Post

maskenfreiheit fucked around with this message at 20:36 on Mar 13, 2017

# ? Jan 17, 2012 21:19

The Gripper: Sep 14, 2004; i am winner

Running that same code on my system (Python 2.7.1) doesn't throw any error like that, just the expected SMTPAuthenticationError from the dummy username and password. That's from purely copying and pasting what you put on pastebin.

It's happening at the smtp.login() call, but nothing you've changed should have affected that. Maybe some kind of unicode/non-ascii character in your password is bonering it up, but I don't know how that would have snuck in.

It could also be the server being a dick and just dropping the connection when you identify yourself, anti-spam or whatever.

Sorry for the half-assed answer, but it's one of those problems that is frustratingly hard to pinpoint without being able to reproduce

# ? Jan 17, 2012 21:53

maskenfreiheit: Dec 30, 2004

Edit: Double Post

maskenfreiheit fucked around with this message at 20:36 on Mar 13, 2017

# ? Jan 17, 2012 22:09

The Gripper: Sep 14, 2004; i am winner

GregNorc posted:

My password has a space in it (weird institutional policy)
Could that cause the issue?

Maybe you could throw your code up on pastebin (less your username/pw) and I could diff it, maybe I introduced something weird somewhere...

Space should be no problem.

http://pastebin.com/0hinLuGm

That's the code I just tested with destination emails and username+pw removed, everything under the smtp.login() call shouldn't make any difference since the error you were getting was there.

Debug output is the same as yours up to the send: "ehlo...." line, only it follows through until the message is sent and closed with a final retcode(221) connection closed message.

Edit; if you try just logging into gmail's smtp server (smtp.gmail.com) with any login or password, does it give the same error? If it doesn't then it's a pretty good sign that your institutions relay/smtp server is doing something weird to you.

The Gripper fucked around with this message at 22:25 on Jan 17, 2012

# ? Jan 17, 2012 22:19

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

I'm trying to tie together .NET and native environments as much as possible. I am more familiar with IronPython, but I understand there's an alternative Python for .NET. IronPython puts the engine in the .NET runtime, whereas I believe Python for .NET is the CPython engine with hooks into .NET. Unfortunately I need both .NET and native stuff, and some COM just to really screw things up. I'm trying to figure out what options I have for making this as painless as possible.

An example of my woes: I'm trying to get Ironclad to load into IronPython because it claims to cover a lot of ctypes stuff. I can't get it to load into the .NET 4.0 runtime by default. I might figure it out; there's some app.config variable I think I can turn on to make it happy, or I might be able to recompile it. Still, that's own problem of many.

# ? Jan 18, 2012 05:50

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

Has anyone in the history of the universe convinced pyodbc to compile on CygWin?

Rocko Bonaparte posted:

I'm trying to tie together .NET and native environments as much as possible. I am more familiar with IronPython, but I understand there's an alternative Python for .NET. IronPython puts the engine in the .NET runtime, whereas I believe Python for .NET is the CPython engine with hooks into .NET. Unfortunately I need both .NET and native stuff, and some COM just to really screw things up. I'm trying to figure out what options I have for making this as painless as possible.

An example of my woes: I'm trying to get Ironclad to load into IronPython because it claims to cover a lot of ctypes stuff. I can't get it to load into the .NET 4.0 runtime by default. I might figure it out; there's some app.config variable I think I can turn on to make it happy, or I might be able to recompile it. Still, that's own problem of many.

This claims that the PyWin32 project adds COM interop to the CPython windows build, so you might just be able to use 'normal' Python if that works.

# ? Jan 18, 2012 18:58

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 17:47

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

Munkeymon posted:

This claims that the PyWin32 project adds COM interop to the CPython windows build, so you might just be able to use 'normal' Python if that works.

After a day and a half of IronPython self-destruction, I'm going to try from the CPython side with Python for .NET. If that's the case, I think there are generally more tools for COM stuff. But I don't know what's any good with COM; I usually stay the hell away from it. I'll try this out shortly.

# ? Jan 18, 2012 22:15

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »