Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

UberJumper: May 20, 2007; woop

Using ftplib is there anyway to resume binary file uploads?

I've looked and looked, and i can't find anything. REST doesn't seem to work, when uploading to the server. Storing binary on the server, always causes it to start writing from the beginning of the file again.

# ¿ Oct 6, 2008 22:18

Adbot: ADBOT LOVES YOU

# ¿ May 20, 2024 02:50

UberJumper: May 20, 2007; woop

Does anyone know how exactly comtypes rips out all the CLSID information out from COM automon files?

# ¿ Jan 21, 2010 07:47

UberJumper: May 20, 2007; woop

Does anyone have any good suggestions and recommendations for good practices when creating python packages?

Basically i am tasked with combining a dozen or so modules, a lot of them have overlapping functionality, in some cases certain functions have just been copied and pasted.

The other important part is, certain methods have 2-3 different ways of doing certain things depending on what modules currently exist in the system. For example if the user has python 2.4 then generally they will have win32com, however in 2.5 win32com is not installed by default with ArcGIS python install, so we write a method of doing the same thing via ctypes.

So there is alot of:

code:

if user_has_win32com:
  do_win32_foo()
elif user_has_ctypes:
  do_ctypes_foo()
else:
  do_something_else()

A lot of the methods all need to keep and interact with global variables, there is always a reference to the geoprocessor, user's toolbox path, arcgis toolbox path, etc. Most of these variables are populated generally when the module is imported.

So basically this is what i am looking at right now:

code:

foo package\
  __init__.py (populate global variables, and figure out what the user has installed on their system)
  common.py (all of the global variables used within the package)
  constants.py (all of the constants used)
  error.py (exceptions, used by this package, along with some error handling routines)
  log.py (custom logging handlers)
  path.py (basically versions of os.path abspath, exists, etc, these basically do some extra work for arcgis geoprocessor specific api stuff)
  _gp.py (geoprocessor specific functions, these would be imported into __init__.py, so the user can simply do foo.<some gp function>)

But what i am wondering is if its a good idea to simply make the _gp.py into a sub package, and split it off, into categories, since i don't really want to put 60 odd random functions together. Generally these would most likely be categorized by what type, so we would have foo.gp.raster, for all the raster methods. However i kinda feel like thats going to get irritating for anyone to use. Since we have to use the full namespace path ( :cry:

) due to internal reasons.

Also the second thing i also am tasked with, is that we want these modules to have a debugging logger. Such that certain key bits of information is stored. I know i can simply use

code:

_logger = logging.getLogger(__name__)

However that does not seem to work well, since i basically want to add all the loggers as children of a common logger. E.g. such that foo.path uses the logger name foo.path. Any suggestions for this?

Can anyone shed some light on good practices for packages, and is there anything terribly wrong with my design?

# ¿ Feb 16, 2010 17:35

UberJumper: May 20, 2007; woop

Avenging Dentist posted:

Why not just specify a dependency on the ctypes distribution when you're in Python 2.4? If you're using setuptools, it's as easy as adding install_requires = ['ctypes'] to your setup call (when running Python 2.4).

Didn't realize that i'll take a look at it. The ctypes stuff is pretty much a very small user group, who does not have pywin32 installed. I'll take a look tho.

quote:

More to the point though, this should happen only once per file (if at all). What's the point in doing this for every function/function call? It just makes code hard to read. Do this:
code:
if user_has_win32com:
   def foo(): pass
elif user_has_ctypes:
   def foo(): pass
else:
   def foo(): pass

Thats awesome i love it.

quote:

Why is there a root-level __init__.py? That doesn't really even make sense.

I thought by default all python packages require an __init__.py file, in the root of the package?

# ¿ Feb 16, 2010 19:15

UberJumper: May 20, 2007; woop

Avenging Dentist posted:

The root of the package, yes (i.e. foo_package/__init__.py). Not in the directory containing the package directory. It's like index.html for a website.

Blah thanks i think i just screwed up my example.

# ¿ Feb 16, 2010 20:05

UberJumper: May 20, 2007; woop

Quick question bout numpy.

If i have a matrix

code:

matrix([[0, 1, 2, 3, 4],
        [0, 1, 2, 3, 4],
        [0, 1, 2, 3, 4],
        [0, 1, 2, 3, 4],
        [0, 1, 2, 3, 4]])

How can i easily zero the column at 2 and the row at 2? So the matrix becomes:

code:

matrix([[0, 1, 0, 3, 4],
        [0, 1, 0, 3, 4],
        [0, 0, 0, 0, 0],
        [0, 1, 0, 3, 4],
        [0, 1, 0, 3, 4]])

# ¿ Feb 28, 2010 19:36

UberJumper: May 20, 2007; woop

Sock on a Fish posted:

I'm encountering a very strange problem printing a simple text report. I've got some headers, and then some characters that I want inserted under the headers. Here's the code, with some extra stuff thrown in to help me debug:
code:
fmt_string = '%20s%20s%10s%5s%14s%11s'
headers = ('last_name','first_name','chapter','paid','meal_choice','checked_in')
header_underscores = []
for h in headers:
	header_underscores.append('-' * len(h))

print headers
print header_underscores

print len(headers)
print len(header_underscores)

print fmt_string % headers
print fmt_string % header_underscores
Here's the output:
code:
('last_name', 'first_name', 'chapter', 'paid', 'meal_choice', 'checked_in')
['---------', '----------', '-------', '----', '-----------', '----------']
6
6
           last_name          first_name   chapter paid   meal_choice checked_in
Traceback (most recent call last):
  File "poop.py", line 14, in <module>
    print fmt_string % header_underscores
TypeError: not enough arguments for format string
What the hell?

edit : never mind, the problem is that for these operations tuples and lists aren't interchangeable for some reason

This fixes it.

code:

print fmt_string % tuple(header_underscores)

# ¿ Feb 28, 2010 21:21

UberJumper: May 20, 2007; woop

tripwire posted:

To set the first row:
matrix[0, ] = 0

To set the first column:

matrix[ ,0] = 0

Numpy. Is Pure. Awesome.

# ¿ Feb 28, 2010 23:10

UberJumper: May 20, 2007; woop

Is there any decent library for basically doing 2d geometry? Aside from shapely? I keep running into tiresome issues with checking to see if polygons intersect, using a line to splice a polygon, etc.

# ¿ Mar 25, 2010 22:22

UberJumper: May 20, 2007; woop

Does anyone know what can be causing duplicate logging messages to be outputted?

code:

[DEBUG   ] 20100402 21:01:07 | auxpy.util.gen_generator | Interface Found
DEBUG:auxpy.util.gen_generator:Interface Found

In the auxpy.__init__.py i declare the root logger as

code:

_logger = logging.getLogger(__name__)
_logger.addHandler(NullHandler())

then i have a simple test file that changes this value into a stream logger for debugging.

Ive checked and all the loggers use, __name__ to get the logger.

Is there anyway to check this?

# ¿ Apr 3, 2010 02:12

UberJumper: May 20, 2007; woop

BeefofAges posted:

Try checking how many handlers that logger has. Each handler will do its own log output. You can try printing _logger.handlers

Bah. i found it, i was calling logging.debug instead of _logger.debug. Why would they make logging like that :psyduck:

# ¿ Apr 3, 2010 02:33

UberJumper: May 20, 2007; woop

Okay so i am more or less wondering about something.

For my final Geomatics project, we were tasked with basically creating something to add functionality to the ArcGIS tool suite, or improve on something. E.g. basically ether make a cool tool or write a library. I ended up going a completely different route and basically have been working on porting making ArcObjects work decently from within python through comtypes. (i've done alot of arcobjects programming and using their geoprocessor pains me greatly).

However since ESRI's COM library is buggy as gently caress / broken / :suicide:

for certain things. Comtypes runs into a lot of issues with properly generating modules for ArcObjects, e.g. iterable interfaces are not being treated as iterable, even though ESRI says they should be. So its a pretty big mishmash of stuff. Now to top it off, comtypes is not very customizable, and basically doesn't allow you to externally modify certain things. So what i was wondering, is it a bad pratice to basically copy the comtypes package, rename it so something like _comtypes, update all references within that package and merge it into a larger package? Then just edit the living hell out of it directly, and make it more or less work only for ArcObjects?

A few people have approached or messaged me about being interested in getting a copy of when i get it into a workable state. So i was thinking of just putting it up on google code or codeplex. I have tried contacting the comtypes dev's a few times about asking them if i can just modify comtypes. I only got one response, which was basically "uhmm no". My duedate is coming up, and i pretty much am just going to go ahead with it.

So basically what i am asking is, if i did the above would:
a. comtypes people get pissy, and can they basically tell me to take it down?
b. I am not giving out the com type libs, since you need a valid arc install to generate the bindings. Does this really count as reverse engineering?
c. In the TOS ESRI States that you cannot use ArcObjects from any non ESRI approved
languages. However ESRI gives users a python library, just a very limited one.

Is there anything i really should be weary of doing?

# ¿ Apr 7, 2010 02:30

UberJumper: May 20, 2007; woop

Scaevolus posted:

It doesn't really matter. It's distributed under the MIT License, so they can't prevent you from modifying it to work for your particular case.

thanks!

# ¿ Apr 7, 2010 16:32

UberJumper: May 20, 2007; woop

Thermopyle posted:

Concise method of creating a new list from every nth item of a list?

I seem to write multiple lines of code for this and it seems kinda ridiculous.

Example:

Every 3rd item from list a makes list b.

a = [1, 2, 3, 4, 5, 6, 7, 8, 9]
b = [3, 6, 9]

b = [x for x in a[::3]]

# ¿ Apr 9, 2010 03:10

UberJumper: May 20, 2007; woop

loving comtypes. If anyone has any ideas to how to deal with these things i would absolutely love to know.

1. There is a handful of interfaces/coclasses that are not being picked up by comtypes typelib parser. Now i thought this was simply because of me monkey patching / reworking chunks of their automatic code generation. However after spending a night debugging / struggling i pretty much found out they just are not getting extracted from the typelib. However visual studio / some other random tools reports they are there. Looking through the source of how comtypes makes me want to :suicide:

.

2. Comtypes appears in certain situations to be doing a double release. This occurs kinda at random and i cannot nail this down to a single problem, however i am pretty sure it has to do with me overloading __new__ for interfaces, and calling the baseclass new. However this seems to be somewhere screwing with all the bullshit contained in _cominterface_meta.

3. I still have not found a a good way to handle COMMethod, since in certain cases i would love for the user to be able to get the HRESULT in certain situations. Like an optional parameter that takes an HRESULT object, and fills in the values. Right now anything that is S_XXX, just gets tossed away. However when HRESULT tells you information you need that is not an error, your hosed.

Looking through the code i noticed you can add a private method that instead will be called. E.g.:

IRaster::GetPixelCoordX(__in row, __in col, __out output_value )

code:

class IRaster(IUnknown):
    def GetPixelCoordX(row, col):
       output_value = c_int()
       hr = self.__com_GetPixelCoordX(row, col, byref(output_value))
       return hr, output_value.value

That works except, generating a stub for every single corresponding method, when the file size is already so stupidly large. This becomes a major hassle. Attempting to modify COMMETHOD, is a nightmare....

Also why would you ever name a class like this...?

code:

class _(partial.partial, POINTER(p)):

# ¿ Apr 29, 2010 15:56

UberJumper: May 20, 2007; woop

Is there anyway to make a python list iterator to go backwards?

Basically i have this

code:

    class IterTest(object):
        def __init__(self, data):
            self.data = data
            self.__iter = None
    
        def all(self):
            self.__iter = iter(self.data)
            for each in self.__iter:
                mtd = getattr(self, type(each).__name__)
                mtd(each)
    
        def str(self, item):
            print item
    
            next = self.__iter.next()
            while isinstance(next, int):
                print next
                next = self.__iter.next()
    
        def int(self, item):
            print "Crap i skipped C"
    
    if __name__ == '__main__':
        test = IterTest(['a', 1, 2,3,'c', 17])
        test.all()

Running this code results in the output:

a
1
2
3
Crap i skipped C

I know why it gives me the output, however is there a way i can step backwards in the str() method, by one step?

# ¿ May 5, 2010 23:24

UberJumper: May 20, 2007; woop

Scaevolus posted:

No. Why not do this?

code:

    class IterTest(object):
        def __init__(self, data):
            self.data = data
    
        def all(self):
            for each in self.data:
                mtd = getattr(self, type(each).__name__)
                mtd(each)
    
        def str(self, item):
            print item
    
        def int(self, item):
            print item
    
    if __name__ == '__main__':
        test = IterTest(['a', 1, 2,3,'c', 17])
        test.all()

Because that makes grouping subelements difficult. Essentially i am looking for bidirectional iteration support within python.

# ¿ May 6, 2010 00:10

UberJumper: May 20, 2007; woop

Quick question what is everyone using for an IDE? I am mainly looking for something with intellisense and a debugger.

In the past i have Tried:
Komodo
- Its awesome, but its too expensive for a university student (like seriously a student gimped license still costs several hundred dollars?)
Pyscripter
- crashes often / intellisense simply does not work for anything complex / debugger doesn't really work well
PyDev
- ohgodihateeclipse.

Has anyone given PyCharm a shot?

# ¿ May 9, 2010 14:24

UberJumper: May 20, 2007; woop

quote:

<lots of responses about python ide>

Well i decided to just try all the other ones i could find :eng101:

, that met my requirements. Now i don't really care for VI/Emacs bindings and such. I am trying to basically find an idea that does debugging and auto completion, and something that i can actually afford.

[WingIDE Professional (Trial)]
Pros:
- Worked pretty much flawlessly, everything felt extremely professional and well done
Cons:
- Slow/Choppy for no apparent reason
- Debugger dies on me(well it comes back if i wait long enough ~3 minutes or so), the program pauses, then whole debug panel basically becomes non responsive. I am guessing its trying to inspect all the elements in the current frame, and is choking on the amount of objects and such.
- certain modules in my site-package it just refuses to generate intellisense.

[Netbeans]
Pros:
- Its nice that it basically follows PEP's style guide for working with intellisense.
- Looks pretty and is pretty clean interface...? (i'm digging)
Cons:
- Slow, crashes, does random things, refuses to hit certain breakpoints for no loving reason.
- gently caress it.

[PyCharm]
Pros:
- Free right now (until beta is over)
- Looks good
- Intellisense worked perfectly
- pretty light memory footprint
- Lots of little things that are great ideas
Cons:
- It generated 5 gigs in cache files, :suicide:

- Some of their random context sensitive intellisense is nags the living gently caress of me, even after turning it off. It still nags me.
- Other little gripes.
- Debugger hangs randomly.

But right now basically i think i am going to stick with PyCharm, i've had the best experience so far with it.

# ¿ May 11, 2010 21:18

UberJumper: May 20, 2007; woop

Might be a really dumb question, is there anyway to build a python package from source, on windows when i don't have visual studio 2003? Or mingw? I have visual studio 2008, and its the only version i can get my hands on. I looked around online but couldn't find anything definate. Generally i like to just stick to the precompiled binaries to get around this headache, however i don't have that option for this library (http://effbot.org/media/downloads/ftpparse-1.1-20021124.zip).

I can't get my hands on visual studio 2003. If i were to use mingw, would the egg be portable to another computer where mingw is not installed, or would i have to install mingw on that computer?

# ¿ Sep 30, 2010 18:12

UberJumper: May 20, 2007; woop

tripwire posted:

You could use cygwin.

Turns out setting up mingw for python is insanely painless now:

http://www.develer.com/oss/GccWinBinaries

# ¿ Sep 30, 2010 20:17

UberJumper: May 20, 2007; woop

tripwire posted:

Intellij Idea may be a bit slow and clunky, but it's light years ahead of anything that exists for python. gently caress eclipse though

They also have a python IDE that finally left beta.

http://www.jetbrains.com/pycharm/

Its probably the best python IDE out there, imo. I switched from Komodo to Pycharm and haven't looked back.

# ¿ Oct 12, 2010 07:40

UberJumper: May 20, 2007; woop

Lurchington posted:

PyCharm cool thing of the day: VCS (we use mercurial) history of the active file your editing.

'who messed up my poo poo?'
*check the history*
'hmm, sammy jones in changeset 63, that bastard'

Yeah their VCS stuff is pretty drat nifty. The only major issue i have left with pycharm is:

- Debugger seems to have problems with hitting breakpoints of modules i don't import directly:

code:

<test.py>
import hat.cats

<hat/cats.py>
import hat.common

No break points get hit in hat.common. I know it gets executed. But i want it to break

- It can sometimes bug the living poo poo out of me with its inspections, most of the time its fantastic. "Yes PyCharm i know that line isn't right because i only started typing the first loving letter of it"

But poo poo this is easily the best IDE i have used for python.

Also i have a question for you guys i am trying to wrap our existing hellish ftplib wrapper library ( :suicide:

) into something more Pythonic. Pythonic is the new buzzword at work, since we are porting/rewriting our existing pre 2.0 era python libraries into the much more modern 2.5.4 era (blame arcgis) everything should be Pythonic (<insert something>-like objects, list comphrensions, packages, etc). Its a good idea but its getting to the point where its difficult to make certain things "Pythonic".

Anyways the library does stuff like resuming transfers, partial downloading, and properly expanding, parsing some internal binary formats and lots of other stuff. All in all the library is ~8k lines of code (excluding comments) crammed into half a dozen functions in a single .py file:

I have already done everything else, i have the core of it, its just when it comes to resumable downloads(this gets used alot, ~2GB files we want to resume, and it happens alot). To resume a download, we append a .lok extension to the filename, and only when the transfer is completed do we rename the file to the correct extension. This way if the transfer fails, when we reconnect we see .lok file and just continue the transfer from where it left off.

However i have no idea how to work something like this into a file like object :

code:

### Downloading a binary...
# Open a file on the ftp server in read-binary mode
f = fooftp.open('/hello/pr0n/folder/wtfcaterpie.jpg', 'rb', block_size=8192)
local_file = ?!?!?!
# wander through each of the blocks
for block in f:
   local_file.write(block)

# close
local_file.close()

# send quit to ftp
f.close()

I did doing the following

code:

remote_file = fooftp.open('/hello/pr0n/folder/wtfcaterpie.jpg', 'rb', block_size=8192)
local_file = remote_file.open_local(r'C:\workspace\dog.jpg', 'wb')

This however is considered unpythonic by our code review and was rejected :suicide:

Does anyone have any ideas of how the gently caress i am supposed to make this pythonic? Uploading a file is fine because i can wrap all of the resuming stuff. I feel pretty stupid atm =(

:commissar:

Pythonic

# ¿ Oct 19, 2010 18:28

UberJumper: May 20, 2007; woop

Can someone tell me what is the preferred practice for doing imports inside a package for third party packages?

Which is better:

# Foo/bar.py

code:

import moo

def do_something():
    # lots of code
    hats = moo.cat(....)
    # more code

Or doing this?

Foo/utils.py

code:

from moo import cat

Foo/bar.py

code:

from .utils import cat

def do_something():
    # lots of code
    hats = cat(....)
    # more code

Basically hiding usages of the third party library moo in utils, or using the third party moo explicitly. Some people at work say the first one others say the second one. We already have a pretty big utils sub package that is more or less imported throughout the project.

UberJumper fucked around with this message at 17:38 on Aug 23, 2016

# ¿ Aug 23, 2016 17:15

Adbot: ADBOT LOVES YOU

# ¿ May 20, 2024 02:50

UberJumper: May 20, 2007; woop

tef posted:

it depends

This is good when you're doing something a lot with moo, or cat isn't a clear function

This is good when you're only doing cat and it's an obvious name: from urlparse import urlparse

If you're writing a standalone executable script, from x import * is acceptable, but rarely otherwise.

`
This is good when you might change the moo library later. It's heavyweight when you won't do it. It might not be worth wrapping pytz or requests, but if the API is ugly, a wrapper gives you the chance to make a smaller easier api for the things you need without coupling your code.

It's all a tradeoff.

I tend to

- wrap any big 3rd party module, especially one i'm trying out for size. Often just to set up defaults.
- use import foo over from foo import bar

but usually when i'm writing larger chunks of code, for cheap hacks i'll do whatever

Thanks!

I am not really wrapping the moo library (it is actually a fairly nice library, just has horrible method names). There is just a bunch of really cryptic method names, that are not upto the PEP8 standard (e.g. 'g_cfg_l' which literally means load_config).

It is also a library that almost every single module in our entire package (probably 100+ files), imports. So i have been debating just putting a bunch of "from moo import g_cfg_l as load_config" into utils/__init__.py then in our code just use:

code:

from .utils import load_config

UberJumper fucked around with this message at 18:15 on Aug 23, 2016

# ¿ Aug 23, 2016 17:47

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.