Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Tigren: Oct 3, 2003

Is anyone willing to look through some code for me and give me a critique? This is the first time I've ever built something that's more than a few lines.

I feel like I need a new set of eyes on this to tell me where and what to clean up.

The json file in the repo is a sample output and should render nicely with Github geojson features.

I think my next step is to break out some of the output functions into their own module.

https://github.com/bwainstock/beeradvocate_scraper/blob/master/ba_scraper.py

# ? Dec 8, 2014 05:04

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 12:35

BigRedDot: Mar 6, 2008

Here's your holiday Bokeh 0.7

* super tight ipython widget integration (no bokeh server required!)
* touch ui support finally
* lots of new tools: lasso, poly, point select; crosshair, data table,
* several more charts types

Since SA won't embed vines and I can't be arsed to do better, here are some lovely gfycat gifs

# ? Dec 8, 2014 22:19

Dominoes: Sep 20, 2007

Thanks for making/working on Bokeh - seems pretty cool!

# ? Dec 8, 2014 22:21

Dominoes: Sep 20, 2007

Tigren posted:

Is anyone willing to look through some code for me and give me a critique? This is the first time I've ever built something that's more than a few lines.

I feel like I need a new set of eyes on this to tell me where and what to clean up.

The json file in the repo is a sample output and should render nicely with Github geojson features.

I think my next step is to break out some of the output functions into their own module.

https://github.com/bwainstock/beeradvocate_scraper/blob/master/ba_scraper.py

Seems pretty good.

Minor suggestions from a lazy skim:
-PEP 8 says 2 lines between functions that aren't part of a class. Also says put periods at the end of your docstring sentences.

Type annotations would clarify what your functions are doing, since it's not obvious for some of them. ie:

Python code:

def parse(response_data: list, city: str) -> list:
    ...

You could also clarify in the docstring for the more complex structures you're passing around, like response_data. Your docstrings are general; you have a few complex functions that could use a better description.

if 'fail' not in response.url: - You should add a comment about what would cause 'fail' to be in the url.

Dominoes fucked around with this message at 22:50 on Dec 8, 2014

# ? Dec 8, 2014 22:32

BigRedDot: Mar 6, 2008

Dominoes posted:

Thanks for making/working on Bokeh - seems pretty cool!

No need to thank me, I am super grateful that the opportunity came my way!

# ? Dec 9, 2014 03:43

ShadowHawk: Jun 25, 2000; CERTIFIED PRE OWNED TESLA OWNER

Coin posted:

Yeah, I've heard this before. However, our instructor teaches us to "code our comments": basically we write everything out in plain English before actually coding it. I can see why this might not be helpful in a production environment, but I'd be marked down for not having that stuff.

Here's an example of doing that the right way. Instead of:

code:

for n in mp_affiliation:
	#If the party isn't in the party_size dictionary yet, I want to add it as a key with a value of one.
	if n not in party_size:

Try:

code:

for party in mp_affiliation:
	#If the party isn't in the party_size dictionary yet, I want to add it as a key with a value of one.
	if party not in party_size:

Using a better variable name, the comment becomes totally meaningless (and can thus be deleted).

Also doing this you would have found the bug in your code more obviously (for party in mp_affiliation is actually not the party -- iterating like that gives you the key not the value)

# ? Dec 9, 2014 05:28

SurgicalOntologist: Jun 17, 2004

A really good talk on conventions for naming things: https://www.youtube.com/watch?v=YklKUuDpX5c

# ? Dec 9, 2014 05:43

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

I'm pretty bad (read: green) at knowing how to do things in a way that makes sense in the long run, wondering if there is a good pattern for this.

I'm importing strings from files whose ordering is always the same for each file, but always different between files. So, title, author, date / date, title, author. I want to import all of them and then normalize them so I can display/sort them in a readable way.

The first way that came to mind to accomplish this was to create an expected_order variable for each file and then do something like:

code:

imported_book = ['The Cat in the Hat', 'Dr. Seuss', '1957']
expected_order = ['title', 'author', 'date']

a_book = dict(zip(expected_order, imported_book)

And then running sorted() w/ some key on the whole lot of 'em to get a nice list of dicts that I can do things like:

code:

book_list = [{book1}, {book2}]
order = ['title', 'author', 'date']

def print_books(book_list, order):
    for book in book_list:
        print("{0}, {1}, {2}").format(d[order[0]], d[order[1]], d[order[2]])

Does this sound like I'm falling into any pitfalls or anything?

Also, does anyone have a book recommendation for making me better at thinking about code structure/writing maintainable code? I guess design patterns, project structure, etc? I feel like that is something I've just never read about.

It could also be that I should just buckle down and read through the source for big popular well-written libraries or something if that's what I think I need to do, thoughts on that as an approach?

# ? Dec 9, 2014 06:57

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

*whispers "namedtuple"*

Python code:

from collections import namedtuple

Book = namedtuple('Book', ('title', 'author', 'date'))

imported_book = ['The Cat in the Hat', 'Dr. Seuss', '1957']

a_book = Book(*imported_book)

# yaddayadda

book_list = [book1, book2]

def print_books(book_list):
    for book in book_list:
        print("{0}, {1}, {2}").format(book.title, book.author, book.date)

# ? Dec 9, 2014 08:00

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

Symbolic Butt posted:

*whispers "namedtuple"*

Python code:

from collections import namedtuple

Book = namedtuple('Book', ('title', 'author', 'date'))

imported_book = ['The Cat in the Hat', 'Dr. Seuss', '1957']

a_book = Book(*imported_book)

# yaddayadda

book_list = [book1, book2]

def print_books(book_list):
    for book in book_list:
        print("{0}, {1}, {2}").format(book.title, book.author, book.date)

I was using a dictionary because it allows me to bypass having to reorder the books on/slightly after import but maybe I'm missing something?

Say I have:

file_1 = title, date, author
file_2 = date, title, author
file_3 = author, date, title

The idea of assigning each book an order list on import and then just zipping them up in a dict was that I could avoid having multiple functions to deal w/ rearranging everything.

Would it still be better to use a namedtuple just because book.title is infinitely more readable than [digging[ into[ dicts?]]]

e; I guess I'm just super stupid. I could always just have a single function that takes the same kind of expected_order variable and then just rearranges based on that before assigning it to the namedtuple.

The March Hare fucked around with this message at 15:21 on Dec 9, 2014

# ? Dec 9, 2014 14:45

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Oops I actually thought about it but then forgot to type it out that you can do something like this:

Python code:

a_book = Book(**dict(zip(expected_order, imported_book)))

# ? Dec 10, 2014 03:54

Tigren: Oct 3, 2003

Dominoes posted:

Seems pretty good.

Minor suggestions from a lazy skim:
-PEP 8 says 2 lines between functions that aren't part of a class. Also says put periods at the end of your docstring sentences.

Type annotations would clarify what your functions are doing, since it's not obvious for some of them. ie:
Python code:
def parse(response_data: list, city: str) -> list:
    ...
You could also clarify in the docstring for the more complex structures you're passing around, like response_data. Your docstrings are general; you have a few complex functions that could use a better description.

if 'fail' not in response.url: - You should add a comment about what would cause 'fail' to be in the url.

Thanks for the comments Dominoes.

Are type annotations a thing in python? A quick search turns up talk about Guido's PEP proposal, but I didn't see anything official. Would it be better/helpful to just rename the variables (response_data to response_list and city to city_list)?

# ? Dec 10, 2014 04:58

TURTLE SLUT: Dec 12, 2005

Has anyone in the entire loving world ever gotten setuptools develop mode working? Mine never seems to put the project on the Python path correctly. I thought it was just the weird directory structure in my project, but I then made a dummy project that looks just like this:

code:

/pythingy
    setup.py
    /src
        __init__.py
        somemod.py

I've tried various setup.py configurations, right now I'm doing this:

code:

setup(
    name='pythingy',
    version='0.1',
    packages=[
        'pythingy'
    ],
    package_dir={'pythingy': 'src'},
)

When I install this with pip install -e ., I get an .egg-link in /Library/Python/2.7/site-packages, which is in my Python path as I have several other libraries installed there that work fine. The egg-link points to the pythingy folder. The pythingy folder, after installing, has an egg-info folder that looks entirely normal. For example SOURCES.txt looks like this:

code:

pythingy.egg-info/PKG-INFO
pythingy.egg-info/SOURCES.txt
pythingy.egg-info/dependency_links.txt
pythingy.egg-info/top_level.txt
src/__init__.py
src/somemod.py

Which is exactly similar to other egg-infos I can see in the site-packages folder. However every time I try import pythingy, it's not found. Installing without -e works fine but of course the files it points to are in site-packages which is not exactly where I want to put my repos.

I'm on OSX, I dunno if that could conceivably have anything to do with it. It looks like it's simply just not reading egg-info files. Considering giving up and writing a bunch of boilerplate scripts that allow me to switch from "dev mode" and "library mode".

# ? Dec 10, 2014 12:01

StormyDragon: Jun 3, 2011

Symbolic Butt posted:

Oops I actually thought about it but then forgot to type it out that you can do something like this:
Python code:
a_book = Book(**dict(zip(expected_order, imported_book)))

Python code:

a_book = Book._make(imported_book)

Self-explanatory.

# ? Dec 10, 2014 13:20

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

StormyDragon posted:

Python code:
a_book = Book._make(imported_book)
Self-explanatory.

_make is order dependent though, which would bring me back to needing a function to reorder each imported_book list which seems sloppier than just passing in the order and zipping it up on creation.

As it sits, I can just do something like:

code:

Book = namedtuple('Book', ('title', 'author', 'date'))

for imported_book in this_file_full_of_books:
    a_book = Book(**dict(zip(expected_order, imported_book)

Where expected order is something like ['author', 'date', 'title'], and the imported_book might look something like ['rick james' '1967', 'slap'].

It seems like unpacking the dict shuffles the values into the Book by key automatically, so I don't have to make a function to reorder each possible field order.

That said, I am far from an expert, so feel free to tell me how wrong I am - because that's what I'm here for.

# ? Dec 10, 2014 15:13

Cingulate: Oct 23, 2012; by Fluffdaddy

This is basically the only thing I'm saying in this thread, but TMH, have you considered using Pandas?

# ? Dec 10, 2014 15:29

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

Cingulate posted:

This is basically the only thing I'm saying in this thread, but TMH, have you considered using Pandas?

It's for an exercise, stdlib only~

# ? Dec 10, 2014 15:37

StormyDragon: Jun 3, 2011

The March Hare posted:

_make is order dependent though, which would bring me back to needing a function to reorder each imported_book list which seems sloppier than just passing in the order and zipping it up on creation.

How about;

Python code:

Book = namedtuple('Book', expected_order)

for imported_book in this_file_full_of_books:
    a_book = Book._make(imported_book)

Of course some could guffaw at the audacity of building a namedtuple from data you picked out of a file, but as long as the field names don't change you will be safe. The tuples are then not ordered and shouldn't be used with that expectation.

# ? Dec 10, 2014 16:41

Dominoes: Sep 20, 2007

Tigren posted:

Thanks for the comments Dominoes.

Are type annotations a thing in python? A quick search turns up talk about Guido's PEP proposal, but I didn't see anything official. Would it be better/helpful to just rename the variables (response_data to response_list and city to city_list)?

Sort of - IDEs like PyCharm recognize them, and use them to check your code, but they're not enforced by the interpreter. They're also good for people reading your code. Guido's proposal you mentioned would expand on them for Python 3.5. Ie standardize syntax, add more flexibility. It would be nice to specify that an argument or output is a list of a certain type of objects, but you currently can't do that. Or specify something like a typeclass.

Descriptive variable names are important, but I don't think you should include the type in the variable name. You could use comments instead.

Dominoes fucked around with this message at 23:16 on Dec 10, 2014

# ? Dec 10, 2014 23:12

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Also Guido wants to bring something like mypy into core python. In fact, you can use mypy with type annotations right now!

Python code:

def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a+b

http://www.mypy-lang.org/tutorial.html

# ? Dec 11, 2014 00:01

Dominoes: Sep 20, 2007

Thermopyle posted:

Also Guido wants to bring something like mypy into core python. In fact, you can use mypy with type annotations right now!
Python code:
def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a+b
http://www.mypy-lang.org/tutorial.html

Nice - looks like mypy addresses both issues I mentioned. Just tried it - Pycharm recognizes the collection type syntax too.

Dominoes fucked around with this message at 00:31 on Dec 11, 2014

# ? Dec 11, 2014 00:28

namaste friends: Sep 18, 2004; by Smythe

I've started using Q to debug code. https://github.com/zestyping/q

Anyone ever used this on Windows? I've tried setting an environment variable for TEMPDIR in windows but it doesn't work.

I loving love this tool btw.

e: figured it out. Need to set permissions to your TEMPDIR to something more permissive.

namaste friends fucked around with this message at 20:06 on Dec 11, 2014

# ? Dec 11, 2014 19:47

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Cultural Imperial posted:

I've started using Q to debug code. https://github.com/zestyping/q

Anyone ever used this on Windows? I've tried setting an environment variable for TEMPDIR in windows but it doesn't work.

I loving love this tool btw.

e: figured it out. Need to set permissions to your TEMPDIR to something more permissive.

If you think that is good you should try out PyCharm.

# ? Dec 11, 2014 23:18

namaste friends: Sep 18, 2004; by Smythe

Thermopyle posted:

If you think that is good you should try out PyCharm.

I am using pycharm!

Are you talking about using the debugger?

# ? Dec 12, 2014 00:28

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Cultural Imperial posted:

I am using pycharm!

Are you talking about using the debugger?

Yeah. PyCharm has a fantastic debugger.

# ? Dec 12, 2014 02:13

SelfOM: Jun 15, 2010

Thermopyle posted:

Also Guido wants to bring something like mypy into core python. In fact, you can use mypy with type annotations right now!
Python code:
def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a+b
http://www.mypy-lang.org/tutorial.html

I really like this, but why not use syntax closer to Cython? I guess this is more pythonic but it would be great if it could all converge such that optionally statically typed data in python could then be easily compiled with cython. Cython is great because it already has integration with numpy and all the C libraries.

# ? Dec 12, 2014 03:43

Dominoes: Sep 20, 2007

SelfOM posted:

I really like this, but why not use syntax closer to Cython? I guess this is more pythonic but it would be great if it could all converge such that optionally statically typed data in python could then be easily compiled with cython. Cython is great because it already has integration with numpy and all the C libraries.

My guess: this way, it's valid Python.

# ? Dec 12, 2014 10:58

ShadowHawk: Jun 25, 2000; CERTIFIED PRE OWNED TESLA OWNER

SelfOM posted:

I really like this, but why not use syntax closer to Cython? I guess this is more pythonic but it would be great if it could all converge such that optionally statically typed data in python could then be easily compiled with cython. Cython is great because it already has integration with numpy and all the C libraries.

If you're defining new syntax or language changes you should just make the best choice from a linguistic ("pythonic"/"beautiful") perspective. Cython will follow.

# ? Dec 12, 2014 17:30

Mortanis: Dec 28, 2005; It's your father's lightsaber. This is the weapon of a Jedi Knight.; College Slice

I'm still working on my Python text adventure game, and I'm stumbling around deleting objects. It looks like Python doesn't release objects while there's still references to them in memory, so if I've got an item with a use() function that deletes itself (say a single use health potion), but it still exists in the player's inventory, it's still going to sit there. Is there no way to force a deletion of an object that removes it from all containers somehow (I really doubt this... I haven't seen it in other languages and it seems like a dangerous precedent)?

Otherwise I'm trying to think on how I'd remove items on use short of returning a boolean back to whatever called it - but then something else is responsible for the actual deletion and removal from the list, rather than the object itself, and that doesn't seem very clean. You might be tempted to say that the player object can be responsible for that since inventory always resides in that, but I've got the game set up so that the player can interact with the room's inventory as well: "Use oil on gate" (which would remove the oil from the player's inventory when using it on the room's gate) or "use fire on torch" (which would remove the fire from the room inventory while torch is in player's inventory).

I'd really not rather force everything to be in the player's inventory just because it'd be more convenient.

# ? Dec 15, 2014 19:15

Cingulate: Oct 23, 2012; by Fluffdaddy

Thermopyle posted:

Also Guido wants to bring something like mypy into core python. In fact, you can use mypy with type annotations right now!
Python code:
def fib(n: int) -> Iterator[int]:
    a, b = 0, 1
    while a < n:
        yield a
        a, b = b, a+b
http://www.mypy-lang.org/tutorial.html

Not that Python is too slow for me, but is it realistic that building on this, performance benefits will be achieved?

I understand that that's not the focus and the expected main benefit though.

# ? Dec 15, 2014 19:48

SurgicalOntologist: Jun 17, 2004

IIRC from reading the announcement or a talk about it or something from a while back, the only implication for performance is that something like PyPy could use it to aid its optimizations. Perhaps numba, though I',m not too familiar with it, or Cython. In the latter case it wouldn't be an increase in performance as much as the same old performance boost but using standard language features rather than however it works now.

# ? Dec 15, 2014 19:59

accipter: Sep 12, 2003

Mortanis posted:

I'm still working on my Python text adventure game, and I'm stumbling around deleting objects. It looks like Python doesn't release objects while there's still references to them in memory, so if I've got an item with a use() function that deletes itself (say a single use health potion), but it still exists in the player's inventory, it's still going to sit there. Is there no way to force a deletion of an object that removes it from all containers somehow (I really doubt this... I haven't seen it in other languages and it seems like a dangerous precedent)?

Otherwise I'm trying to think on how I'd remove items on use short of returning a boolean back to whatever called it - but then something else is responsible for the actual deletion and removal from the list, rather than the object itself, and that doesn't seem very clean. You might be tempted to say that the player object can be responsible for that since inventory always resides in that, but I've got the game set up so that the player can interact with the room's inventory as well: "Use oil on gate" (which would remove the oil from the player's inventory when using it on the room's gate) or "use fire on torch" (which would remove the fire from the room inventory while torch is in player's inventory).

I'd really not rather force everything to be in the player's inventory just because it'd be more convenient.

Why not remove the item from the players inventory on use?

Python code:

class Object:
    class use(self, owner=None):
        if owner and self in owner.inventory:
            del owner.inventory[self]
        
	# Code here

Edit: You might want to put some test in there if the object is consumable.

# ? Dec 15, 2014 20:18

supercrooky: Sep 12, 2006

Mortanis posted:

I'm still working on my Python text adventure game, and I'm stumbling around deleting objects. It looks like Python doesn't release objects while there's still references to them in memory, so if I've got an item with a use() function that deletes itself (say a single use health potion), but it still exists in the player's inventory, it's still going to sit there. Is there no way to force a deletion of an object that removes it from all containers somehow (I really doubt this... I haven't seen it in other languages and it seems like a dangerous precedent)?

Otherwise I'm trying to think on how I'd remove items on use short of returning a boolean back to whatever called it - but then something else is responsible for the actual deletion and removal from the list, rather than the object itself, and that doesn't seem very clean. You might be tempted to say that the player object can be responsible for that since inventory always resides in that, but I've got the game set up so that the player can interact with the room's inventory as well: "Use oil on gate" (which would remove the oil from the player's inventory when using it on the room's gate) or "use fire on torch" (which would remove the fire from the room inventory while torch is in player's inventory).

I'd really not rather force everything to be in the player's inventory just because it'd be more convenient.

Add a reference to an inventory instance as an attribute of your item class, set it when adding the item to the inventory, and have the item call a delete method on the inventory within its own use method. Something like this:

code:

class Inventory:
    def __init__(self):
        self.items = []

    def add(self, new_item):
        new_item.set_inventory_reference(self)
        self.items.append(new_item)

    def delete(self, item):
        self.items = [x for x in self.items if x is not item]


class Item:
    def __init__(self):
        self.inventory = None

    def set_inventory_reference(self, inventory):
        self.inventory = inventory

    def use(self):
        self.inventory.delete(self)


inv = Inventory()
item = Item()
inv.add(item)

In[5]: inv.items
Out[5]: [<__main__.Item instance at 0x0000000007BDAE48>]
In[6]: item.use()
In[7]: inv.items
Out[7]: []

# ? Dec 15, 2014 20:23

Mortanis: Dec 28, 2005; It's your father's lightsaber. This is the weapon of a Jedi Knight.; College Slice

supercrooky posted:

code:

class Inventory:
    def __init__(self):
        self.items = []

    def add(self, new_item):
        new_item.set_inventory_reference(self)
        self.items.append(new_item)

    def delete(self, item):
        self.items = [x for x in self.items if x is not item]


class Item:
    def __init__(self):
        self.inventory = None

    def set_inventory_reference(self, inventory):
        self.inventory = inventory

    def use(self):
        self.inventory.delete(self)


inv = Inventory()
item = Item()
inv.add(item)

In[5]: inv.items
Out[5]: [<__main__.Item instance at 0x0000000007BDAE48>]
In[6]: item.use()
In[7]: inv.items
Out[7]: []

That's perfect, thank you. Easily adaptable for creatures in a room as well so that they can drop their items and destroy themselves on death to boot.

# ? Dec 15, 2014 22:51

namaste friends: Sep 18, 2004; by Smythe

This is more of a design question. I have a bunch of log files that are generated by a custom application. They need to be forwarded to a Flume of Fluentd server. Aside from telling the original dev to write their log files through syslogd, is there a better way of forwarding this information other than reading it via python script and rewriting it to Flume/fluent?

# ? Dec 15, 2014 22:55

BigRedDot: Mar 6, 2008

supercrooky posted:

Add a reference to an inventory instance as an attribute of your item class, set it when adding the item to the inventory, and have the item call a delete method on the inventory within its own use method. Something like this:

Alterntatively, the python standard lib has weak references built in: https://docs.python.org/2/library/weakref.html

# ? Dec 16, 2014 02:30

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Cingulate posted:

Not that Python is too slow for me, but is it realistic that building on this, performance benefits will be achieved?

I understand that that's not the focus and the expected main benefit though.

I don't think there's anything theoretically wrong with that idea. I haven't heard anyone talking about making that a goal going forward, but it seems like something that would be taken account of in the future.

# ? Dec 16, 2014 02:33

boner confessor: Apr 25, 2013; by R. Guyovich

i'm trying to learn python and i'm still very babbyish. i did something like this

code:

## vehicle class
class Vehicle(object):
	def name(self):
		print "this is a vehicle!" #test
		pass

## car subclass
class Car(Vehicle):
	def __init__(self, name):
		self.name = name
		pass
		
## model sub subclass
class Model(Car):
	def __init__(self, passenger_cap):
		self.passenger_cap = passenger_cap
		passenger_cap = int(4)
		pass

what i'm trying to do is create different objects and write them to an array. i differentiate between Car and Truck or whatever so I can randomly generate different sets of names. that works fine

when i print out the records in my array, everything in the model subclass returns the output "<bound method model.name of <__main__.model at blah blah memory address" i get that it's returning the metadata about the attribute i want instead of the value of that attribute because it's not inheriting correctly, but i'm having a hard time figuring out exactly how i'm making this mistake

also please feel free to point out any other mistakes and bad conventions i'm doing here, i just started learning how to code

# ? Dec 16, 2014 06:36

QuarkJets: Sep 8, 2008