Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

Murodese: Mar 6, 2007; Think you've got what it takes?
We're looking for fine Men & Women to help Protect the Australian Way of Life.

Become part of the Legend. Defence Jobs.

code:

    72     38.6 MiB      0.0 MiB                       num = config.scan.get('message_scan_limit') * mult
    73     78.9 MiB     40.3 MiB                       for i in range(start, target, num):

Anyone able to explain the massive jump in memory usage? Python 3, so range should be a generator. Start/target are quite large numbers, so I can imagine building a list out of that would be worth about 40mb of memory, but... :v:

# ? Apr 2, 2015 14:06

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 16:26

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

duck monster posted:

Its *still* cool to render on the server. The whole serve-a-blob-of-js style of web development is hopelessly misguided in my opinion. Sure google parses javascript these days, but its not that bright about it, and its still going to favor HTML5 over "poo poo all over the browser with javascript".

And yeah AngularJS is insufferable. Check out Ractive, hook it up with JQuery, and it'll do 90% of what you'll want out of Angular, with a half-day learning curve instead of a few weeks of tearing your hair out fighting with Angulars bizzare and opaque ways.

If I never have to develop an "enterprise" Angular app again, I'll be a very happy man.

edit:

Except the type of apps that should be a blob of js are the the ones that don't make any sense to be indexable by Google.

I mean, what point would it be for Asana's web app to be indexed by Google? It just doesn't make any sense.

Anyway, don't use AngularJS if you have a choice, use React. If you have a site that should be indexable by Google, then just do isomorphic JS and render it on the server for Google's bots.

# ? Apr 2, 2015 17:46

QuarkJets: Sep 8, 2008

Murodese posted:

code:
    72     38.6 MiB      0.0 MiB                       num = config.scan.get('message_scan_limit') * mult
    73     78.9 MiB     40.3 MiB                       for i in range(start, target, num):
Anyone able to explain the massive jump in memory usage? Python 3, so range should be a generator. Start/target are quite large numbers, so I can imagine building a list out of that would be worth about 40mb of memory, but...

It could be due to your config.scan.get line

# ? Apr 2, 2015 18:54

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

A Jinja2 templating problem:

I'm trying to print out a list of urls. (This is for my personal website which is built in Pelican.) repos is that list of urls and I render them as a link with the url address as the text:

code:

<ul>
{% for a in repos %}
   <li><A HREF="{{ a }}">{{ a }}</A></li>
{% endfor %}
</ul>

Fine. Now, if it's a Github url or a PyPi url, I'd like the text to be 'Github' or "PyPi'. And here I start having problems. I wrote a filter to search for the relevant substrings and return True or False:

code:

<ul>
{% for a in repos %}
   {% if a | match 'github' %}
      {% set repo_name = 'Github' %}
   {% else %}
      {% set repo_name = a %}
   {% endif %}
   <li><A HREF="{{ a }}">{{ reponame }}</A></li>
{% endfor %}
</ul>

And I get the error:

code:

jinja2.exceptions.TemplateSyntaxError: expected token 'end of statement block', got 'string

The filter appears to work. So, what's the problem? How I fix this? Is there a better way of doing this?

# ? Apr 2, 2015 20:12

Space Kablooey: May 6, 2009

I can't tell whats wrong, but it's this line:

Python code:

{% if a | match 'github' %}

As for doing differently, I would pass a list of tuples, (url, text), like this:

code:

<ul>
{% for url, text in repos %}
   <li><A HREF="{{ url }}">{{ text }}</A></li>
{% endfor %}
</ul>

The problem with this is when you have a lot of these, but that can be easily generated.

# ? Apr 2, 2015 20:25

Fergus Mac Roich: Nov 5, 2008; Soiled Meat

What's the most idiomatic way to copy a list?
1. newlist = oldlist.copy()
2. newlist = list(oldlist)
3. newlist = oldlist[:](don't think I'd consider this, but you never know)

This is Python 3. Mostly an academic question, it came up in a project for my own amusement.

Fergus Mac Roich fucked around with this message at 22:58 on Apr 2, 2015

# ? Apr 2, 2015 22:47

nonathlon: Jul 9, 2004; And yet, somehow, now it's my fault ...

Fergus Mac Roich posted:

What's the most idiomatic way to copy a list?
1. newlist = oldlist.copy()
2. newlist = list(oldlist)
3. newlist = oldlist[:](don't think I'd consider this, but you never know)

This is Python 3. Mostly an academic question, it came up in a project for my own amusement.

I think I'd use copy to make intent clear, but I don't know there's much in it.

# ? Apr 2, 2015 23:19

Nippashish: Nov 2, 2005; Let me see you dance!

The most idiomatic way would be to not need to copy the list, but barring that I'd go with option 1.

# ? Apr 2, 2015 23:43

hooah: Feb 6, 2006; WTF?

I'm trying to send information from a Matlab script to Python. For the Python side, I followed this guide for TCP/IP communication (modified for Python 3), and for Matlab I used this and ended up with this code:

code:

t = tcpip('localhost', 10000);
fopen(t);
i = 0;
while true
    if mod(i, 100) == 0
        fprintf(t, 'i is now %d', i);
    end
    i = i + 1;
end

This works pretty well, except the Python script prints b'i is now 0'. Does anyone know what that leading b is about? I just now tried fprintf(t, i); in Matlab, but that printed b'\n' rather than b'0' (or whatever i happens to be at the time). Also, I'd like to not have the single quotes, but that wouldn't be too hard to scrub out.

# ? Apr 2, 2015 23:59

EAT THE EGGS RICOLA: May 29, 2008

hooah posted:

This works pretty well, except the Python script prints b'i is now 0'. Does anyone know what that leading b is about? I just now tried fprintf(t, i); in Matlab, but that printed b'\n' rather than b'0' (or whatever i happens to be at the time). Also, I'd like to not have the single quotes, but that wouldn't be too hard to scrub out.

From Python 3.3's docs:

quote:

Bytes literals are always prefixed with 'b' or 'B'; they produce an instance of the bytes type instead of the str type. They may only contain ASCII characters; bytes with a numeric value of 128 or greater must be expressed with escapes.

The 'b' prefix is ignored in Python 2 (it just indicates that it should become a bytes literal when migrated to Python 3)

# ? Apr 3, 2015 00:17

hooah: Feb 6, 2006; WTF?

So is there a better way I should be doing this? Ultimately I'll want to periodically send a triplet of numbers between -180 and 180 to Python.

# ? Apr 3, 2015 01:11

Hed: Mar 31, 2004; Fun Shoe

There's no problem with having a bytestring, you just need to run the .decode() method on it, and then you can parse it into python native datatypes the same way you would parse a string.

# ? Apr 3, 2015 02:37

IAmKale: Jun 7, 2007; やらないか; Fun Shoe

Edit: taking this to the Django thread

IAmKale fucked around with this message at 06:46 on Apr 3, 2015

# ? Apr 3, 2015 05:51

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

You should probably ask on the Django thread.

# ? Apr 3, 2015 06:01

Hughmoris: Apr 21, 2007; Let's go to the abyss!

I'm trying to get a basic grasp on using APIs but its kicking my butt.

I've signed up for a rottentomatoes API and am using the rottentomatoes python module.

I want to get a list of new dvd releases so I use the command rt.lists('dvds, 'new_releases') and I got a long list that looks like:

{u'movies': [{u'ratings': {u'critics_score': 72, u'audience_score': 86, u'critics_rating': u'Certified Fresh', u'audience_rating': u'Upright'}, u'links': {u'reviews': u'http://api.rottentomatoes.com/api/public/v1.0/movies/771351912/reviews.json', u'cast': u'http://api.rottentomatoes.com/api/public/v1.0/movies/771351912/cast.json', u'self': u'http://api.rottentomatoes.com/api/public/v1.0/movies/771351912.json', u'alternate': u'http://www.rottentomatoes.com/m/interstellar_2014/', u'similar': u'http://api.rottentomatoes.com/api/public/v1.0/movies/771351912/similar.json'}, u'title': u'Interstellar', u'critics_consensus': u'', u'release_dates': {u'dvd': u'2015-03-31', u'theater': u'2014-11-07'}, u'abridged_cast': [{u'name': u'Matthew McConaughey', u'characters': [u'Cooper'], u'id': u'162652350'}, {u'name': u'Anne Hathaway', u'characters': [u'Brand'], u'id': u'162656190'}, {u'name': u'Jessica Chastain', u'characters': [u'Murph'], u'id': u'770760183'}, {u'name': u'Michael Caine', u'characters': [u'Prof. Brand'], u'id': u'162652646'}, {u'name': u'Casey Affleck', u'characters': [u'Tom'], u'id': u'162706324'}], u'synopsis': u'With our time on Earth coming to an end, a team of explorers undertakes the most important mission in human history; traveling beyond this galaxy to discover whether mankind has a future among the stars. (C) Paramount', u'mpaa_rating': u'PG-13', u'year': 2014, u'alternate_ids': {u'imdb': u'0816692'}...
How in the world would I get the title out of that? I tried using the json module but had no luck.

*Well, I think I got it figured out. Turns out its returning a dictionary.

Python code:

movies = rt.lists('dvds', 'new_releases')
for each_movie in movies['movies']:
	print each_movie['title']

That will give me the title of the movie.

Hughmoris fucked around with this message at 06:44 on Apr 3, 2015

# ? Apr 3, 2015 06:30

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Hughmoris posted:

*Well, I think I got it figured out. Turns out its returning a dictionary.
Python code:
movies = rt.lists('dvds', 'new_releases')
for each_movie in movies['movies']:
	print each_movie['title']
That will give me the title of the movie.

I know this is only quick-and-dirty demonstration code, but don't use a variable name like "each_movie" in that loop - it's confusing to the reader! At any given time the variable contains information related to one specific movie, so call it "movie" or something else that doesn't potentially suggest it contains information on all the movies.

Python code:

movies = rt.lists('dvds', 'new_releases')['movies']
for movie in movies:
	print(movie['title'])

# ? Apr 3, 2015 12:25

hooah: Feb 6, 2006; WTF?

Hed posted:

There's no problem with having a bytestring, you just need to run the .decode() method on it, and then you can parse it into python native datatypes the same way you would parse a string.

Oh! I didn't know about that method (because I know next to nothing about Python, but have to work with it anyway). Thanks!

# ? Apr 3, 2015 12:49

Cingulate: Oct 23, 2012; by Fluffdaddy

Hughmoris posted:

Python code:

movies = rt.lists('dvds', 'new_releases')
for each_movie in movies['movies']:
	print each_movie['title']

If I had written that, I'd have done

Python code:

print([movie['title'] for movie in rt.lists('dvds', 'new_releases')['movies']])

Assuming performance not being critical and no side effects, is something like this considered disfavoured due to readability?

# ? Apr 3, 2015 12:55

Edison was a dick: Apr 3, 2010; direct current only

Cingulate posted:

If I had written that, I'd have done
Python code:
print([movie['title'] for movie in rt.lists('dvds', 'new_releases')['movies']])
Assuming performance not being critical and no side effects, is something like this considered disfavoured due to readability?

Also because it produces different output. One title per line is more readable than python's dump of a list's contents.
For closer to being equivalent you'd use

Python code:

for title in (movie['title'] for movie in rt.lists('dvds', 'new_releases')['movies']):
    print title

But to get back on topic, it varies from project to project, but I find the following easiest to follow at a glance

Python code:

movies = rt.lists('dvds', 'new_releases')['movies']
for movie in movies:
    print movie['title']

# ? Apr 3, 2015 13:02

Dominoes: Sep 20, 2007

Different, probably less-readable approach.

Python code:

movies = rt.lists('dvds', 'new_releases')
list(map(print, (movie['title'] for movie in movies)))

Or swap the second line with

Python code:

list(map(lambda movie: print(movie['title']), movies)

Dominoes fucked around with this message at 13:31 on Apr 3, 2015

# ? Apr 3, 2015 13:24

Cingulate: Oct 23, 2012; by Fluffdaddy

Dominoes posted:

Different, probably less-readable approach.
Python code:
movies = rt.lists('dvds', 'new_releases')
list(map(print, (movie['title'] for movie in movies)))
Or swap the second line with
Python code:
list(map(lambda movie: print(movie['title']), movies)

I've started using map a bit, but I still find list comprehension both more readable and easier to write.
Once you introduce lambda, I'm usually out though. Though maybe that's practice, I've used it before.

If I wanted exactly the same output, I'd probably also have done that as a list comp, like [print(movie['title']) for ... ] or something like that.

# ? Apr 3, 2015 13:35

Edison was a dick: Apr 3, 2010; direct current only

Cingulate posted:

I've started using map a bit, but I still find list comprehension both more readable and easier to write.
Once you introduce lambda, I'm usually out though. Though maybe that's practice, I've used it before.

That's because map with a lambda is inferior when compared to list comprehensions or generator expressions.
The only time I can think of where a call to map may be theoretically better than a list comprehension, would be if the body of the lambda were sufficiently complicated that there's a form that is written in C which you can pass in place of the lambda, and the operation itself is sufficiently slow that it's worth the overhead in marshalling the data between C and python.

# ? Apr 3, 2015 14:16

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Dominoes posted:

Different, probably less-readable approach.
Python code:
movies = rt.lists('dvds', 'new_releases')
list(map(print, (movie['title'] for movie in movies)))
Or swap the second line with
Python code:
list(map(lambda movie: print(movie['title']), movies)

While we're bikeshedding (yes I know I started it), this list(map(print, ...)) stuff is bananas IMO. Argument-unpacking is your friend!

Python code:

movie_titles = (m['title'] for m in rt.lists('dvds', 'new_releases')['movies'])
print(*movie_titles, sep = '\n')

# ? Apr 3, 2015 14:18

Cingulate: Oct 23, 2012; by Fluffdaddy

Hammerite posted:

While we're bikeshedding (yes I know I started it), this list(map(print, ...)) stuff is bananas IMO. Argument-unpacking is your friend!
Python code:
movie_titles = (m['title'] for m in rt.lists('dvds', 'new_releases')['movies'])
print(*movie_titles, sep = '\n')

* and ** is right up there with lambda and map though :colbert:

Also you could do a list comprehension instead and avoid the sep = thing, [print(movie) for movie in movie_titles].

Edison was a dick posted:

That's because map with a lambda is inferior when compared to list comprehensions or generator expressions.
The only time I can think of where a call to map may be theoretically better than a list comprehension, would be if the body of the lambda were sufficiently complicated that there's a form that is written in C which you can pass in place of the lambda, and the operation itself is sufficiently slow that it's worth the overhead in marshalling the data between C and python.

I use map for multiprocessing (as I learned in this thread).
I wish there was a parallel list comprehension, then I'd never ever optimise code ever again and instead spend half my time apologising for crashing the server by filling up all the memory.

# ? Apr 3, 2015 14:22

Edison was a dick: Apr 3, 2010; direct current only

Hammerite posted:

Python code:

movie_titles = (m['title'] for m in rt.lists('dvds', 'new_releases')['movies'])
print(*movie_titles, sep = '\n')

FFS! Let's just

Python code:

print '\n'.join(m['title'] for m in rt.lists('dvds', 'new_releases')['movies'])

and get on with our lives!

# ? Apr 3, 2015 14:25

Dominoes: Sep 20, 2007

Cingulate posted:

I've started using map a bit, but I still find list comprehension both more readable and easier to write.
Once you introduce lambda, I'm usually out though. Though maybe that's practice, I've used it before.

If I wanted exactly the same output, I'd probably also have done that as a list comp, like [print(movie['title']) for ... ] or something like that.

I prefer map if I don't need to use lambda or a a list comp with it. Ie the function already exists. In this example, I might prefer it if the input list was already set up; it it didn't need the ['title'] lookup.

# ? Apr 3, 2015 14:39

Cingulate: Oct 23, 2012; by Fluffdaddy

Edison was a dick posted:

FFS! Let's just
Python code:
print '\n'.join(m['title'] for m in rt.lists('dvds', 'new_releases')['movies'])
and get on with our lives!

Python code:

[print(m['title']) for m in rt.lists('dvds', 'new_releases')['movies'])

is still shorter and holds the smallest number of distinct elements out of all the ways to reproduce the original example :colbert:

Dominoes posted:

I prefer map if I don't need to use lambda or a a list comp with it. Ie the function already exists. In this example, I might prefer it if the input list was already set up; it it didn't need the ['title'] lookup.

Why? I also use list comprehensions with such functions, what's the advantages with the current stage of Python?
One I found googling just now was you can maybe more easily switch between parallel and serial implementations by doing an optional map = multiprocessing.pool.map ...

# ? Apr 3, 2015 15:12

Hammerite: Mar 9, 2007; And you don't remember what I said here, either, but it was pompous and stupid.; Jade Ear Joe

Cingulate posted:

Python code:
[print(m['title']) for m in rt.lists('dvds', 'new_releases')['movies'])
is still shorter and holds the smallest number of distinct elements out of all the ways to reproduce the original example

Generating a list as a side effect is ugly and makes it harder to appreciate at a glance what's happening. Shortness is secondary to clarity.

# ? Apr 3, 2015 15:16

OnceIWasAnOstrich: Jul 22, 2006

I open up the Python thread this morning and find that someone has hit the code-golfball into the bikeshed.

# ? Apr 3, 2015 15:19

Cingulate: Oct 23, 2012; by Fluffdaddy

Hammerite posted:

Generating a list as a side effect is ugly and makes it harder to appreciate at a glance what's happening. Shortness is secondary to clarity.

Okay, I get that first point.
Although I think that in this case, shortness is clarity as shortness comes from not introducing additional words (functions), like join or map.

# ? Apr 3, 2015 15:21

SurgicalOntologist: Jun 17, 2004

IMO, using comprehensions/generator expressions or map with functions with side effects is gross. If you want to print each thing, use a regular for loop.

SurgicalOntologist fucked around with this message at 15:37 on Apr 3, 2015

# ? Apr 3, 2015 15:32

Cingulate: Oct 23, 2012; by Fluffdaddy

Yeah I've done some reading and I'm getting the point. Great, now I'm gonna rewrite like half my code.

# ? Apr 3, 2015 16:19

Dominoes: Sep 20, 2007

Cingulate posted:

Why? I also use list comprehensions with such functions, what's the advantages with the current stage of Python?
One I found googling just now was you can maybe more easily switch between parallel and serial implementations by doing an optional map = multiprocessing.pool.map ...

Python code:

map(func, my_list)

Seems a bit cleaner than

code:

(func(x) for x in my_list)

, but either is fine. In cases where you have to define a function in map() with lambda, or where you're running a procedure instead of producing a result (Both of these are true for the movie example we've been discussing), map is messier than a listcomp or genexp.

Dominoes fucked around with this message at 16:31 on Apr 3, 2015

# ? Apr 3, 2015 16:28

Edison was a dick: Apr 3, 2010; direct current only

Dominoes posted:

Python code:
map(func, my_list)
Seems a bit cleaner than
code:
(func(x) for x in my_list)

They are semantically different. The former returns a list, the latter returns a generator, which when evaluated evaluates func for every entry in my_list.

Python code:

list(func(x) for x in my_list)

is equvalent to the map version.

This is important for the above case where you just want the side-effects.

# ? Apr 3, 2015 16:42

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

Edison was a dick posted:

They are semantically different. The former returns a list, the latter returns a generator, which when evaluated evaluates func for every entry in my_list.
Python code:
list(func(x) for x in my_list)
is equvalent to the map version.

This is important for the above case where you just want the side-effects.

No, they're semantically equivalent. One returns an unevaluated generator expression, the other returns an iterable map object.

# ? Apr 3, 2015 16:55

Cingulate: Oct 23, 2012; by Fluffdaddy

In case either of you feels this is all semantics: I'm definitely learning things.

# ? Apr 3, 2015 18:43

Dominoes: Sep 20, 2007

Hey, so in Python 3, they're both iterators. In Python 2, the second one's an iterator, and the first is a list.

# ? Apr 3, 2015 18:46

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Best because it's its got several short lines that do one explicit thing (note that just because its best doesn't mean I always do it, sometimes I'm just lazy):

Edison was a dick posted:

Python code:

movies = rt.lists('dvds', 'new_releases')['movies']
for movie in movies:
    print movie['title']

In other news, I spent the past few days getting to know the asyncio built-in module.

I think I like gevent a bit more (though that may be that I'm just used to gevent), but asyncio works just fine. It's definitely more explicit than gevent which does a lot of stuff implicitly...

Anyway, if you're not familiar with asyncio and similar-ish libraries like gevent, asyncio lets you do something like (using the aiohttp library here):

Python code:

@asyncio.coroutine
def print_page(url):
    response = yield from aiohttp.request('GET', url)
    body = yield from response.read_and_close(decode=True)
    print(body)

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait([print_page('http://example.com/foo'),
                                      print_page('http://example.com/bar')]))

The above code will download the provided urls in parallel without threads or multiprocessing.

# ? Apr 3, 2015 18:47

Dominoes: Sep 20, 2007

Thx for reminding me to learn that module. Any tutorials you recommend?

# ? Apr 3, 2015 18:49

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 16:26

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Dominoes posted:

Thx for reminding me to learn that module. Any tutorials you recommend?

There's nothing great, but here's the bookmarks I've got in my learning-asyncio folder:

http://www.buzzcapture.com/en/2014/05/python-asyncio-inside/
http://ntoll.org/article/asyncio
http://sahandsaba.com/understanding-asyncio-node-js-python-3-4.html

This presentation by David Beazley is the greatest of all, but it's not about asyncio! It's about how coroutines actually work.

# ? Apr 3, 2015 18:54

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »