Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

Tigren: Oct 3, 2003

I am not much of a coder, but I'm trying to hack together a python script that parses a JSON file and sends a SQL API call to online PostGIS database CartoDB.

Python code:

from cartodb import CartoDBAPIKey, CartoDBException
import spot2carto_keys
import urllib
import json

# CartoDB Authentication
cartodb = CartoDBAPIKey(spot2carto_keys.API_KEY, spot2carto_keys.carto_domain)

#SPOT Authentication
url = urllib.urlopen('https://api.findmespot.com/spot-main-web/consumer/rest-api/2.0/public/feed/' +
        spot2carto_keys.spotAPIkey + '/message.json')

#Convert SPOT into usable format
json_data = json.loads(url.read())
data = json_data['response']['feedMessageResponse']['messages']['message']

#Find last added point
try:
    latest = cartodb.sql("SELECT MAX(unixtime) FROM pct_spot", False, True, 'csv')
    max = int(latest[6:-2])
except CartoDBException as e:
    print ("Error: ", e)

#Insert newer points
for messages in data:
    if (messages['unixTime'] > max):
        point_data = {
                'the_geom': 'ST_SetSRID(ST_MakePoint(%s, %s),4326)' % (messages['longitude'], messages['latitude']),
                'id': messages['id'],
                'longitude': messages['longitude'],
                'unixtime': messages['unixTime'],
                'messagetype': messages['messageType'].encode('utf-8'),
                'messengerid': messages['messengerId'].encode('utf-8'),
                'batterystate': messages['batteryState'].encode('utf-8'),
                'latitude': messages['latitude'],
                'hidden': messages['hidden'],
                'modelid': messages['modelId'].encode('utf-8')
                }
        keys = ','.join(['%s' % k for k in point_data.iterkeys()])
        values = []
        for v in point_data.itervalues():
            values.append(v)
        query = 'INSERT INTO pct_spot (%s) VALUES %s' % (keys, tuple(values))
        print cartodb.sql(query)

This yields the following SQL query:

code:

INSERT INTO pct_spot (messagetype,batterystate,latitude,unixtime,hidden,modelid,the_geom,id,longitude,messengerid) 
VALUES ('UNLIMITED-TRACK', 'GOOD', 48.66172, 1412475115, 0, 'SPOT3', 'ST_SetSRID(ST_MakePoint(-120.73102, 48.66172),4326)', 329199789, -120.73102, '0-2414146')

I used the list/tuple combo because the 'values' are a mixture of strings and numbers, but 'the_geom' value (ST_SetSRID) needs to be passed as a SQL function, not a string. I'm not quite sure how to go about this. Can anyone offer any tips? Again, I'm not much of a programmer, more of a dabbler, but I'm open to all suggestions.

Edit:

Instead of building a list and then converting to a tuple, I tried building a string piece by piece for 'values' using the repr() of each value and the str() representation when I pass the SQL function. This seems to work, but is it the best way to do it?

Python code:

        s = '('
        for x in point_data.iterkeys():
                if (s == '('):
                    s = s + repr(point_data[x])
                elif (x != 'the_geom'):
                    s = s + ',' + repr(point_data[x])
                else:
                    s = s + ',' + point_data[x]
        s = s + ')'

Tigren fucked around with this message at 02:56 on Oct 6, 2014

# ¿ Oct 5, 2014 21:37

Adbot: ADBOT LOVES YOU

# ¿ Apr 27, 2024 11:51

Tigren: Oct 3, 2003

Is anyone willing to look through some code for me and give me a critique? This is the first time I've ever built something that's more than a few lines.

I feel like I need a new set of eyes on this to tell me where and what to clean up.

The json file in the repo is a sample output and should render nicely with Github geojson features.

I think my next step is to break out some of the output functions into their own module.

https://github.com/bwainstock/beeradvocate_scraper/blob/master/ba_scraper.py

# ¿ Dec 8, 2014 05:04

Tigren: Oct 3, 2003

Dominoes posted:

Seems pretty good.

Minor suggestions from a lazy skim:
-PEP 8 says 2 lines between functions that aren't part of a class. Also says put periods at the end of your docstring sentences.

Type annotations would clarify what your functions are doing, since it's not obvious for some of them. ie:
Python code:
def parse(response_data: list, city: str) -> list:
    ...
You could also clarify in the docstring for the more complex structures you're passing around, like response_data. Your docstrings are general; you have a few complex functions that could use a better description.

if 'fail' not in response.url: - You should add a comment about what would cause 'fail' to be in the url.

Thanks for the comments Dominoes.

Are type annotations a thing in python? A quick search turns up talk about Guido's PEP proposal, but I didn't see anything official. Would it be better/helpful to just rename the variables (response_data to response_list and city to city_list)?

# ¿ Dec 10, 2014 04:58

Tigren: Oct 3, 2003

I am trying to use a Python module pacparser which is basically a wrapper for a C library. When the C library encounters an error, it spits out information and then raises a Python exception.

Python code:

In [5]: _pacparser.find_proxy('http://www.google.com', 'www.google.com')
JSERROR: NULL:285:
    ReferenceError: FindProxyForURL is not defined
pacparser.c: pacparser_find_proxy: Problem in executing findProxyForURL.
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-5-0d32bdb423d6> in <module>()
----> 1 _pacparser.find_proxy('http://www.google.com', 'www.google.com')

error: Could not find proxy

I know how to capture the exception that Python raises, but how do I capture the information that is output from C?

code:

JSERROR: NULL:285:
    ReferenceError: FindProxyForURL is not defined
pacparser.c: pacparser_find_proxy: Problem in executing findProxyForURL.

Tigren fucked around with this message at 22:35 on Jan 20, 2016

# ¿ Jan 20, 2016 21:15

Tigren: Oct 3, 2003

Munkeymon posted:

Try

Python code:

import os
import sys

null = open(os.devnull, 'w')
temp = sys.stderr
sys.stderr = null

#let the extension do its thing here

sys.stderr = temp #restore it
null.close()

And where am I supposed to be seeing the output that I want to capture?

Before resetting sys.stderr, sys.stderr is just <open file '/dev/null', mode 'w' at 0x7fad40d5ec90>

# ¿ Jan 20, 2016 22:35

Tigren: Oct 3, 2003

Munkeymon posted:

Oh sorry I got it in my head that you just wanted to suppress it.

Python code:

import os
import sys
from io import TextIOWrapper, BytesIO

capture = TextIOWrapper(BytesIO(), sys.stderr.encoding)
temp = sys.stderr
sys.stderr = capture

#let the extension do its thing here

sys.stderr = temp #restore it
capture.seek(0) #readable as a normal file

Python code:

import os
import sys
from io import TextIOWrapper, BytesIO

import pacparser

capture = TextIOWrapper(BytesIO(), sys.stderr.encoding)
temp = sys.stderr
sys.stderr = capture

pacparser.init()
pacparser.parse_pac_file('test.pac')
pacparser.find_proxy('http://www.google.com')

sys.stderr = temp
capture.seek(0)
if capture:
    print "error captured"

No 'error captured' printing. 'capture' is empty.

# ¿ Jan 20, 2016 23:56

Tigren: Oct 3, 2003

Munkeymon posted:

You'd still have to do something about the exception:
Python code:
import sys
from io import TextIOWrapper, BytesIO
import pacparser

capture = TextIOWrapper(BytesIO(), sys.stderr.encoding)
temp = sys.stderr
sys.stderr = capture

try:
    pacparser.init()
    pacparser.parse_pac_file('test.pac')
    pacparser.find_proxy('http://www.google.com')
except Exception:
    print("that did not go well!")
finally:
    sys.stderr = temp

capture.seek(0)
print(capture.read())
Also capture is truthy as long as the call to TextIOWrapper didn't throw, so if capture: would always run the if body.

Thanks for being patient with me. stderr/stdout redirection seems like kind of a funky beast.

I tried running that from within a REPL and from a script

code:

$ python test.py
JSERROR: PAC script:3:
    ReferenceError: isnNet is not defined
pacparser.c: pacparser_find_proxy: Problem in executing findProxyForURL.
that did not go well!

I just get a blank line output for console.read(0).

Once I realized what you were doing, I looked into stderr redirection and saw mention of stderr caching. Do you think the JSERROR is cached and therefore not directly accessible from sys.stderr redirection?

# ¿ Jan 21, 2016 18:36

Tigren: Oct 3, 2003

Munkeymon posted:

Caching shouldn't be an issue because we're supplying our own buffer before the output is generated. Could be I made an invalid assumption and it might be sending the output on stdout. Could be that there's an init in the module that grabs copies of the std* streams before we're replacing them. This should mitigate both:
Python code:
import sys
from io import TextIOWrapper, BytesIO

capture = TextIOWrapper(BytesIO(), sys.stderr.encoding)
temp_err = sys.stderr
temp_out = sys.stdout
sys.stderr = sys.stdout = capture #mess with all the outputs

import pacparser #don't let it init until after we mess with outputs

try:
    pacparser.init()
    pacparser.parse_pac_file('test.pac')
    pacparser.find_proxy('http://www.google.com')
except Exception:
    print("that did not go well!")
finally:
    sys.stderr = temp_err
    sys.stdout = temp_out

capture.seek(0)
print("output: {0}|".format(capture.read()))
And, someone correct me if I'm wrong here, but I think the extension could be opening its own std* streams, which I don't think you can do anything about other than shove it in a subprocess and capture all the output.

code:

JSERROR: PAC script:3:
    ReferenceError: isnNet is not defined
pacparser.c: pacparser_find_proxy: Problem in executing findProxyForURL.
output: |

Seems like it's either time to subprocess it or write my own module.

# ¿ Jan 21, 2016 22:28

Tigren: Oct 3, 2003

QuarkJets posted:

I think that those languages do recognize that "-2" is its own integer rather than the result of the operation "0-2", but they also respect a specific order of operations when evaluating a line, and in doing so the negative sign attached to the 2 gets ignored until after the exponent gets evaluated. There are a few languages that make an exception for this case by giving unary minus a higher precedence, so that -2**2 results in +4 (I know that Excel is one of them but I don't use any of the others)

It seems like the statement is being evaluated as -1*2**2 which, following the order of operations I learned in whatever grade, would mean 2^2 then multiplied by -1.

# ¿ Apr 25, 2016 02:38

Tigren: Oct 3, 2003

Hed posted:

Yeah I don't know it's how every hand calculator I've ever had has worked so it didn't surprise me at all.

Hand... calculator? Like, you push buttons on a thing that only calculates?

# ¿ Apr 26, 2016 23:01

Tigren: Oct 3, 2003

Loving Africa Chaps posted:

Just doing tabs but for some reason the code tags make it look weird

Here's the error:

Traceback (most recent call last):
File "test.py", line 12, in <module>
cursor2.execute(query, name)
File "/home/#/foambot/venv/lib/python2.7/site-packages/mysql/connector/cursor.py", line 515, in execute
self._handle_result(self._connection.cmd_query(stmt))
File "/home/#/foambot/venv/lib/python2.7/site-packages/mysql/connector/connection.py", line 488, in cmd_query
result = self._handle_result(self._send_cmd(ServerCmd.QUERY, query))
File "/home/#/foambot/venv/lib/python2.7/site-packages/mysql/connector/connection.py", line 395, in _handle_result
raise errors.get_exception(packet)
mysql.connector.errors.ProgrammingError: 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '%s' at line 1

Edit i should clarify that i'm using mysql.connector which didn't want to install via pip until i specified the version number so maybe that's got something to do with it?

I think your params need to be a tuple or dictionary:

quote:

cursor.execute(operation, params=None, multi=False)
iterator = cursor.execute(operation, params=None, multi=True)

This method executes the given database operation (query or command). The parameters found in the tuple or dictionary params are bound to the variables in the operation.

# ¿ May 5, 2016 18:42

Tigren: Oct 3, 2003

the talent deficit posted:

i know python hates functional programming, but is there really no filtermap in the standard library?

filter() map() reduce()???

# ¿ May 17, 2016 02:04

Tigren: Oct 3, 2003

BigRedDot posted:

Hey we did a thing: https://bokeh.github.io/blog/2016/6/28/release-0-12/

Now, excuse me while go collapse in a heap.

Congrats BigRedDot, that's awesome. Bokeh is such a fun library.

# ¿ Jun 28, 2016 22:34

Tigren: Oct 3, 2003

I'd be willing to call myself an advanced beginner Python programmer, and I just completed a small project that someone actually paid me for!

Iterating through the project made me realize I need to start implementing better testing into my design. I've already completed the project and I want to go back and build it again using TDD.

For the life of me I can't figure out where to start. Does anyone have any good reads or tutorials on test-driven design? I've found a couple that show TDD with Django, but I'm not really looking to get into the whole Django workflow while learning TDD.

# ¿ Jul 15, 2016 16:24

Tigren: Oct 3, 2003

JVNO posted:

Dex: I realize that's the error with her version of the code- I pointed that out. I just don't know how to 'passively' record the stream while the code advances, then dumping that entire stream to a wave file.

Baka Kaba:The program becomes immediately non responsive, to the point where the 'End Process?' prompt will come up. I can't test. The crash errors are unspecified if it hangs like this

I'm not sure why the callback function would be necessary still; I want one single recording. If the wav recording is simply too large for a discontinuous recording, is there a way I could stick a couple lines of code at the bottom of each iteration of the experimental trial 'FOR' loop that says 'Write frames to wav file; clear buffer for next trial'? Each trial individually is less than 5 seconds- just the total time exceeds 30 minutes.

Specifically the part of the code here:

"#Present study trials
*TRIMMED FOR BREVITY- but displays multiple trials of words which participants must read aloud*"

Is the giant 'FOR ___ in ____' that displays the hundreds of trials.

I haven't used PyAudio, but it sounds like "immediately non responsive" is because the program is waiting for the recording to end. That's where the callback function comes in. In callback mode, PyAudio will call a specified callback function whenever there is new (recorded) audio data available. Note that PyAudio calls the callback function in a separate thread. The separate thread is what allows the audio to record while the rest of the script still plays through.

# ¿ Sep 13, 2016 21:07

Tigren: Oct 3, 2003

LochNessMonster posted:

I'm definately going to write the data to a database. What I'm struggling with is how to structure the data with my script before writing to disk.

The program currently scrapes html pages that are saved to disk. Each page has 10 vehicles in it, so I loop through the divs parsing the values for brand/type/milage/year/dealer.

That's where I am right now.

To get that data to a database I figured I should probably "store" that data inside the script for at least all vehicles on 1 page, or for all vehicles on all the pages.

When I have parsed 1 page (and thus 10 vehicles) I write all of them to the database at once, instead of opening (and closing) a db connection for each iteration of the page loop(s).

So I think I need to put the vehicle info in a list, tuple or dictionary before writing it to the database.

I could be missing something obvious though.

That's exactly what dicts and lists are for. So you could have a list called list_of_motorcycles and each motorcycle could be a dict, which links keys to values. Once you have parsed all the information for one motorcycle, append that dictionary to the end of your list_of_motorcycles and move to the next one. Then, at the end, you've got a big ol' list of motorcycle dictionaries.

But really, you'll already have the db connection open and I guarantee you that you won't be taxing sqlite by calling 100 INSERTs. So you can just add each entry instead of storing them in another object before adding them.

Python code:

list_of_motorcycles = list() # This establishes an empty list called list_of_motorcycles

# loop through the divs parsing the values for brand/type/milage/year/dealer. 
brand = 'Kawasaki'
type = 'Ninja'
mileage = 10000
year = '2016'
dealer = 'Crazy Ed'
motorcycle = {'brand': brand,
                          'type': type,
                          'mileage': mileage,
                          'year': year,
                          'dealer': dealer}
list_of_motorcycles.append(motorcycle)
# end of loop

Tigren fucked around with this message at 23:07 on Sep 15, 2016

# ¿ Sep 15, 2016 22:49

Tigren: Oct 3, 2003

LochNessMonster posted:

I'm probably going to figure this out within 5 minutes after posting, but I've been struggling with inserting data into sqlite 3 and I'm not quite sure why.

code:

import sqlite3
conn = sqlite3.connect('vehicles.db')
cursor = conn.cursor()

testDict = {'DealerId': '1',
            'Brand': 'bmw',
            'Milage': '13.909',
            'Dealer': 'crazy jack',
            'Price': '5.000',
            'Type': 'r90',
            'Year': '1980'}

for (DealerId, Dealer, Brand, Type, Price, Milage, Year) in testDict:
    cursor.execute("INSERT INTO motorcycles VALUES(?, ?, ?, ?, ?, ?, ?)",
                   DealerId,
                   Dealer,
                   Brand,
                   Type,
                   Price,
                   Milage,
                   Year)

Running this results in the following error message

code:

Traceback (most recent call last):
  File "/path/to/script/dbinsert.py", line 13, in <module>
    for (DealerId, Dealer, Brand, Type, Price, Milage, Year) in testDict:
ValueError: too many values to unpack (expected 7)

Please point and laugh at my failures :downs:

It looks like other people squared you away on your for loop. I wanted to show you this option for inserting from a dict as well. Instead of the question mark "parameterization", you can use named placeholders. Those names are the keys of your dict.

Python code:

In [38]: testDict                                                     
Out[38]:
{'Brand': 'bmw',
 'Dealer': 'crazy jack',
 'DealerId': '1',
 'Milage': '13.909',
 'Price': '5.000',
 'Type': 'r90',
 'Year': '1980'}

In [39]: cursor.execute('create table motorcycles (brand, dealer, dealerid, mileage, price, type, year)')
Out[39]: <sqlite3.Cursor at 0x7f1bf8216dc0>

In [40]: cursor.execute("INSERT INTO motorcycles VALUES (:Brand, :Dealer, :DealerId, :Milage, :Price, :Type, :Year)", testDict)
Out[40]: <sqlite3.Cursor at 0x7f1bf8216dc0>

In [41]: cursor.execute("SELECT * FROM motorcycles")
Out[41]: <sqlite3.Cursor at 0x7f1bf8216dc0>

In [42]: cursor.fetchall()
Out[42]: [('bmw', 'crazy jack', '1', '13.909', '5.000', 'r90', '1980')]

There's also the option of using cursor.executemany, which allows you to loop over a list of entries like so:

Python code:

In [82]: # starting with an empty db

In [83]: cursor.execute("SELECT * FROM motorcycles")
Out[83]: <sqlite3.Cursor at 0x7f1bebdd6d50>

In [84]: cursor.fetchall()
Out[84]: []

In [85]: # testList has two motorcycle dictionaries, but could have more
                                                                                                                                                                                                                                    
In [86]: testList
Out[86]:
[{'Brand': 'bmw',
  'Dealer': 'crazy jack',
  'DealerId': '1',
  'Milage': '13.909',
  'Price': '5.000',
  'Type': 'r90',
  'Year': '1980'},
 {'Brand': 'bmw',
  'Dealer': 'crazy jack',
  'DealerId': '2',
  'Milage': '13.909',
  'Price': '5.000',
  'Type': 'r90',
  'Year': '1980'}]

In [87]: cursor.executemany("INSERT INTO motorcycles VALUES (:Brand, :Dealer, :DealerId, :Milage, :Price, :Type, :Year)", testList)
Out[87]: <sqlite3.Cursor at 0x7f1bebdd6d50>

In [88]: cursor.execute("SELECT * FROM motorcycles")
Out[88]: <sqlite3.Cursor at 0x7f1bebdd6d50>

In [89]: cursor.fetchall()
Out[89]:
[('bmw', 'crazy jack', '1', '13.909', '5.000', 'r90', '1980'),
 ('bmw', 'crazy jack', '2', '13.909', '5.000', 'r90', '1980')]

# ¿ Sep 21, 2016 18:35

Tigren: Oct 3, 2003

LochNessMonster posted:

Both options seem like a pretty efficient way of doing things. I didn't get the for loop working yet so I'll give this a try as well.

Just for my understanding, the: in/out [number] is from your python environment?

Ya, those numbers are from iPython, which is an alternative python REPL. You can ignore those parts.

# ¿ Sep 21, 2016 20:51

Tigren: Oct 3, 2003

LochNessMonster posted:

Thanks for all the feedback. I managed to get it working with Tigrens way. The code is now looking like this:
Python code:
<scraping part>
motorcycle = { 'Brand': getBrand.get_text(),
                       'Type': getType.get_text(),
                       'Price': getPrice.get_text(),
                       'Milage': getMilage.get_text(),
                       'Dealer': dealerName,
                       'Year': getYear.get_text(),
                       'DealerId': getDealerId['id']}
        processingList.append(motorcycle)

dbPath = str(dataDir + dbName)

try:
    conn = sqlite3.connect(dbPath)
except Exeption as err:
    print('Could not connect to vehicles.db because reasons: ' + str(err))
cursor = conn.cursor()

try:
    cursor.executemany("INSERT INTO motorcycles VALUES (:DealerId, :Dealer, :Brand, :Type, :Price, :Milage, :Year)", processingList)
    conn.commit()

except Exception as err:
    print('An exception happened: ' +str(err))

conn.close()
It's really cool to see results while learning a useful new skill. I'm really enjoying this, and already have lots of other things in mind to incorporate in it.

For now it's still a lot of trial and error, and thinking about what I'm doing wrong but when I started a few weeks back I knew nothing at all (compared to you guys I guess that's still the case) but now I seem to find my way a bit.

Congrats, feels good doesn't it? Now read through these comics and check them off one by one when they apply to you. They will all apply to you eventually.

https://medium.freecodecamp.com/coding-explained-in-25-profound-comics-8847ea03819c#.lzka3j5rh

Tigren fucked around with this message at 17:55 on Sep 22, 2016

# ¿ Sep 22, 2016 17:48

Tigren: Oct 3, 2003

LochNessMonster posted:

When I started I didn't have a real idea what to do, I'm just trying to learn and think qbout new things to do when I've finished something.

I was reading about Django and Flask a bit earlier and I was leaning towards Django as it appears to be a large framework which might suit additional features I might think about later. One of the thinks I'd like to do is pricetracking and visualize that with a graphing framework or something.

Flask seems to be in really early development (version 0.10). Does that matter?

Django and Flask (with the right plugins) will accomplish the same tasks. When Django is advertised as "batteries included", it really means it. It includes database migrations, user authentication, admin panels, ORM, caching frameworks, etc. If you'll need all of that or even most of that, Django is an awesome project to work with and has fantastic documentation. We even have a Django thread here and people like Thermopyle are great helps in there.

If you just need an easy way to pull info from a sqlite database and throw it onto a webpage, look at Flask. It can do everything Django can, you'll just need to install each piece as an extension. Don't worry about the 0.11 version number. The project has been around for 5+ years and is very well maintained and mature. Pinterest and LinkedIn use Flask at scale, and I'm sure there are more examples.

Tigren fucked around with this message at 18:53 on Oct 16, 2016

# ¿ Oct 16, 2016 18:50

Tigren: Oct 3, 2003

LochNessMonster posted:

Django sounds awesome, but it might be overkill for a newbie like me.

Will I be able to add other functionality (like creating graphs) to a Flask web application later on?

The creating graphs functionality is entirely independent from your web framework. If you just want interactive visualizations, you can look into Bokeh, which is developed by BigRedDot. If you want to display those interactive visualizations along with other content, Bokeh will integrate with Flask very nicely.

# ¿ Oct 16, 2016 21:27

Tigren: Oct 3, 2003

Suspicious Dish posted:

I haven't found many of Django's batteries to be useful, though. The comment system is mostly a joke.

You'll have to expand on this. Django's batteries, if and when you need them, are definately useful. The admin interface, ORM and auth system themselves are worth the price of admission. Throw in migrations and django-admin and you really are getting a lot of features out of the box.

Thermopyle posted:

Flask is fine, but I want to beat the drum I beat every time this comes up (this isn't directed at you, I'm just throwing this out into the ether):

Django is batteries-included, but that doesn't make it inferior to Flask. It's kind of like python itself...just because Python has seemingly millions of libraries in its standard library doesn't mean that makes it hard or inferior in some way for writing a for loop.

You can use Django to write the web equivalent of Hello World in basically as minimal a fashion as Flask. A complete Django site:
Python code:
from django.conf import settings
from django.conf.urls import url
from django.http import HttpResponse

settings.configure(
    DEBUG=True,
    SECRET_KEY='A-random-secret-key!',
    ROOT_URLCONF=sys.modules[__name__],
)


def index(request):
    return HttpResponse('Hello World')

urlpatterns = [
    url(r'^$', index),
You don't get this idea when you first approach Django because all the tutorials want to expose you to everything Django can do.

The nice thing about starting out with Django is that when you want to expand your site's capabilities, those capabilities are "in the box". With Flask, you've got to find a third-party project (or write it yourself) that implements the functionality you want and cross your fingers that it's well maintained and has as many eyes on it watching it for security and quality as Django does.

The nice thing about starting out with Flask is that it's easier to get started with because the documentation and tutorials don't cover all the stuff that doesn't come with Flask.

A similar Flask app:

Python code:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def index():
    return 'Hello world!'

if __name__ == '__main__':
    app.run(debug=True)

I agree that Django tutorials are often over complicated and make it hard to see the forest for the trees. I find Flask's syntax and setup easier for my uses, but I don't build complex web apps. I use the batteries so rarely that I don't mind throwing a Flask extension into my app when I need it.

# ¿ Oct 16, 2016 23:14

Tigren: Oct 3, 2003

cinci zoo sniper posted:

Is there a laconic way to return every X out of every Y elements of the list? Say, with X=2 and Y=4, output of foo([1, 2, 3, 4, 5, 6], X, Y) would be [1, 2, 5, 6]. I can write a for loop doing but, but it's going to be way too slow.

If I understand your question, you're looking for the basic slicing feature of python lists.

Python code:

In [1]: foo = [1,2,3,4,5,6,7,8,9,10]

In [2]: foo
Out[2]: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

In [3]: foo[::2]
Out[3]: [1, 3, 5, 7, 9]

In [4]: foo[1::2]
Out[4]: [2, 4, 6, 8, 10]

In [5]: foo[::-1]
Out[5]: [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]

Tigren fucked around with this message at 15:10 on Oct 27, 2016

# ¿ Oct 27, 2016 15:07

Tigren: Oct 3, 2003

LochNessMonster posted:

I'm trying to generate a page for each dealer in my sqlite db, which kinda works, but when trying to display all vehicles for that dealer I'm running into some issues.

I think my 2nd sql query isn't picking up the dealer name. And if it would, I'm not sure if it'd actually return all vehicles for each page. I'm kinda stuck on how to figure this out.
Python code:
@app.route('/dealers/<dealerName>/')
def show_dealer(dealerName):
    dealerName = query_db('select distinct dealer from motorcycles')
    for dealer in dealerName:
        dealerVehicles = query_db("select * from motorcycles where dealer = %s") % (dealer)
    return render_template('layout.html', dealerName=dealerName, dealerVehicles=dealerVehicles)

Your string substitution is out of place.

dealerVehicles = query_db("select * from motorcycles where dealer = %s" % (dealer))

If you're using python3, take a look at .format() instead. https://pyformat.info/

Tigren fucked around with this message at 21:17 on Nov 2, 2016

# ¿ Nov 2, 2016 21:11

Tigren: Oct 3, 2003

LochNessMonster posted:

I'll read up on that, thanks. I have done several tutorials and didn't come across string substitution yet, so I'v been googling that part. I'm indeed using python3 so that's probably my first issue.

I noticed you mentioned my for loop overwriting the variable with each iteration but changed your reply so it's gone. That was a relevant remark as well though, I'm kinda ashamed I didn't notice that myself.

Sorry about that. When I first read your post, I thought this was for displaying all vehicles for all dealers. I figured you were iterating through a list of dealerNames, but only getting one dealer's worth of dealerVehicles. If that's the case, then yes, you'll want to append dealerVehicles to a list or dictionary at the end of the loop, like this:

Python code:

@app.route('/dealers/<dealerName>/')
def show_dealer(dealerName):
    dealerName = query_db('select distinct dealer from motorcycles')
    dealerVehicles = dict()
    for dealer in dealerName:
        vehicles = query_db("select * from motorcycles where dealer = %s" % (dealer))
        dealerVehicles[dealer] = vehicles
    return render_template('layout.html', dealerName=dealerName, dealerVehicles=dealerVehicles)

Then your layout.html might have:

HTML code:

{% for dealer, vehicles in dealerVehicles.items() %}
<div>
  <h1>{{ dealer }}</h1>
  <ul>
    {% for vehicle in vehicles %}
    <li>{{ vehicle }}</li>
    {% endfor %}
  </ul>
</div>
{% endfor %}

# ¿ Nov 2, 2016 22:02

Tigren: Oct 3, 2003

Cingulate posted:

str.format is also in py2, what do you guys mean?

>= py2.6 to be exact I guess

# ¿ Nov 2, 2016 23:02

Tigren: Oct 3, 2003

Kit Walker posted:

So I'm pretty new to coding in general but I started learning Python a couple weeks ago. I've gone through the Codecademy course and I'm nearly done with Learn Python the Hard Way. So far it's been pretty easy and understandable. I just don't really know where to go from here, either in terms of developing my skills with Python or as a programmer in general. Any advice?

Why are you learning to code? After Codecademy and LPtHW, you should have the skills to start taking a crack at building whatever it was you hoped to build. Start on that project.

# ¿ Nov 21, 2016 23:57

Tigren: Oct 3, 2003

Cockmaster posted:

What are some good resources for learning GUI programming in Python? I'll be writing some interface software for a robot - nothing too fancy, just standard off-the-shelf GUI objects. Plus I'd need the code to run on either a Windows 10 system or a Raspberry Pi, so I would assume I'd want to use something like Tkinter or wxPython.

Any hangups making a small web app and then you can design the GUI using HTML5/CSS/JS?

# ¿ Dec 5, 2016 18:01

Tigren: Oct 3, 2003

Paul MaudDib posted:

What's the best all-around choice for an ORM? I am doing a simple project right now so it's not super critical but I'm trying to pick up skills that I could apply to other projects and I might as well start out properly.

edit: is there anything with a feature like Hibernate's hbm2ddl.auto functionality where it can create a valid database from the mapping configs?

SqlAlchemy if not using Django. Django's ORM if you are.

# ¿ Dec 21, 2016 01:29

Tigren: Oct 3, 2003

Whybird posted:

Wow! Thank you all for the advice, this is really helpful!

I've just been watching https://www.youtube.com/watch?v=C4Kc8xzcA68 on hash tables and I realise I've been assuming that membership testing in a set is much, much slower than it is (also, I'm absolutely astounded by just how clever hash tables are!) -- this was why I was wary to have each entity check in turn whether or not it was in a player's visible_entities list, because it felt like that would be a lot of time-consuming checks to make. The idea of using the intersection of all visible entities instead is a really clever one, and I like it a lot!

I haven't watched that video, but 7 years is a long time in Python time. Check out this video by Raymond Hettinger (a super awesome core dev who does great, accessible presentations) about dicts in Python 3.6.

https://www.youtube.com/watch?v=p33CVV29OG8

A few other Hettinger videos

https://www.youtube.com/watch?v=wf-BqAjZb8M
https://www.youtube.com/watch?v=OSGv2VnC0go
https://www.youtube.com/watch?v=Bv25Dwe84g0

# ¿ Jan 19, 2017 23:15

Tigren: Oct 3, 2003

Eela6 posted:

This sounds like a job for coroutines and async/await, rather than multiprocessing. Are you sure you need multiprocessing? This sounds like an IO-bound rather than cpu-bound process, especially since you're hitting the filesystem so often.

If you use async , you can avoid locks entirely. Which is pretty cool! If you're not familiar with async in python 3.6 (and considering it's so new, few are), consider watching Raymond Hettinger's recent talk from the python convention in Russia. The asyncio module in the standard library is what you're looking for.

(asyncio is available in python 3.4 and 3.5, but it's easier and clearer in python 3.6 with the new async and await keywords, not to mention asynchronous comprehensions)

Like one of these!

Tigren posted:

https://www.youtube.com/watch?v=Bv25Dwe84g0

https://www.youtube.com/watch?v=p33CVV29OG8
https://www.youtube.com/watch?v=wf-BqAjZb8M
https://www.youtube.com/watch?v=OSGv2VnC0go

Tigren fucked around with this message at 04:46 on Jan 20, 2017

# ¿ Jan 20, 2017 04:41

Tigren: Oct 3, 2003

Jose Cuervo posted:

I have written a small web scraper script that gets a certain piece of information from a website.

I want to have the script run (i.e., get the piece of information from the website) every hour for a week.

Is the Pythonic way to do this to simply write a for loop that calls the web scraper script and uses the time.sleep() function to implement an hour delay in between calls?

One downside to this way is that if the computer restarts for updates etc. the for loop will stop. Is there a solution that works around this?

Leave the scheduling to the OS (cron if linux or mac and Task Scheduler if windows) and store state in a sqlite DB or CSV file.

Simple is better than complex.
Although practicality beats purity.

# ¿ Jan 30, 2017 21:55

Tigren: Oct 3, 2003

Jose Cuervo posted:

I'm running Windows 7.

So get task scheduler to run the script once an hour for a week and store the output in a file/ database so that if the computer restarts I have the information saved? Cool.

Not quite sure what you mean by 'practicality beats purity'.

Yep, you got it. Python has fantastic Sqlite support as well as csv support.

Like FoiledAgain said, check out PEP 20. Sometimes, if it's more practical, it's better to do things either unpythonic or even outside of Python. PEP 20 is not the law, but typically a great set of guidelines.

# ¿ Jan 30, 2017 23:38

Tigren: Oct 3, 2003

Thermopyle posted:

In other news I've been spending some time recently converting a bunch of bash scripts to python because gently caress bash.

I came across the plumbum library that's pretty neat for easing this type of work.
code:
>>> from plumbum import local
>>> ls = local["ls"]
>>> ls
LocalCommand(<LocalPath /bin/ls>)
>>> ls()
u'build.py\ndist\ndocs\nLICENSE\nplumbum\nREADME.rst\nsetup.py\ntests\ntodo.txt\n'
>>> notepad = local["c:\\windows\\notepad.exe"]
>>> notepad()                                   # Notepad window pops up
u''                                             # Notepad window is closed by user, command returns
code:
>>> chain = ls["-a"] | grep["-v", "\\.py"] | wc["-l"]
>>> print chain
/bin/ls -a | /bin/grep -v '\.py' | /usr/bin/wc -l
>>> chain()
u'13\n'
...and a ton of other features.

That does seem fun, but why not learn the Standard Library os module and rely on subprocess when needed? They're already included for you.

# ¿ Feb 5, 2017 03:18

Tigren: Oct 3, 2003

huhu posted:

I'm building a webcam that pushes images to the internet after capture. I found a script to upload the images to Dropbox and it works great. However, I've found an issue that if the internet goes down mid upload, no error is returned and the upload just gets stuck at whatever percent its at. This wouldn't really be an issue however my hackspace's internet is utter garbage and I suspect the internet will go down pretty often. Is there some nifty way to detect a stuck upload and break?

If your script uses the Dropbox v2 API, it looks like it raises a requests.exceptions.ConnectionError when it times out.

code:

ConnectionError: ('Connection aborted.', timeout('The write operation timed out',))

Tigren fucked around with this message at 03:34 on Feb 10, 2017

# ¿ Feb 10, 2017 02:14

Tigren: Oct 3, 2003

huhu posted:

I think that error message is hidden away in the script I found on Github.

Can you link to the script you're using? The Dropbox python API is very straight forward if you just want to write your own.

Python code:

In [1]: import dropbox
In [2]: import requests
In [3]: dropbox_client = dropbox.Dropbox(API_TOKEN)
In [4]: try:
            with open('test', 'rb') as f:
                upload = dropbox_client.files_upload(f.read(), '/path/to/file')
        except (requests.exceptions.ConnectionError) as err:
            print('Error uploading picture: ', err)
        else:
            print('Upload succeeded: ', upload)

quote:

Upload succeeded: FileMetadata(name='file', id='id:UmsjUZ9eogAAAAAAAAAACg', client_modified=datetime.datetime(2017, 2, 10, 7, 1, 10), server_modified=datetime.datetime(2017, 2, 10, 7, 1, 10), rev='8540e8f17', size=6, path_lower='/path/to/file', path_display='/path/to/file', parent_shared_folder_id=None, media_info=None, sharing_info=None, property_groups=None, has_explicit_shared_members=None)

# ¿ Feb 10, 2017 08:08

Tigren: Oct 3, 2003

Rosalind posted:

I am about to give up on trying to learn Python.

I have completed every Python tutorial recommended to me. They all speak of this beautiful, elegant programming language and everything is so easy to do. Oh yes I know how to set strings to variables and make them all lower case and count the length of a string and add, divide, and subtract. Oh boy do I know how to do these things. But then I get to actually getting started with actually using Python to do anything useful and it turns into a poo poo show.

I spent 5 hours tonight just trying to do the following tonight and have failed miserably: import a CSV that I exported from R and then run basic descriptive statistics on it. I thought this would be a good first task as I have analyzed these data before in R and know them like the back of my hand.

I googled "Import csv from R and descriptive statistics python" (or some approximation thereof). I found people talking about something called "pandas." I then found people discussing Anaconda and Rodeo as "good environments" for data analysis in Python.

I installed Anaconda. It gave me a choice of several different things which did not seem relevant including RStudio??? I googled again and found that the one called Spyder was the one I needed. Every piece of code I tried to run in this Spyder thing gave me various error messages and no amount of googling got me any useful answers.

I then installed Rodeo. This looked like RStudio so I was happy at first, but again nothing seemed to work right. I found some code that someone posted to import a csv using a package called rpy2. I spent 2 hours getting rpy2 installed but I think I finally did after so much searching and then manually installing it because apparently it doesn't work normally in Windows and monkeying about with Paths and writing some custom pip thing that I don't even really know why it was necessary.

At this point at least I would run "import rpy2" and no error message popped up. Maybe I had broken it so much that it didn't even know to give me an error message because none of the rpy2 functions would actually work.

My questions now are: 1. How the heck does anyone ever learn how to use this? I'm the first to admit that I'm no programming guru, but I have some experience with several different languages and actually getting started with Python makes absolutely no sense. Everything feels like the biggest clusterfuck of dependencies and "install X, Y, and Z to get W to work but to get X to work install A and B which require C to run on Windows." 2. Is there a simple, idiot-proof (because that's what I am apparently) guide to moving from R to Python for data analysis and hopefully eventually machine learning?

https://www.dataquest.io/blog/python-vs-r/

That's from the first page of a Google search for "r vs python statistics".

# ¿ Feb 11, 2017 04:17

Tigren: Oct 3, 2003

Dominoes posted:

Hey dudes. What are the best ways to learn Python, for someone with no experience? I learned from Codeacademy, which was OK. The OP looks out of date.

https://learnpythonthehardway.org/python3/

# ¿ Feb 27, 2017 18:54

Tigren: Oct 3, 2003

huhu posted:

I have some Flask code:
code:
    {% for image in gallery %}
        <img src ="{{ url_for('static', filename = 'img/image') }}">
    {% endfor %}
gallery is a list containing a bunch of filenames (i.e. "a.jpg", "b.jpg") How do I pass image (i.e. "a.jpg") from the for loop into 'img/image' so that it appears as 'img/a.jpg' instead?

Edit - Is this the best way?
code:
    {% for image in gallery %}
        <img src ="{{ url_for('static', filename = 'img/')}}{{ image }}">
    {% endfor %}

I'm pretty sure anything inside {{}} is just Python, so normal string concatenation applies. Use a + for ease.

# ¿ Mar 7, 2017 04:33

Adbot: ADBOT LOVES YOU

# ¿ Apr 27, 2024 11:51

Tigren: Oct 3, 2003

# ¿ Mar 7, 2017 20:47

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.