Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

politicorific: Sep 15, 2007

finally, django documentation out I've been waiting, but for now I'm going to focus on building the basic stuff until I fully understand what all's going on. So here's my SQL question:

I'm using Gadfly to learn how to access databases from python. I'm building a prototype for a website that will end up storing about 3000-5000 rows. A lot of the information is very similar - 10 columns and maybe only 3 records are slightly different. The problem is that those 3 different records are all I can search on. I'm making this too complicated.

Let's say I have 2 records in the database and I do the following:

code:

select * from table

Let's say this gives two results:

code:

(first, result)
(second, result)

now I want to store the result to a variable

code:

for row in cursor.fetchall():
      print row

This ends up randomly assigning either (first, result) or (second, result) to row.

How do I manipulate multiple rows, return the two options, or ask the user for more input?

# ¿ Dec 20, 2007 13:22

Adbot: ADBOT LOVES YOU

# ¿ May 5, 2024 17:11

politicorific: Sep 15, 2007

okay I guess I should've used my exact code

code:

c = g.cursor()
c.execute("""SELECT
combined,root1,root2,root3,
root4,root5,root6,participle,
tablenumber,intrans,english
FROM englishtable
ORDER BY
combined,root1,root2,root3,
root4,root5,root6,participle,
tablenumber,intrans,english
""")
c.description
print c.pp()
print "remember - right to left encoding is messing with the output"
for row in c.fetchall():
    print row

this returns this:

code:

('\xd7\x90\xd7\x91\xd7\x93', '\xd7\x90', '\xd7\x91', '\xd7\x93', '', '', '', '', '118', 'intrans', 'lose')
('\xd7\x90\xd7\x91\xd7\x93', '\xd7\x90', '\xd7\x91', '\xd7\x93', '', '', '', '', '91', 'trans', 'lose')

then if I do something like

code:

 z = row
print z

This will assign a single cursor selection to row, print it, then go ahead and assign another cursor selection to row and overwrite it.
How do I store multiple rows to different variables so that python/gadfly isn't overwriting them? The problem isn't the select statement, I'm getting the data I want, I just want all of it - it's the fetchall() statement that I'm having a hard time understand

# ¿ Dec 21, 2007 03:55

politicorific: Sep 15, 2007

here's an easy one hopefully.
let's say I have a bunch of different strings stored in class "Shop"
so that Shop.a = "string", Shop.b = "strang", Shop.c = "streng", ect...

now I want to remove a number of characters from those strings and replace them. First I try writing a function:

code:

def remove(x):
    x=x.split('i')  #'i' represents a unicode string
    x=''.join(x)
    return x

but if I run remove(Shop.a) it doesn't store the output to Shop.a, x is treated as a local variable.

Am I doing this wrong? Should I define this function within the "Shop" class?

The next thing that's bothering me, I don't want to have to write

code:

 remove(Shop.a)
remove(Shop.b)
remove(Shop.c)
remove(Shop.d)
...

can I define a list and then use a "for" command save my wrists?

# ¿ May 29, 2008 17:31

politicorific: Sep 15, 2007

posted this in the django thread but I didn't get any response. Here's my issue - I have a database encoded in utf-8 values which I am trying to port to google's appengine code. The thing is that appengine doesn't support unicode natively in it's tools to convert databases and the suggested patches are not working with my csv files. So I'm exploring alternatives.

the alternative idea I've come up with is to build a program that loads a url/fills out forms and saves it in the appengine datastore.

I'm not exactly sure what would be the best method for doing this, is there a decent tutorial on how to use the http aspects of python for this? Or a better way to get my data into appengine?

# ¿ Jun 25, 2008 16:53

politicorific: Sep 15, 2007

I want to check the last character of a string, and if it matches my condition, I want to replace it with a specific character, but only the end character, not anything in the middle.

code:

def check():
    a = "reader"
    if a[-1:]=="r":
        a=a[0:-1]
        a=a+"R"

Each character has a single mapping, so that r=R, s=S, p=V, ect, only once.

Rather than make hundreds of copies of this code, specifying each condition, how can I do this elegantly? I'd like to learn and grow my skills.

# ¿ Feb 12, 2009 08:05

politicorific: Sep 15, 2007

tef posted:

code:

awesome working code

Thank you, took me a little while to make it work with unicode, but hooray - reusable code!

Next question, how do I return on a value? Python docs say "a bare return indicates that the generator is done and will cause StopIteration to be raised."

code:

>>> mv = "here is a string"
>>> mcv = ("a","e","i","o","u")
>>> def stripvowels(foo,bar):
...	for x in bar:
...		if x in foo:
...			foo = foo.replace(x,'')
...			print foo			
>>> stripvowels(mv,mcv)
here is  string
hr is  string
hr s  strng

if I add "return foo" after the "print foo" line, I get:

code:

>>> stripvowels(mv,mcv)
here is  string
'here is  string'
>>>

How do I return this?

# ¿ Feb 12, 2009 20:43

politicorific: Sep 15, 2007

deimos posted:

The last one is the return value.

Ah, I was trying to change the variable outside the function without the use of globals or classes... trying to hammer this all back inside my head.

# ¿ Feb 12, 2009 21:09

politicorific: Sep 15, 2007

It's been 6 years since I read this thread(!). Well, I'm still a lovely programmer, but am looking to fix that.

In short, I'm looking for information on running loops over lists containing lists.

I built this 7 segment LED display using shift registers to interface with my raspberry pi:
http://forums.somethingawful.com/showthread.php?threadid=2734977&pagenumber=311#post447919014

I need assistance building the "rendering engine" because the existing libraries I found, listed below, are designed as demonstrations or work with only a single digit. Is there a library out there that already works with multiple 7 segment displays? My final device will be between 9-12 digits.

https://github.com/mignev/shiftpi
https://github.com/shrikantpatnaik/Pi7SegPy
https://github.com/m01/sevensegment

Addressing the segments looks like this: 10 millions place far left, highest outputs(56-63), ones-place (0-7). Here's what 8 digits looks like, even though I only have 5 digits built at the moment.

10,000,000 1,000,000 100,000 10,000 1,000 100 10 1
[55-63] [48-55] [40-47] [32-39] [24-31] [16-23] [8-15] [0-7]

These are the pin outs from the 595 to the segments. Technically the 595 has these alphabetically labeled.
A = 1
B = 0
C = 4
D = 5
E = 6
F = 3
G = 2
DP = 7 #decimal point

Here are some examples - to get these familiar outputs bring these pins high:
patterns =
"0": [A, B, C, D, E, F],
"1": [B, C],
"2": [A, B, D, E, G],
"3": [A, B, C, D, G],
"4": [B, C, F, G],
"5": [A, C, D, F, G],
"6": [A, C, D, E, F, G],
"7": [A, B, C],
"8": [A, B, C, D, E, F, G],
"9": [A, B, C, D, F, G],
".": [DP]
}

Okay so here's my idea, I wonder if there's a better way:

Let's say I have an IP address which I want to display: "192.168.0.1" - a string with 8 digits and 3 decimal points. Let's assume I have 8 shift registers/digits so I don't have overflow/or scroll the device.

First, my rendering engine is going to split that string into a list [1,9,2,.,1,6,8,.,0,.,1]. #Note that this has 11 elements

Next, each element of the list is looked up in the patterns dictionary/truth table to find which pins to bring high.

The result being : [[B, C],[A, B, C, D, F, G],[A, B, D, E, G],[DP],[B, C],[A, C, D, E, F, G],[A, B, C, D, E, F, G],[DP],[A, B, C, D, E, F],[DP],[B, C]].

Now run a loop to combine the decimal point, [DP] with the preceding list element. This is just the lazy way I see of doing it. I could eventually make a truth table with 255 patterns.

[[B, C],[A, B, C, D, F, G],[A, B, D, E, G, DP],[B, C],[A, C, D, E, F, G],[A, B, C, D, E, F, G, DP],[A, B, C, D, E, F],[DP, B, C]]. # back to 8 elements

The numerical values of this list are:
[[0, 4],[1, 0, 4, 5, 3, 2],[1, 0, 5, 6, 2, 7],[0, 4],[1, 4, 5, 6, 3, 2],[1, 0, 4, 5, 6, 3, 2, 7],[1, 0, 4, 5, 6, 3],[7, 0, 4]]

Next, flip the list so that "1" or [0,4] or [B,C] segments are in the left side's 10 millions place.

[[7, 0, 4],[1, 0, 4, 5, 6, 3],[1, 0, 4, 5, 6, 3, 2, 7],[1, 4, 5, 6, 3, 2],[0, 4],[1, 0, 5, 6, 2, 7],[1, 0, 4, 5, 3, 2],[0, 4]] # probably good to copy this variable before going further; I guess for strings larger than 8 digits, I can pop values out and do a marquee scrolling effect

If I write this to the shift register, it will only address the "ones" place.

I need to add multiples of 8 to the values of each element in the list: [0]+0, [1]+8, [2]+16, [3]+24, [4]+32, [5]+40, [6]+48, 7[+56]

So on so that I end up with:

[[7, 0, 4],[9, 8, 12, 13, 14, 11],[17, 16, 20, 21, 22, 19, 18, 24],[25, 28, 29, 30, 27, 26],[32, 36],[41, 40, 45, 46, 42, 47],[49, 48, 52, 53, 51, 50],[56, 60]]

At this point, I need to write a loop that goes over every element in the list and sublist and turns it "high" or "on". I suppose this just turns into a string of 64 rising and falling voltages. I then latch the values, display it, and then repeat with new information.

Does this make sense?

There is a lot more I want to do: like setting PWM for individual segments. Centering/right justifying/left justifying. Building digits/characters segment by segment, have digits "fly" in from the left and right.
But for now, I just need a way to use the entire display, and not just the first digit. Does this sound like a smart way to go about it?

# ¿ Jul 22, 2015 16:33

politicorific: Sep 15, 2007

I would like a gentle push in the right direction.

I document web interfaces at my job. I would like to automate capturing screen shots.

I've chosen Python as it's my strongest language. I've already checked this out:
https://automatetheboringstuff.com/chapter11/

Here are my problems:

Must work in Internet Explorer
Need to do rolling captures for extra long pages
Might need to keep track of sessions/logins/passwords
Need to avoid following "Save/Apply" pages which may reboot the router
I don't have a list of all the pages
The webui uses frames

For the first two points, I can use the screen capture program Greenshot to do rolling screen captures of Internet explorer and bind this to a short cut, or otherwise manipulate the program in the notification area. I also don't know how to pass passwords to a box using python.

Unfortunately the last point is beyond my css/html understanding. Our company uses frames on our web UIs to keep the menu options visable. The address bar simply reads "192.168.1.1" or whatever with the real 'page' hidden. I want to automatically scrape the the webpages for links, build a list, take a screenshot, and name them based on some text embedded in the page until finished. I've thought about using an offline page viewer application, but I'm not sure how that helps me.

If I had code that could do this, it would probably save me a few hundred hours of work per year, and perhaps a couple weeks of man-hours when multiplied out to my team.

# ¿ Dec 9, 2015 06:16

politicorific: Sep 15, 2007

Munkeymon posted:

Use Selenium's Python bindings and http://stackoverflow.com/questions/3422262/take-a-screenshot-with-selenium-webdriver

Part of their API lets you pick a frame to interact with http://stackoverflow.com/questions/13567128/selenium-frames

Or you can just record your own actions and replay them automagically if hand-coding the workflow is too much http://www.seleniumhq.org/projects/ide/

Rats... selenium is Firefox only. Unless I learn this: https://www.youtube.com/watch?v=GxTHU_91Z1Q

I spent several hours researching possible solutions. This program webshot gave me some ideas:
http://www.websitescreenshots.com/usage.php#faq
But, unfortunately I will still need to take some screenshots by hand for certain dialogs which can't be automated and webshot isn't rendering the images to match the screenshots I took. I tried adjusting the registry to change Internet Explorer 11 to run in standards mode instead of quirks mode, but I didn't have any luck.

Here's what I came up with:

code:

from bs4 import BeautifulSoup
import webbrowser
import time
soup = BeautifulSoup(open("menu.html"))
linklist =[]
for link in soup.find_all('a'):
    print(link.get('href'))
    linklist.append(link.get('href'))
print(linklist)
    
for x in linklist:
        print(x)
        webbrowser.get("C:/Program Files/Internet Explorer/iexplore.exe %s").open(x)
        #time.sleep(2)

This scrapes a previously downloaded .html file, which I got using wget+cookie info, and then uses the webbrowser command to open the page in IE.

Next I want to use Greenshot to take a screenshot. I tried using SendKeys, but it hasn't been updated since 2003/2008 and doesn't work without visual studio installed.

Instead I found pywinauto.

https://github.com/pywinauto

However, I'm having trouble calling up a list of available windows, plus Greenshot runs in the taskbar.
https://pywinauto.googlecode.com/hg/pywinauto/docs/code/pywinauto.taskbar.html
pywinauto.taskbar.ClickSystemTrayIcon(button)

Anyone have any experience using this?

# ¿ Dec 10, 2015 06:12

politicorific: Sep 15, 2007

Okay, I'm getting somewhere with pywinauto, I just need another kick in the right direction:

This clicks on the greenshot icon in the system tray/notification area. Greenshot produces a popup that disappears as soon as I click away, so wrote some code to try and find the window ID.

code:

from time import sleep
import pywinauto
from pywinauto.application import Application
from pywinauto import findwindows

app = pywinauto.application.Application().connect_(path = "explorer")
shelltraywnd = app.Shell_TrayWnd
shelltraywnd.Wait('ready')
toolbarwindow = shelltraywnd.Toolbar
toolbar_button = toolbarwindow.Button(u'Greenshot')
temp1=(findwindows.find_windows())
print(temp1)
toolbar_button.ClickInput()
sleep(2)
temp2 = (findwindows.find_windows())
print(temp2)

This produces two lists of open windows:
[262580, 525414, 265030, 722260, 132856, 1180792, 657050]
[66056, 461540, 262580, 525414, 265030, 722260, 132856, 1180792, 657050]

Which leads me to believe that 66056 is the greenshot popup window. Now all I need to do is tell it to click the 5th menu option down. But how do I do that? How do I connect to this window? How do I confirm that greenshot is 66056?

edit - found this text: "Often when you click/right click on an icon - you get a popup menu. The thing to remember at this point is that the popup menu is part of the application being automated not part of explorer." So it's possible I'm connected to the wrong thing.

politicorific fucked around with this message at 11:27 on Dec 10, 2015

# ¿ Dec 10, 2015 10:10

politicorific: Sep 15, 2007

Hi, I'm trying to grab some air quality data using beautiful soup from this page, but I'm not sure what I'm doing wrong.

Visit this page, I'm trying to get the column under "list" and the AQI - the biggest rectangular box on the page.

http://aqicn.org/city/newyork

code:

from bs4 import BeautifulSoup
import urllib.request, urllib.parse, urllib.error

r="http://aqicn.org/city/newyork/"
page = urllib.request.urlopen(r).read()
soup = BeautifulSoup(page)

print(soup.find_all(id="cur_pm25"))
print(soup.find_all(id="cur_pm10"))
print(soup.find_all(id="cur_o3"))
print(soup.find_all(id="cur_no2"))
print(soup.find_all(id="cur_co"))
print(soup.find_all(id="cur_uvi"))
print(soup.find_all(id="cur_t"))
print(soup.find_all(id="cur_p"))
print(soup.find_all(id="cur_h"))
print(soup.find_all(id="aqivalue"))

The output does not match what's in my browser. I think I'm supposed to have "td#cur_XX.tdcur"

# ¿ Jan 28, 2016 13:56

politicorific: Sep 15, 2007

Awesome. Thank you both. That really is a bit more complicated than I expected, but the beautiful soup documentation I found was very basic and with simple examples. I knew Aqi was a different form of data, but I'm still unsure of the difference.

I believe my visual studio version is broken, so pip won't install Lxml on my desktop, but my raspberrypi should do just fine.

# ¿ Jan 28, 2016 19:42

politicorific: Sep 15, 2007

Jesus christ...

So I tried New York, but if i put in Taipei... no dice.

url = 'http://aqicn.org/taiwan/songshan'

Also, Anaconda/Spyder is really finicky about lxml... half the time it cannot find "import lxml.html"

# ¿ Jan 29, 2016 10:00

Adbot: ADBOT LOVES YOU

# ¿ May 5, 2024 17:11

politicorific: Sep 15, 2007

Hi accipter, sorry for the late reply, life got in the way.

I'm having trouble getting your scraping code to work with a different URL

accipter posted:

Second, I prefer to scrape with lxml.html rather than BeautifulSoup. I know this doesn't answer your question, but the following should help.
Python code:
import pprint
import lxml.html
import requests

url = 'http://aqicn.org/city/newyork'
r = requests.get(url)

cut
Edit: Note that selecting the table, and then rows within the table isn't strictly needed for this website. The rows could have been selected directly from the root with: rows = root.xpath('//td[@class="tdcur"]'), however I thought it would be good to show a relative path (notice the start with . in the code) for html that is not as nicely formatted.

code:

import pprint
import lxml.html
import requests

url = 'http://aqicn.org/city/usa/newyork/ps-314/'
#url = 'http://aqicn.org/city/taiwan/songshan/'

r = requests.get(url)

root = lxml.html.fromstring(r.content)

# Find the table containing all of the information. To get this XPath
# I inspect an element in Chrome and then copy the XPath via the context 
# menu.
e_table = root.xpath('//*[@id="citydivmain"]/div/div/div/table[3]')[0]
#e_table = root.xpath('//*[@id="citydivmain"]/div/div/div/table[3]')[0]
#print(type(e_table))

# Now we want to get all of the rows that have current values
rows = root.xpath('//td[@class="tdcur"]')

def clean_id(s):
    return s.split('_')[-1]

def to_value(s):
    try:
        return int(s)
    except ValueError:
        return None

data = {clean_id(r.get('id')): to_value(r.text) for r in rows}

# Add the AQI value
data['aqi'] = to_value(root.xpath('//div[@class="aqivalue"]/text()')[0])

pprint.pprint(data)

I get this error:

code:

Traceback (most recent call last):
  File "E:\Patrick White - Personal Time\python\AQI\aqi.py", line 31, in <module>
    data = {clean_id(r.get('id')): to_value(r.text) for r in rows}
  File "E:\Patrick White - Personal Time\python\AQI\aqi.py", line 31, in <dictcomp>
    data = {clean_id(r.get('id')): to_value(r.text) for r in rows}
  File "E:\Patrick White - Personal Time\python\AQI\aqi.py", line 27, in to_value
    return int(s)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

My xpath appears to be close enough, but a little different when I inspect:
//*[@id="citydivmain"]/div/div/div/table[3]/tbody

This is really maddening because the tutorials I found barely scratch the surface of what I want to be able to do.

# ¿ Mar 3, 2016 12:59

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.