Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

mekkanare: Sep 12, 2008; We have detected you are using ad blocking software.

Please add us to your whitelist to view this content.

Hey I've got a few (hopefully) simple questions.

I'm in a networking class right now and we're using Python, and one of our upcoming projects is to implement a basic server to do remote commands.
So I'm trying to get the base case down to expand into this project when we officially begin it.
What I have so far can be seen here.
I've left a comment describing my issue but I'll post it here as well.

What I want to do:
Client connects to server.

Server sends message "Name?" to client.
Client replies with name.
Server says "You are *name* at *ip*"
Repeat again with the same client connected.

What happens:
On the first repeat, the message received by the client says "Name?" but it doesn't follow through the if statement again and just hangs.

What I think is happening:
e2:Server sends the messages one after another, but if I put in a sleep command it works fine (obviously).
My questions:
e: Thanks tef.
e3: Solved (Just do an OK response :downswords:

), thanks again.
Any help is appreciated, thanks in advance goons.

mekkanare fucked around with this message at 16:27 on Feb 28, 2016

# ? Feb 27, 2016 23:50

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 19:52

tef: May 30, 2004; -> some l-system crap ->

mekkanare posted:

My questions:
Is there such a thing as flushing a TCP buffer? If so, how do I do it for Python?

You could try

code:

f = connectionSoc.makefile(); f.readline()

# ? Feb 28, 2016 00:28

Dominoes: Sep 20, 2007

Looking for SQLalchemy advice. I'm trying to filter by a [json] list, and day.

Python code:

s.query(a).filter(symbol in json.loads(a.symbols), a.published >= day, a.published < day +
                             dt.timedelta(days=1)).all()

s is the session; a is the model. The >/< thing is a workaround because SQLalchemy can't filter using the date() method. I receive the following error: 'TypeError: the JSON object must be str, not 'InstrumentedAttribute'', related to the JSON. I could probably work around this with a symbols table and many-to-many, but that seems overkill. Alternatively, I could query just by date, and use python to filter by the json. (slower than a direct query):

Python code:

    articles = s.query(a).filter(a.published >= day, a.published < day +
                             dt.timedelta(days=1)).all()
    return (art for art in articles if symbol in json.loads(art.symbols))

That works, but is slower than a direct query would be. (and still uses the janky date filtering) Any recommendations for a module to improve the SQLAlchemy API? Seems kind of a mess, hence the s and a code shorteners.

Dominoes fucked around with this message at 21:54 on Feb 28, 2016

# ? Feb 28, 2016 21:48

Space Kablooey: May 6, 2009

If symbols were a relationship I would do:

Python code:

s.query(a).filter(a.symbols.in_(symbol), a.published >= day, a.published < day + dt.timedelta(days=1)).all()

I'm not really sure that this will work in your case, considering your symbol is (probably) a string and not a relationship.

Space Kablooey fucked around with this message at 22:07 on Feb 28, 2016

# ? Feb 28, 2016 22:04

hooah: Feb 6, 2006; WTF?

Say I have a string test='hello\tworld'. Why does test.find(string.whitespace) return -1? I know tab is in string.whitespace.

# ? Mar 1, 2016 17:15

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

hooah posted:

Say I have a string test='hello\tworld'. Why does test.find(string.whitespace) return -1? I know tab is in string.whitespace.

I'm guessing tab isn't the only thing in string.whitespace? .find searches for a string(needle) in a string(haystack), not any arbitrary sub-part of the needle in the haystack.

E: are you trying to figure out if there's any whitespace or find the first whitespace character?

Munkeymon fucked around with this message at 17:25 on Mar 1, 2016

# ? Mar 1, 2016 17:20

vikingstrike: Sep 23, 2007; whats happening, captain

Yeah, string.whitespace is a longer string than just "\t"

# ? Mar 1, 2016 17:22

hooah: Feb 6, 2006; WTF?

I'm trying to find the first whitespace character to use for slicing. The file I'm working with (average wage index from the Social Security Administration) unfortunately uses spaces for the first two and tabs for the rest, and the last line has a wage range 50,000,000.00 and over, so I want to replace the "over" with some large number so I can split that line the way I do all the others.

# ? Mar 1, 2016 17:35

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

Well the regular expression /\s/ will match all white space characters, hint hint

Also it's really lovely of them to mix tab and space delimiters JFC

# ? Mar 1, 2016 17:51

hooah: Feb 6, 2006; WTF?

Munkeymon posted:

Well the regular expression /\s/ will match all white space characters, hint hint

Also it's really lovely of them to mix tab and space delimiters JFC

Yeah, I know about regular expressions, but I'm a TA working through the students' project so I can help them out better, and they don't know about those. I guess since the file will always have the same format I can just hard-code the first two as spaces and then the tab.

Edit: duh, I can just replace "over" with some large number before I process that particular line.

hooah fucked around with this message at 18:33 on Mar 1, 2016

# ? Mar 1, 2016 18:12

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

you could do something like

Python code:

def find_space(msg):
    for i, c in enumerate(msg):
        if c in string.whitespace:
            return i
    return -1

and that should give you the index to the first whitespace character (or just pass back the character which is probably what you want anyway)

What does the data look like anyway? Are the tab-delimited lines using spaces that you need to avoid splitting on?

# ? Mar 1, 2016 19:21

hooah: Feb 6, 2006; WTF?

Here's the first three lines:

pre:

Distribution of wage earners by level of net compensation
Net compensation interval	Number	Cumulative Number	Percent of total	Aggregate amount	Average amount
0.01 - 4,999.99	22,574,440	22,574,440	14.27075	46,647,919,125.68	2,066.40

Note that in addition to the delimiter fuckery, they're also using some bizarre character for the dash that shows up as — in my default Firefox encoding.

# ? Mar 1, 2016 19:28

Munkeymon: Aug 14, 2003; Motherfucker's got an
armor-piercing crowbar! Rigoddamndicu𝜆ous.

They're not loving with the delimiter, you're just trying to treat the first column as two columns when it's actually a bucket. The CSV module with delimiter set to tabs should handle that just fine.

# ? Mar 1, 2016 19:51

ManicJason: Oct 27, 2003; He doesn't really stop the puck, but he scares the hell out of the other team.

I've spent somewhere around four hours trying to figure out how to stub a function in a unit test, and I am totally lost.

I'm working on an IRC bot, specifically a Supybot plugin. The bot does a ton of fetching from the web. I would like to replace those requests with stub data for unit tests. My plugin has the IRC bot callback class (subclass of Supybot's callbacks.Plugin), a data class, and a web connection class. The web connection class has tons of methods that will be very easy to stub, such as "fetchRawMainPageHTML." All I need to do is replace the implementation of a specific method in the web connection class with one that shovels some test data back.

I have tried a billion different ways of stubbing the function and hopelessly bashed my face into the screen a thousand times with mock and magicmock and patch, and nothing makes sense.

I am coming from Objective-C world where it would be trivial to swap the implementation of the "fetchRawMainPageHTML" method with a stub, and that would affect all instances of the class no matter who owns them or if they already existed. I'm getting the impression that Python is going to require a specific namespace of exactly who owns the relevant instance of the web connection class. I am totally lost here since my tests are running from within Supybot's test framework; I am not directly instantiating my callback or any of its helper classes.

Am I overlooking some stupidly simple solution like MyWebConnectionClass.fetchHomePage = stubFunctionLol?

# ? Mar 1, 2016 23:36

hooah: Feb 6, 2006; WTF?

Another kind of stupid question. I'm processing a file that represents 32 instances of an experiment, which get grouped into chunks of 8 runs. So 1-8 => chunk 1, 9-16 => chunk 2, etc. Is there a nice way to map the number from 1-32 to the chunk number (1-4)? I tried modulo and that didn't work because 8 % 8 = 0, when it should be 1. Right now I'm just doing it with a big dictionary.

# ? Mar 2, 2016 02:36

pmchem: Jan 22, 2010

Divide by 8 (ensure float) and ceiling round.

# ? Mar 2, 2016 02:51

hooah: Feb 6, 2006; WTF?

pmchem posted:

Divide by 8 (ensure float) and ceiling round.

Totally forgot about floor/ceiling. Thanks, that's much cleaner.

# ? Mar 2, 2016 02:59

pmchem: Jan 22, 2010

hooah posted:

Totally forgot about floor/ceiling. Thanks, that's much cleaner.

I cut my teeth on Fortran. No dictionaries there, you use the tools to manipulate floats that 1977 gave you!

# ? Mar 2, 2016 03:01

hooah: Feb 6, 2006; WTF?

Ack, I misspoke. I had figured that mapping out. The one I couldn't figure out without a dictionary was how to map 1, 9, 17, & 25 to 1; 2, 10, 18, & 26 to 2, and so on. The end result is to turn e.g. run #18 into chunk 3, run 2.

# ? Mar 2, 2016 03:32

pmchem: Jan 22, 2010

N_desired = 1 + modulo 8 of (N_raw -1) ? Posting from iPad so no code or deep thoughts, watching election returns.

# ? Mar 2, 2016 03:41

yippee cahier: Mar 28, 2005

pmchem posted:

N_desired = 1 + modulo 8 of (N_raw -1) ? Posting from iPad so no code or deep thoughts, watching election returns.

Seems right. hooah just needs to account for the 1-based index the input and output is presented in.

# ? Mar 2, 2016 04:55

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

You could use integer division since it's one of the good uses for it!

Python code:


chunk = ((i - 1) // 8) + 1
run = ((i - 1) % 8) + 1

Using non zero-based counting makes it more complicated since you have to translate to that and then back again. It would probably be neater to use an index starting at zero and add one to it when you're outputting readable numbers

Python code:


chunk = i // 8
run = i % 8

# ? Mar 2, 2016 05:06

politicorific: Sep 15, 2007

Hi accipter, sorry for the late reply, life got in the way.

I'm having trouble getting your scraping code to work with a different URL

accipter posted:

Second, I prefer to scrape with lxml.html rather than BeautifulSoup. I know this doesn't answer your question, but the following should help.
Python code:
import pprint
import lxml.html
import requests

url = 'http://aqicn.org/city/newyork'
r = requests.get(url)

cut
Edit: Note that selecting the table, and then rows within the table isn't strictly needed for this website. The rows could have been selected directly from the root with: rows = root.xpath('//td[@class="tdcur"]'), however I thought it would be good to show a relative path (notice the start with . in the code) for html that is not as nicely formatted.

code:

import pprint
import lxml.html
import requests

url = 'http://aqicn.org/city/usa/newyork/ps-314/'
#url = 'http://aqicn.org/city/taiwan/songshan/'

r = requests.get(url)

root = lxml.html.fromstring(r.content)

# Find the table containing all of the information. To get this XPath
# I inspect an element in Chrome and then copy the XPath via the context 
# menu.
e_table = root.xpath('//*[@id="citydivmain"]/div/div/div/table[3]')[0]
#e_table = root.xpath('//*[@id="citydivmain"]/div/div/div/table[3]')[0]
#print(type(e_table))

# Now we want to get all of the rows that have current values
rows = root.xpath('//td[@class="tdcur"]')

def clean_id(s):
    return s.split('_')[-1]

def to_value(s):
    try:
        return int(s)
    except ValueError:
        return None

data = {clean_id(r.get('id')): to_value(r.text) for r in rows}

# Add the AQI value
data['aqi'] = to_value(root.xpath('//div[@class="aqivalue"]/text()')[0])

pprint.pprint(data)

I get this error:

code:

Traceback (most recent call last):
  File "E:\Patrick White - Personal Time\python\AQI\aqi.py", line 31, in <module>
    data = {clean_id(r.get('id')): to_value(r.text) for r in rows}
  File "E:\Patrick White - Personal Time\python\AQI\aqi.py", line 31, in <dictcomp>
    data = {clean_id(r.get('id')): to_value(r.text) for r in rows}
  File "E:\Patrick White - Personal Time\python\AQI\aqi.py", line 27, in to_value
    return int(s)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

My xpath appears to be close enough, but a little different when I inspect:
//*[@id="citydivmain"]/div/div/div/table[3]/tbody

This is really maddening because the tutorials I found barely scratch the surface of what I want to be able to do.

# ? Mar 3, 2016 12:59

Plasmafountain: Jun 17, 2008

Im trying to kludge together a queuing system. I have a bunch of dicts that hold various parameters, but what I'd really like to do in this first instance is to sort by priority before checking for the presence of files to see if the task has been run or not.

If Ive got dicts structured like:

code:

simQ['Folders']['Folder1']['Priority']=3
simQ['Folders']['Folder2']['Priority']=1
simQ['Folders']['Folder3']['Priority']=2
simQ['Folders']['Folder4']['Priority']=4

How can I sort by the priority of each folder in folders?

# ? Mar 4, 2016 12:15

Jewel: May 2, 2009

Zero Gravitas posted:

Im trying to kludge together a queuing system. I have a bunch of dicts that hold various parameters, but what I'd really like to do in this first instance is to sort by priority before checking for the presence of files to see if the task has been run or not.

If Ive got dicts structured like:
code:
simQ['Folders']['Folder1']['Priority']=3
simQ['Folders']['Folder2']['Priority']=1
simQ['Folders']['Folder3']['Priority']=2
simQ['Folders']['Folder4']['Priority']=4
How can I sort by the priority of each folder in folders?

You can't sort dicts since they're unordered, so you'd have to put the data into an array in whatever way you want to structure it.

Python code:

folderList = [(x, simQ['Folders'][x]) for x in simQ['Folders']]

print folderList
print ""

for folder in folderList:
    print "Folder: {0} - Priority: {1}".format(folder[0], folder[1]['Priority'])

print "\n=============\n"

folderList = sorted(folderList, lambda x, y: x[1]['Priority'] - y[1]['Priority'])

print folderList
print ""

for folder in folderList:
    print "Folder: {0} - Priority: {1}".format(folder[0], folder[1]['Priority'])

Output:

code:

[('Folder3', {'Priority': 2}), ('Folder2', {'Priority': 1}), ('Folder1', {'Priority': 3}), ('Folder4', {'Priority': 4})]

Folder: Folder3 - Priority: 2
Folder: Folder2 - Priority: 1
Folder: Folder1 - Priority: 3
Folder: Folder4 - Priority: 4

=============

[('Folder2', {'Priority': 1}), ('Folder3', {'Priority': 2}), ('Folder1', {'Priority': 3}), ('Folder4', {'Priority': 4})]

Folder: Folder2 - Priority: 1
Folder: Folder3 - Priority: 2
Folder: Folder1 - Priority: 3
Folder: Folder4 - Priority: 4

# ? Mar 4, 2016 13:01

Cingulate: Oct 23, 2012; by Fluffdaddy

Zero Gravitas posted:

Im trying to kludge together a queuing system. I have a bunch of dicts that hold various parameters, but what I'd really like to do in this first instance is to sort by priority before checking for the presence of files to see if the task has been run or not.

If Ive got dicts structured like:
code:
simQ['Folders']['Folder1']['Priority']=3
simQ['Folders']['Folder2']['Priority']=1
simQ['Folders']['Folder3']['Priority']=2
simQ['Folders']['Folder4']['Priority']=4
How can I sort by the priority of each folder in folders?

Might be another case for pandas.

# ? Mar 4, 2016 15:23

Dominoes: Sep 20, 2007

Zero Gravitas posted:

Im trying to kludge together a queuing system. I have a bunch of dicts that hold various parameters, but what I'd really like to do in this first instance is to sort by priority before checking for the presence of files to see if the task has been run or not.

If Ive got dicts structured like:
code:
simQ['Folders']['Folder1']['Priority']=3
simQ['Folders']['Folder2']['Priority']=1
simQ['Folders']['Folder3']['Priority']=2
simQ['Folders']['Folder4']['Priority']=4
How can I sort by the priority of each folder in folders?

Something like this. Note that it doesn't take your 'FolderX' bit into account; you'll have to restructure that.

Python code:

sorted(simQ.items(), key=lambda x: x['Folders']['Priority'])

# ? Mar 4, 2016 21:31

ufarn: May 30, 2009

What did I mess up, if pip keeps requiring sudo, and how do I fix it on OS X?

�Terminal.app� posted:

$ pip install -U pip
Collecting pip
Downloading pip-8.1.0-py2.py3-none-any.whl (1.2MB)
100% |████████████████████████████████| 1.2MB 343kB/s
Installing collected packages: pip
Found existing installation: pip 8.0.2
Uninstalling pip-8.0.2:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 209, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 317, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 725, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 752, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 266, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 303, in move
os.unlink(src)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/pip-8.0.2.dist-info/DESCRIPTION.rst'
You are using pip version 8.0.2, however version 8.1.0 is available.
You should consider upgrading via the �pip install �upgrade pip' command.

�Terminal.app� posted:

$ sudo pip install -U pip
Password:
The directory �/Users/ufarn/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory �/Users/ufarn/Library/Caches/pip� or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pip
Downloading pip-8.1.0-py2.py3-none-any.whl (1.2MB)
100% |████████████████████████████████| 1.2MB 390kB/s
Installing collected packages: pip
Found existing installation: pip 8.0.2
Uninstalling pip-8.0.2:
Successfully uninstalled pip-8.0.2
Successfully installed pip-8.1.0

�Terminal.app� posted:

$ which python
/usr/bin/python

$ which pip
/usr/local/bin/pip

$ $PATH
-bash:
/opt/local/bin:
/opt/local/sbin:
/opt/local/bin:
/opt/local/sbin:
/Users/ufarn/.rbenv/shims:
/Users/ufarn/.rbenv/bin:
/usr/local/bin:
/usr/local/bin:
/usr/bin:
/bin:
/usr/sbin:
/sbin:
/opt/X11/bin:
/usr/local/git/bin:
/usr/local/MacGPG2/bin:
/Users/ufarn/Downloads/go/bin:
/Users/ufarn/go/bin:
No such file or directory

Replaced �:� with �:\n�.

ufarn fucked around with this message at 12:09 on Mar 7, 2016

# ? Mar 7, 2016 12:05

sunaurus: Feb 13, 2012; Oh great, another bookah.

ufarn posted:

What did I mess up, if pip keeps requiring sudo, and how do I fix it on OS X?

It makes sense for pip to require sudo when you're installing packages into your whole system. If you want to use pip without sudo, just create a virtualenv for your project and install packages in that virtualenv (which is the best way to develop projects anyway, because then your system packages don't interfere with your project).

sunaurus fucked around with this message at 17:29 on Mar 7, 2016

# ? Mar 7, 2016 12:43

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

I'm trying to identify (but not actually hide) all "ads" on a page using selenium. I started writing the logic for it using easylist and then realized its possible I'm being an idiot. Do any of you know any built solutions for this problem?

My only thought so far has been to run chromedriver w/ ublock origin installed and then load up the page I need to test and then somehow access the log of blocked elements that it produces.

# ? Mar 7, 2016 16:37

Dominoes: Sep 20, 2007

I mentioned earlier that SQLalchemy query syntax is verbose:

Python code:

models.session.query(models.Person).filter(models.Person.first_name == 'David').all()

I've read a few guides implying table objects have query method, but can't find it in the SQLalchemy docs, nor make it work in my code. Is there any reason why I shouldn't use code like this? I rarely use inheritance, but this feels like an appropriate use:

Python code:

class QueryMethod:
    @classmethod
    def filter(cls, *args, **kwargs):
        return session.query(cls).filter(*args, **kwargs).all()


class Person(Base, QueryMethod):
    __tablename__ = 'people'

    id = Column(Integer, primary_key=True)
    first_name = Column(String(255))

New query syntax:

Python code:

models.Person.filter(models.Person.first_name='David')

Is there a way to remove the 'models.Person' bit inside the filter method call?

Dominoes fucked around with this message at 12:08 on Mar 8, 2016

# ? Mar 8, 2016 10:18

Flash Gordon: May 8, 2006; Death To Ming

I'm using numpy and need to perform some transformation. I might eventually end up writing my own solution for this but it seems like something that I might not need to reinvent the wheel on, so I figured I'd ask. The basic setup:

- I have one big array containing pressure values. This array has dimensions (1464, 1, 73, 144). (The singleton dimension isn't important now but might be in the future). The dimensions correspond to time values, pressure levels, latitude and longitude. The actual values for those dimensions are in 1d arrays of length 1464, 1, 73, 144.
- I need to transform this to a 2d array with five columns - time, level, latitude, longitude and pressure value.

My current plan is to flatten the pressure values to get a vector that will make up the pressure column. Then for the other columns I can just repeat or tile those vectors an appropriate number of times so they become 15389568 in length or whatever. I feel like this will work but I'm worried about somehow getting the alignment of the index values with the pressure values wrong. I feel like I might just want to test this by populating a test matrix with values from 1 to 15 million or whatever and make sure after the fact that the values are occurring at the indexed rows as intended.

Is there a more elegant way to do this?

# ? Mar 8, 2016 22:07

Nippashish: Nov 2, 2005; Let me see you dance!

Flash Gordon posted:

- I have one big array containing pressure values. This array has dimensions (1464, 1, 73, 144). (The singleton dimension isn't important now but might be in the future). The dimensions correspond to time values, pressure levels, latitude and longitude. The actual values for those dimensions are in 1d arrays of length 1464, 1, 73, 144.
- I need to transform this to a 2d array with five columns - time, level, latitude, longitude and pressure value.

Kronecker products are your friend.

code:

import numpy as np
from functools import reduce # not needed in python 2.7

# Values along different dimensions
time  = np.arange(5)
level = np.arange(6)
lat   = np.arange(7)
lon   = np.arange(4)

# Your data
pressure = np.random.uniform(size=(5, 6, 7, 4)) 

time_col  = reduce(np.kron, [time, np.ones_like(level), np.ones_like(lat), np.ones_like(lon)], 1)
level_col = reduce(np.kron, [np.ones_like(time), level, np.ones_like(lat), np.ones_like(lon)], 1)
lat_col   = reduce(np.kron, [np.ones_like(time), np.ones_like(level), lat, np.ones_like(lon)], 1)
lon_col   = reduce(np.kron, [np.ones_like(time), np.ones_like(level), np.ones_like(lat), lon], 1)

table = np.vstack([time_col, level_col, lat_col, lon_col, pressure.ravel()]).T

for row in table:
    print([x for x in row])

# ? Mar 8, 2016 23:23

SurgicalOntologist: Jun 17, 2004

I'm not sure if it provides a solution to this particular problem, but given the description of your dataset you may find xarray useful. It lets you label your array dimensions and refer to them by name.

# ? Mar 9, 2016 00:48

Flash Gordon: May 8, 2006; Death To Ming

Nippashish posted:

Kronecker products are your friend.

code:

import numpy as np
from functools import reduce # not needed in python 2.7

# Values along different dimensions
time  = np.arange(5)
level = np.arange(6)
lat   = np.arange(7)
lon   = np.arange(4)

# Your data
pressure = np.random.uniform(size=(5, 6, 7, 4)) 

time_col  = reduce(np.kron, [time, np.ones_like(level), np.ones_like(lat), np.ones_like(lon)], 1)
level_col = reduce(np.kron, [np.ones_like(time), level, np.ones_like(lat), np.ones_like(lon)], 1)
lat_col   = reduce(np.kron, [np.ones_like(time), np.ones_like(level), lat, np.ones_like(lon)], 1)
lon_col   = reduce(np.kron, [np.ones_like(time), np.ones_like(level), np.ones_like(lat), lon], 1)

table = np.vstack([time_col, level_col, lat_col, lon_col, pressure.ravel()]).T

for row in table:
    print([x for x in row])

This is great, thanks!

SurgicalOntologist posted:

I'm not sure if it provides a solution to this particular problem, but given the description of your dataset you may find xarray useful. It lets you label your array dimensions and refer to them by name.

Oh wow, I'm actually working with NetCDF data so this is perfect! Now I gotta rewrite my project to use this

# ? Mar 10, 2016 19:04

Dominoes: Sep 20, 2007

Dominoes posted:

I mentioned earlier that SQLalchemy query syntax is verbose:
Python code:
models.session.query(models.Person).filter(models.Person.first_name == 'David').all()
I've read a few guides implying table objects have query method, but can't find it in the SQLalchemy docs, nor make it work in my code. Is there any reason why I shouldn't use code like this? I rarely use inheritance, but this feels like an appropriate use:
Python code:
class QueryMethod:
    @classmethod
    def filter(cls, *args, **kwargs):
        return session.query(cls).filter(*args, **kwargs).all()


class Person(Base, QueryMethod):
    __tablename__ = 'people'

    id = Column(Integer, primary_key=True)
    first_name = Column(String(255))
New query syntax:
Python code:
models.Person.filter(models.Person.first_name='David')
Is there a way to remove the 'models.Person' bit inside the filter method call?

The solution is to use query's filter_by function:

Python code:

class QueryMethods:
    @classmethod
    def filter_by(cls, *args, **kwargs):
        return session.query(cls).filter_by(*args, **kwargs).all()

code:

models.Person.filter_by(first_name='David')

Dominoes fucked around with this message at 16:17 on Mar 11, 2016

# ? Mar 11, 2016 14:59

Space Kablooey: May 6, 2009

You should take a look into converting into Declarative Base, if at all possible.

If you can't or won't, you shouldn't override filter_by to return a list (using .all()). Instead return the BaseQuery that the regular filter_by returns.

# ? Mar 11, 2016 17:07

Dominoes: Sep 20, 2007

HardDisk posted:

You should take a look into converting into Declarative Base, if at all possible.

If you can't or won't, you shouldn't override filter_by to return a list (using .all()). Instead return the BaseQuery that the regular filter_by returns.

AFAIK, I am using the declarative base. Why not return a list?

# ? Mar 11, 2016 18:27

Space Kablooey: May 6, 2009

Then why are you using db.session.query(Model) instead of declarative base's Model.query? It sounds it's what you want regarding SQLAlchemy's verbosity.

As for not returning the list, usually you can chain any other query clause (say a .join, other .filters and so on) as long the methods return a BaseQuery object. If you hide the .all() inside your override, you'll just create confusion down the line.

Besides, BaseQueryes can also function as a generators:

Python code:

for item in Model.query.filter_by(name="butt"):
    print item

# ? Mar 11, 2016 20:15

Adbot: ADBOT LOVES YOU

# ? May 9, 2024 19:52

Dominoes: Sep 20, 2007

The code I posted is straight from the declarative base page you linked. AttributeError: type object 'Person' has no attribute 'query', and I can't find your syntax on that page, although I've seen it mentioned before. Can you provide an example of querying from the model, without the syntactic sugar I posted?

Regarding queries as generators: Per this article, they're not lazy:

quote:

all() just evaluates the whole query and returns each row as a list. Why as a list and not as an iterator? First of all because each object returned is also immediately registered on the session. There are of course ways to bypass that, but unless you have an enormous result count you won't notice, secondly because most Python database adapters don't support streaming results anyways.

Dominoes fucked around with this message at 00:47 on Mar 12, 2016

# ? Mar 12, 2016 00:29

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »