Python information and short questions megathread.

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »

fritz: Jul 26, 2003

BigRedDot posted:

(2) As mentioned reducing algorithmic complexity is the first, best route, if you are processing lots of items.

I disagree. Always profile first. Just because you knocked an O(n^2) loop down to O(nlogn) (or whatever) doesn't mean you actually did anything worthwhile if it was only taking up 1% of the time to begin with.

# ? Jun 21, 2013 06:35

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 01:32

BigRedDot: Mar 6, 2008

I took that as implicit, and all my suggestions were suggestions for known hotspots. But yes if it needs to be stated: make the common case fast.

# ? Jun 21, 2013 19:40

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

I agree with the sentiment of "make it work, then make it fast (if necessary)", but it still makes me uneasy to leave an asymptotic runtime land mine in anything I write. I don't consider the use of an appropriate data structure to be premature optimization in most cases.

Contrived example: sure, you can get away with using a list as a FIFO queue if you know the queue won't get very big, but it's just as easy to use a collections.deque from the start. Same with using a list as a multiset; if your objects are hashable, using a collections.Counter has many benefits and no drawbacks.

# ? Jun 21, 2013 22:09

Dominoes: Sep 20, 2007

I can't recommend LFD.UCI's pythonlibs website enough for Windows users. I recommend putting it in the OP. I've spend a lot of time over the past few days troubleshooting how to install PyAudio from source. (No win64 binary on their official page; 32 only) I ran into many errors, as always. Found a precompiled 64-bit version on this website; headache over.

Dominoes fucked around with this message at 14:54 on Jun 22, 2013

# ? Jun 22, 2013 14:51

QuarkJets: Sep 8, 2008

Dominoes posted:

I can't recommend LFD.UCI's pythonlibs website enough for Windows users. I recommend putting it in the OP. I've spend a lot of time over the past few days troubleshooting how to install PyAudio from source. (No win64 binary on their official page; 32 only) I ran into many errors, as always. Found a precompiled 64-bit version on this website; headache over.

We may as well include some links to Windows executables that come with big bundles of Python packages, too, such as PythonXY

# ? Jun 22, 2013 21:29

BigRedDot: Mar 6, 2008

Dominoes posted:

I can't recommend LFD.UCI's pythonlibs website enough for Windows users. I recommend putting it in the OP. I've spend a lot of time over the past few days troubleshooting how to install PyAudio from source. (No win64 binary on their official page; 32 only) I ran into many errors, as always. Found a precompiled 64-bit version on this website; headache over.

PyAudio is also a package distributed in the Anaconda python distribution (which should also be included if we are adding distributions to the OP).

# ? Jun 22, 2013 21:54

OnceIWasAnOstrich: Jul 22, 2006

BigRedDot posted:

PyAudio is also a package distributed in the Anaconda python distribution (which should also be included if we are adding distributions to the OP).

How does the Anaconda distribution compare to the EPD/Canopy distribution? They seem pretty similar, although it looks like Anaconda is free, but you have to pay for Intel MKL versions, whereas Canopy has a limited free version or a full free version for academic licenses.

Seems like Canopy is nicer if you can get an academic license but the completely free Anaconda maybe has more libraries?

I haven't used easier because Homebrew + Pip has worked for everything I've ever tried to install, but I like the Enthought visualization libraries like Chaco.

# ? Jun 22, 2013 22:30

BigRedDot: Mar 6, 2008

I'm glad you like Chaco, I used to work on it.

I added the image and contour plots and some other features a good many years ago. Currently working on something that will hopefully be even better. I can't really comment on Canopy as I have not had any experience with it.

As for Anaconda, it does have many packages, and more are added regularly. It is also simple to create your own packages and distribute them directly, or upload them to public or private channels on binstar. The main thing I like about Anaconda is that it is segregated from the system python and you can create isolated environments with different versions of packages. I teach alot of courses so that comes in very handy. MKL is not free, on the other hand it's only 30 bucks.

However, there are also academic licenses available. [disclaimer: I work for Continuum]

# ? Jun 23, 2013 05:54

evilentity: Jun 25, 2010

I did a thing:
https://github.com/piotr-j/wf_alerts

Simple alerts about tweets with specific keywords. Windows only due to winsound. Do your worst.

# ? Jun 24, 2013 19:59

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

evilentity posted:

I did a thing:
https://github.com/piotr-j/wf_alerts

Simple alerts about tweets with specific keywords. Windows only due to winsound. Do your worst.

Neat. I just started playing Warframe this weekend, and I like python, so it's a double cool thingamabob!

# ? Jun 24, 2013 20:07

Dominoes: Sep 20, 2007

Is there a good reason the datetime module has a type named datetime? It seems confusing. Why not name the type date_time, or something else that isn't the exact name of its parent?

# ? Jun 25, 2013 02:56

good jovi: Dec 11, 2000

Dominoes posted:

Is there a good reason the datetime module has a type named datetime? It seems confusing. Why not name the type date_time, or something else that isn't the exact name of its parent?

If it were written today, I suspect it would be datetime.DateTime. A lot of the stdlib is pretty gross by modern standards.

# ? Jun 25, 2013 03:41

QuarkJets: Sep 8, 2008

Dominoes posted:

Is there a good reason the datetime module has a type named datetime? It seems confusing. Why not name the type date_time, or something else that isn't the exact name of its parent?

The datetime module is a module holding objects that deal with dates and times. It holds many classes, such as date. One of those classes, datetime, deals with specific date sand times. So datetime is a module, datetime.datetime is a class in the module

It's no different than creating a file "myclass.py" and then defining "class myclass" inside of it.

e: Actually renaming the datetime class to DateTime would break everything that uses the datetime class for what is ultimately just a style change, so there's not a lot of impetus to do that. However, you are free to set up an alias in your own code!

code:

from datetime import datetime as DateTime
>>> x = DateTime(5,4,3)
>>> x
datetime.datetime(5, 4, 3, 0, 0)
>>> x.month
4

QuarkJets fucked around with this message at 05:44 on Jun 25, 2013

# ? Jun 25, 2013 03:53

Dren: Jan 5, 2001; Pillbug

Dominoes posted:

Is there a good reason the datetime module has a type named datetime? It seems confusing. Why not name the type date_time, or something else that isn't the exact name of its parent?

The current convention would dictate datetime.DateTime but the class is older than that so we're sort of stuck with it. Other libraries maintain cruft as well. For instance the threading library was updated in 2.6 to include PEP8 style naming, however, the old names have not been deprecated.

# ? Jun 25, 2013 04:11

Dominoes: Sep 20, 2007

QuarkJets posted:

The datetime module is a module holding objects that deal with dates and times. It holds many classes, such as date. One of those classes, datetime, deals with specific date sand times. So datetime is a module, datetime.datetime is a class in the module

It's no different than creating a file "myclass.py" and then defining "class myclass" inside of it.

e: Actually renaming the datetime class to DateTime would break everything that uses the datetime class for what is ultimately just a style change, so there's not a lot of impetus to do that. However, you are free to set up an alias in your own code!
code:
from datetime import datetime as DateTime
>>> x = DateTime(5,4,3)
>>> x
datetime.datetime(5, 4, 3, 0, 0)
>>> x.month
4

Thanks for the info on Datetime naming, dudes. I'm going to try importing datetime as 'dt'. ie: 'dt.datetime.now()'.

Want to clarify before I go down a rabbit hole: I'm creating an instrument tuner using pyuaudio, numpy and scipy. I've gotten it to (repeatedly) print the current fundamental frequency from microphone input, but when I try to use it with a GUI (pyQT), the GUI is frozen while the audio analysis (FFT) is running. Is threading, using the threading module what I'm looking for?

Dominoes fucked around with this message at 01:12 on Jun 26, 2013

# ? Jun 26, 2013 01:09

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

Yes, that's the correct way to keep your GUI from freezing. Look into the Queue class for an easy way to communicate updates from your FFT thread back to your GUI thread.

# ? Jun 26, 2013 02:37

Suspicious Dish: Sep 24, 2011; 2020 is the year of linux on the desktop, bro; Fun Shoe

What you really want is a worker thread. A worker thread does process and not share any state besides a queue that you can post messages to / from. You set up the environment and then kick off the worker thread, letting it post messages back to the UI thread occasionally, like its progress with the task.

The advantage of this model is that when you start to run into GIL issues, you can easily make it a worker process instead. Additionally, parts of Qt aren't thread-safe (because parts of the underlying windowing system, like Xlib / Win32 aren't also), so you really need to have one thread that controls the UI. You also won't have any race conditions, as there's no shared state.

# ? Jun 26, 2013 02:59

QuarkJets: Sep 8, 2008

Dominoes posted:

Thanks for the info on Datetime naming, dudes. I'm going to try importing datetime as 'dt'. ie: 'dt.datetime.now()'.

Want to clarify before I go down a rabbit hole: I'm creating an instrument tuner using pyuaudio, numpy and scipy. I've gotten it to (repeatedly) print the current fundamental frequency from microphone input, but when I try to use it with a GUI (pyQT), the GUI is frozen while the audio analysis (FFT) is running. Is threading, using the threading module what I'm looking for?

Pretty much, yeah; if your GUI and your FFTs are running in the same thread, then the UI can't do anything until the FFTs are done. Python has a lot of cool concurrency tools built into it that can fix this issue for you to try, but having a dedicated thread for your UI like Suspicious Dish suggested is probably a good thing to try

# ? Jun 26, 2013 08:12

Dren: Jan 5, 2001; Pillbug

Whatever you do, be very careful if you decide to kick off a worker thread every time you click tune (or whatever your button is). If you go with an architecture like that it's very easy to introduce bugs where you can accidentally fire off multiple workers that compete for the same resource and do bad things like deadlock. I think dish was suggesting that you have a long running worker thread and communicate with it via a queue, that would be a safe way to approach the problem.

# ? Jun 26, 2013 12:36

accipter: Sep 12, 2003

Dominoes posted:

Thanks for the info on Datetime naming, dudes. I'm going to try importing datetime as 'dt'. ie: 'dt.datetime.now()'.

Want to clarify before I go down a rabbit hole: I'm creating an instrument tuner using pyuaudio, numpy and scipy. I've gotten it to (repeatedly) print the current fundamental frequency from microphone input, but when I try to use it with a GUI (pyQT), the GUI is frozen while the audio analysis (FFT) is running. Is threading, using the threading module what I'm looking for?

Just use the QThread module. It has slots and signals that will allow you to easily integrate it with the GUI.

http://srinikom.github.io/pyside-docs/PySide/QtCore/QThread.html

# ? Jun 26, 2013 14:38

Tighran: Jun 28, 2004

I decided to try out Python after doing a little bit of C++ and Java in high school many years ago. A friend pointed me towards some tutorial stuff called Python the Hard Way and though I went through those and forgot the syntax all over, I want to try it out again and make some programs of my own instead of just the usual Hello World stuff.

Basically I want to make a Spyrograph type of program based on some math stuff I did while bored at work one day. I want to display points on screen at given coordinates and draw lines connecting certain points, and probably save these to an image file. I'm assuming I will need a module to do this, but which? The wiki pointed me towards pyqtgraph and matplotlib, both of which seem to have far more stuff than I need. Is there a better, simpler module to use to start out with, or am I better off just using one of these and trying to figure out what their most bare-bones functions are?

# ? Jun 26, 2013 20:12

chippy: Aug 16, 2006; OK I DON'T GET IT

What's the best Python/Pygame version combo? I'm a bit confused as the installers on http://www.pygame.org/download.shtml (for Windows) only seem to go up to Python 3.2 instead of 3.3. Should I just use 2.7?

# ? Jun 26, 2013 20:25

QuarkJets: Sep 8, 2008

Tighran posted:

I decided to try out Python after doing a little bit of C++ and Java in high school many years ago. A friend pointed me towards some tutorial stuff called Python the Hard Way and though I went through those and forgot the syntax all over, I want to try it out again and make some programs of my own instead of just the usual Hello World stuff.

Basically I want to make a Spyrograph type of program based on some math stuff I did while bored at work one day. I want to display points on screen at given coordinates and draw lines connecting certain points, and probably save these to an image file. I'm assuming I will need a module to do this, but which? The wiki pointed me towards pyqtgraph and matplotlib, both of which seem to have far more stuff than I need. Is there a better, simpler module to use to start out with, or am I better off just using one of these and trying to figure out what their most bare-bones functions are?

Matplotlib is very easy to use if you have a simple task, it just has a lot of features under the hood. Rather than digging through documentation, though, you should try just googling for example code with queries like "matplotlib lines connect points". What you want is something like:

Python code:

import matplotlib.pyplot as plt

#Generate a list of X points as Xdata and Y points as Ydata

#Plot the data as little blue x's at each data point, connected by dashed lines
plt.plot(xdata, ydata, marker='x', linestyle='--', color='b')

#This command displays everything that has been done and pauses the execution of the rest of the script
plt.show()

# ? Jun 26, 2013 20:31

Lysidas: Jul 26, 2002; John Diefenbaker is a madman who thinks he's John Diefenbaker.; Pillbug

chippy posted:

What's the best Python/Pygame version combo? I'm a bit confused as the installers on http://www.pygame.org/download.shtml (for Windows) only seem to go up to Python 3.2 instead of 3.3. Should I just use 2.7?

As Dominoes mentioned above, LFD.UCI's pythonlibs website is fantastic for Windows Python libraries. It has 32- and 64-bit builds for 3.3, and the official Pygame site only has 32-bit builds.

# ? Jun 27, 2013 01:08

Brannock: Feb 9, 2006; by exmarx; Fallen Rib

New to Python. IDLE and especially the shell is laggy as gently caress - like to the point where I can type a lot faster than it outputs. Dragging the window around is super choppy. (Using Windows 7.) Honestly not a big deal but it's really distracting.

What's causing this? Is there a way to fix that, or should I find another IDE? Not really asking for recommendations, I was planning on exploring other IDEs on my own.

# ? Jun 27, 2013 01:24

Kumquat: Oct 8, 2010

Brannock posted:

New to Python. IDLE and especially the shell is laggy as gently caress - like to the point where I can type a lot faster than it outputs. Dragging the window around is super choppy. (Using Windows 7.) Honestly not a big deal but it's really distracting.

What's causing this? Is there a way to fix that, or should I find another IDE? Not really asking for recommendations, I was planning on exploring other IDEs on my own.

It's generally suggested that you don't use IDLE. Instead you can just use your text editor of choice and a terminal to execute code. A lot of people suggest Sublime Text 2 which is a very polished text editor with integration for different languages.

# ? Jun 27, 2013 01:39

king salmon: Oct 30, 2011; by Cowcaster

Brannock posted:

New to Python. IDLE and especially the shell is laggy as gently caress - like to the point where I can type a lot faster than it outputs. Dragging the window around is super choppy. (Using Windows 7.) Honestly not a big deal but it's really distracting.

What's causing this? Is there a way to fix that, or should I find another IDE? Not really asking for recommendations, I was planning on exploring other IDEs on my own.

I used IDLE for a long time and never noticed any lag. How much memory is it using?

# ? Jun 27, 2013 06:15

BeefofAges: Jun 5, 2004; Cry 'Havoc!', and let slip the cows of war.

IDLE can be a bit clunky, but it definitely shouldn't be that slow.

# ? Jun 28, 2013 05:50

Lexical Unit: Sep 16, 2003

I need to have some kind of ORM, I guess, that will allow me to declaratively define my schema and then output raw SQL as text without ever needing to connect to an actual database at any point. I've been using SmartSQL's SQLBuilder library but it's functionality is severely lacking. It doesn't seem to do joins based on defined relationships and doesn't support common functions like SUM(), for example.

The database I'm working with is never going to be supported by any major library but it accepts relatively standard SQL. If I'm forced to define a dialect of SQL (like with SQLAlchemy for example) then I can just say it's postgresl or mssql as it doesn't make a difference for what I'm going to be doing.

Speaking of SQLAlchemy, it seems like there's some part of it that does what I want but I'll be damned if I can figure out how the hell to get it to do so. Here's some example code, starting from a declarative style:

Python code:

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.schema import Column

from sqlalchemy.types import TIMESTAMP
from sqlalchemy.types import Integer
from sqlalchemy.types import Float

Table = declarative_base()

class UserCounts(Table):
    __tablename__ = 'user_counts'
    event_day = Column(TIMESTAMP, primary_key=True)
    event_count = Column(Integer)

class UserWeights(Table):
    __tablename__ = 'user_weights'
    event_day = Column(TIMESTAMP, primary_key=True)
    weight = Column(Float)

Ok so at this point I would want to somehow specify that if I want to query using both these tables, that they would be joined by event_day_reporting. I'm guessing I'd use a Mapper for this but I haven't really looked into that yet. What I'm stuck at is just going from the above code to generating SQL. I can't call execute() because you have to establish a session and a connection for that. I don't want a session. I don't want a connection. I just want to ask the library "Hey, let's say I wanted to query something like a sum of event_counts.event_count over some given date range... what would be the SQL you'd generate for that?" Then I can take that generated output and execute it in the oddball proprietary database I'm working with that nothing supports.

But no ORM seems to want to offer this kind of functionality. They all tightly hold on to their SQL generation capabilities which they obviously must have since there must be some code generation step before they themselves execute a query. Never exposing an interface to get at it lest the rabble get their hands on such power. :negative:

Anyone have any idea here?

# ? Jun 28, 2013 17:26

accipter: Sep 12, 2003

Lexical Unit posted:

SQL ORM

With SqlAlchemy you can turn on echo or log the session:

Python code:

import logging
logging.basicConfig()
logging.getLogger('sqlalchemy.engine').setLevel(logging.INFO)

http://docs.sqlalchemy.org/en/rel_0_7/core/engines.html#configuring-logging

# ? Jun 28, 2013 19:00

'ST: Jul 24, 2003; "The world has nothing to fear from military ambition in our Government."

Also peruse the section on implicit exucution. Since you suggest that the specific SQL syntax doesn't matter, you should be able to roll with an engine with a more standards-compliant syntax.

Modified from the documentation, something like this might work:

Python code:

engine = create_engine('sqlite:///file.db')
meta.bind = engine

print str(users_table.select().execute())

# ? Jun 28, 2013 19:14

Brannock: Feb 9, 2006; by exmarx; Fallen Rib

hal c encandenza posted:

I used IDLE for a long time and never noticed any lag. How much memory is it using?

BeefofAges posted:

IDLE can be a bit clunky, but it definitely shouldn't be that slow.

IDLE itself runs more or less fine on its own. When I open the Python shell is when I start having responsiveness problems. Both IDLE and the shell become very sluggish; closing the shell restores the responsiveness of IDLE.

IDLE uses roughly 13 MB of memory, the shell about 20 MB.

Kumquat posted:

It's generally suggested that you don't use IDLE. Instead you can just use your text editor of choice and a terminal to execute code. A lot of people suggest Sublime Text 2 which is a very polished text editor with integration for different languages.

This, however, is the best program ever :allears:

Thanks for clueing me in!

# ? Jun 28, 2013 19:27

Lexical Unit: Sep 16, 2003

accipter posted:

With SqlAlchemy you can turn on echo or log the session:

There is no session object and I will never create one. There is no engine object and I will never create one. There is no action for sqlalchemy to log as the only thing I will ever ask it to do is generate SQL text.

'ST posted:

Modified from the documentation, something like this might work:
Python code:
engine = create_engine('sqlite:///file.db')
meta.bind = engine

print str(users_table.select().execute())

This creates a file. I do not want to use SQLAlchemy to physically do anything, ever, under any circumstances. I shouldn't even need to create an engine if I can just specify the dialect I want to use when generating SQL code.

I've been browsing the (horrible) documentation some more and the section about the internal Expression Language seems promising. I've modified my code a bit and I feel like I'm closer, yet still so far away. What I have now:

Python code:

from sqlalchemy import create_engine
from sqlalchemy import Table, Column, MetaData, ForeignKey, Integer, VARCHAR, TIMESTAMP, Float
from sqlalchemy.dialects.mssql import dialect as MSSQLDialect
from sqlalchemy.sql import select

# shouldn't have to do this, should be able to use MSSQLDialect instead somehow
dummy_engine = create_engine('mssql+pyodbc://dummy')

meta = MetaData()
meta.bind = dummy_engine

user_counts = Table('user_counts', meta
    , Column('event_day', TIMESTAMP)
    , Column('event_count', Integer)
)

foo = Table('foo', meta
    , Column('event_day', TIMESTAMP)
    , Column('value', Integer)
)

user_weights = Table('user_weights', meta
    , Column('event_day', TIMESTAMP)
    , Column('weight', Float)
)

print select([user_counts.c.event_count]).where(user_counts.c.event_count > 10)

Output:

code:

SELECT user_counts.event_count 
FROM user_counts 
WHERE user_counts.event_count > ?

Ok so that's just great. But now how do I do joins? I don't want to have to manually spell out the ON clauses myself, I want to somehow tell it "if you're joining these three tables together, join them all on event_day." The documentation on Relationships is abysmally unhelpful and the aforementioned Expression Language section seems to indicate that I need to use a select_from() method with explicitly spelled out JOIN conditions. This is dumb. There must be a way to make it work using some kind of join/relationship-configuration.

Lexical Unit fucked around with this message at 19:54 on Jun 28, 2013

# ? Jun 28, 2013 19:52

Dren: Jan 5, 2001; Pillbug

You're having a lot of problems with sqlalchemy and it seems like it's not really built to do what you want. And it looks an awful lot like you're just writing SQL. Why not do that?

# ? Jun 28, 2013 21:31

Kumquat: Oct 8, 2010

Brannock posted:

This, however, is the best program ever Thanks for clueing me in!

It's absolutely fantastic. I'm not even scratching the surface of its features yet, but adding a folder to a project and opening files by hitting ctrl+p and searching for it is awesome. It's such a nice, clean interface.

# ? Jun 28, 2013 21:39

Haystack: Jan 23, 2005

For what you're doing, you don't need any engine connection. Just metadata will do.

You will in fact have to use either table.join or select_from() to do joins. The shortcuts in SQLalchemy are in the ORM, which requires at least an engine and a session. The expression language is basically just a thin layer over sql. It's main advantage is that it's easy to compose and automate.

# ? Jun 28, 2013 21:44

Lexical Unit: Sep 16, 2003

Dren posted:

You're having a lot of problems with sqlalchemy and it seems like it's not really built to do what you want. And it looks an awful lot like you're just writing SQL. Why not do that?

Because I don't want to basically re-write the entire Expression Language core of SQLAlchemy (or any other ORM for that matter). It's simply not feasible and I don't have the domain knowledge. If I was just writing some one-off SQL script then I'd do that, obviously.

Haystack posted:

For what you're doing, you don't need any engine connection. Just metadata will do.

You will in fact have to use either table.join or select_from() to do joins. The shortcuts in SQLalchemy are in the ORM, which requires at least an engine and a session. The expression language is basically just a thin layer over sql. It's main advantage is that it's easy to compose and automate.

Composition and automation are key

So what if I've got an undetermined number of tables (until runtime) with an undetermined number of columns (until runtime) and I need to construct an arbitrary combination of these based on user input. The only thing I can think of is a "join config" created on the fly based on some external configuration specification. So I'd basically end up with something like

Python code:

join_config = {
	frozenset([table1, table2]): table1.id == table2.id
	, frozenset([table1, table3]): table1.id == table3.id
	, frozenset([table2, table3]): table2.id == table3.id
	, ...
}

additional_configuration = read_from_some_other_database()

insert_addition_join_configuration(join_config, additional_configuration)

q = select()

# based on input, assume the user is requesting table1, table2, table3, foo, bar, baz, ...

for table in requested_tables:
	append_join_clause(q, join_config[frozenset([q.from_table, table])])

If I am going to have to do that, I don't really see the benefit of defining the schema within the python code in the first place.

# ? Jun 28, 2013 22:20

Haystack: Jan 23, 2005

Sorry, I don't really have enough detail about your situation to offer much advice. You say that there's a undefined amount of tables and columns, but is there any underlying structure to them? You're reflecting information about the schema at runtime: does that information include foreignkey constraints?

# ? Jun 28, 2013 23:31

Lexical Unit: Sep 16, 2003

Haystack posted:

Sorry, I don't really have enough detail about your situation to offer much advice. You say that there's a undefined amount of tables and columns, but is there any underlying structure to them? You're reflecting information about the schema at runtime: does that information include foreignkey constraints?

Unfortunately there's no foreignkey constraints in the schema, but if there were, how might that help me?

Basically there's some set of tables in a database that will be relevant to me based on user input and I have to figure out how to join them all together somehow based on some limited data I get from a document stored in a mongo database. Some of that limited data I get relates to keys and such, but other relationships are "universal" in that no matter what user input I get, you will always join them together in the same way. I've got X tables that all join together based on their event_day column but I don't know how to represent that relationship in code without an exponential explosion of code :smith:

# ? Jun 29, 2013 14:48

Adbot: ADBOT LOVES YOU

# ? Jun 11, 2024 01:32

Haystack: Jan 23, 2005

Ok, focusing on the simpler case of how to automate joining n tables:

Python code:

#lovely untested code ahoy

#Get your table metadata and then parse it into python datastructres. 
#I'm assuming the restult is a list of (tablename, key_name, key_type) tuples
raw_data = get_raw_data(whatevs)
parsed = parse_raw_data(raw_data)

meta = MetaData()

def generate_sqla_table(name, key_name, key_type, foreign_key=None):
    """Constructs a table from the metadate you've parsed"""
    if foreign_key is not None:
        table = Table(name, Column(key_name, key_type, meta, ForeignKey(foreign_key)))
    else:
        table = Table(name, Column(key_name, key_type, meta))
        
#First, you need to pick a table to be the "parent" that will be the first join target. I'm just picking the first one
parent_schema = parsed.pop()
parent = generate_sqla_table(*parent_schema)

#Then you set up the chain of relations, placing the parent at the front
key_template = "%s.%s"
key_ident = key_template" % (parent_schema[0], parent_schema[1])
children = []
for schema in parsed:
    #Creates the chain of relations
    children.append(generate_sqla_table(*schema, foreign_key=key_ident))
    key_ident = key_template % (schema[0], schema[1])
    
#Do your big clusterfuck chain join 
joined = parent.join(children.pop())
for child in children:
    joined = joined.join(child)

#Then build your select
s = select(['*']).select_from(joined) #SQL for selecting all columns from the joined tables

_{(This hinges on SQLalchemy being able to function without a full table definition, which I haven't tested. You should.)}

Obviously things get more complex if you're joining on multiple categories of keys, or want to filter your results. You'd probably have to build up (and then flatten) some sort of graph of relations and associated filters. Regardless, the underlying process stays pretty much the same: build your SQLalchemy tables, build up a chain of joins and filters, and then construct your select statement.

# ? Jun 29, 2013 22:27

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python information and short questions megathread.

«‹›484 »