Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
mysom
Aug 10, 2005
I have a question about possible approaches for a project I'm in the process of sketching out and I was hoping someone might have some experience with the "proper" way to deal with dynamic models.

At the crux of my issue are dynamic models, my application will be using GeoDjango and I would like to have the users be able to upload and work with their own data. For the most part I would expect these data to be in the form of shapefiles, which if you are not familiar with them are a file that associates a type of geometry (point, line, polygon, or grouping of the shapes) with some additional values. As such each shapefile should naturally be its own table in the database and have its own model in the application.

The issue is there is no way to know the structure ahead of time and the only two approaches I then see are create some series of incredibly kludgy uber tables that can accommodate any one type of geometry and then secondary tables to define column labels which just seems god awful.

Or alternatively, there does seem to be some hacky approaches to define dynamic models at runtime, but this seems to open a whole can of worms of tracking the new models and maintaining them over time and based on the snippets I've seen their may also be issues with newer versions of Django.

Any advice or experience people have with this kind of problem would be greatly appreciated.

Adbot
ADBOT LOVES YOU

LuckySevens
Feb 16, 2004

fear not failure, fear only the limitations of our dreams

I'm just wondering how you guys think I should about building something like this:

I want to build something that attaches a footer to my blogposts that will contain a table of numbers (numbers from come stock data on yahoo finance or wherever). I'll have a check box in the admin of adding a blog post that will enable this information to be on the bottom. The numbers will be pulled using a scrapper, I might build a manager to do this, then pass the info to to my models which will input into the database. I'll then pass the model into my views and use the template system to display the footer using an if tag, if I've checked the option in the blog post.

Is this the best way to do this, or should I do it differently for some reason? I don't mind hardcoding the formatting of the table into my templates, but there might be a good reason not to do it that way.

king_kilr
May 25, 2007
Just use a template inclusion tag.

LuckySevens
Feb 16, 2004

fear not failure, fear only the limitations of our dreams

cool little feature, looks perfect, thanks

entr0py
Jan 31, 2004


A real-life robot.
I've recently started looking at handling application notifications better and came across the excellent django-tagging. There are a couple of use cases that I am exercising that don't quite fit the observer pattern implemented. While django-notification allows for observations on objects by way of signal dispatching, I'd like to also do that at the model level and expand upon the possibilities.

To use a common example, it's easily possible to observe updates to a blog post by hooking the post_save signal and using the django-notification API (handle_observations) to find object observations for that signal and sender. What if I want to observe object creation for various models themselves (ie: notify any time a new blog post is made)? And what would be the best way expanding on the observation criteria for model-specific fields (ie: observe blog posts with specific tags). For the former, unless I misunderstand something, I don't think this fits with the observer pattern in django-notification and I'd be forced to fork or monkey patch the API. For the latter, I think that could be accomplished by subclassing ObservedItem for model-specific observation needs, but I feel like there might be a more generic solution that I can't quite grasp.

Any ideas?

Brand New Otter
Nov 28, 2009

entr0py posted:

What if I want to observe object creation for various models

observe blog posts with specific tags

Use post_save. In the second case, filter the objects in the handler.

entr0py
Jan 31, 2004


A real-life robot.

Brand New Otter posted:

Use post_save. In the second case, filter the objects in the handler.

I think you misunderstood my intention. The idea is the create ObservedItem objects that indicate a user observing a given object instance and signal. By then invoking that handle_notifications function you can automatically send notifications based on whatever observations your users have defined. The handle_notifications function is the signal handler that you use and it finds observers for the object instance and the signal that was raised.

However, I think this problem can be solved by simply making the GFK on the ObservedItem simply point to the ContentType object itself, though that seems a bit hackish. As well, subclassing ObservedItem and using table inheritance (with a little monkey patching to django-notification) for more model-specific observation rules worked perfectly.

Captain Briney
Jan 5, 2009

This is my life partner and crappy upholstery.
Is there a really great tutorial on how to design themes easily? Some of the Django documentation scares me.

Captain Briney fucked around with this message at 09:52 on Feb 22, 2010

sink
Sep 10, 2005

gerby gerb gerb in my mouf
I'm not sure what you mean. You might be confusing Django with a Content Management System.

Django is a web application framework and doesn't really have its own notion of themes.

Do you mean templates?

Captain Briney
Jan 5, 2009

This is my life partner and crappy upholstery.
Yes, that is what I meant. I got my terminology mixed up there. Thanks.

SlightlyMadman
Jan 14, 2005

Do you know html? That's about 99% of template design.

Jonnty
Aug 2, 2007

The enemy has become a flaming star!

Captain Briney posted:

Yes, that is what I meant. I got my terminology mixed up there. Thanks.

What's confusing you?

It's pretty easy to be honest: use {{variable.something}} for accessing variables passed into the template. You can use that syntax for no-arg methods, lists and dictionaries too:

code:
{{variable.doSomething}} 
{{list.0}}
{{dictionary.key}}
Then there's the {% templatetag %} syntax, and filters which do stuff to the output of those tags, which are used like this: {% tag|filter %}.

I could go on, but I'd just be rehashing the django docs, which really aren't that hard. Sure you're not looking at something about the architecture of the templating system or something like that?

ErIog
Jul 11, 2001

:nsacloud:
I have a question. I'm creating a simple UI for an internal database app that I'm prototyping. This thing has to be really efficient from a UI standpoint because users are going to have to be able to manipulate it rapidly. This means that I'd like to prefill as much information as possible on the admin forms.

I discovered that you could prefill on the forms by doing http://site/admin/app/model/add/?field=whatever

This has given me 90% of what I want except that it doesn't seem to work on the change form, http://site/admin/app/model/objectid.

Does anyone have an easy solution to this or is there something simple I'm missing?

I really don't want to start overwriting or rewriting stuff in the admin because this thing is supposed to be a prototype, and if I rewrite the admin then I don't want to do it while I'm in "hack toward functionality" mode because that's going to make my project a lot less clean. I'm also planning to gut this thing for a proper rewrite after this thing passes early-alpha functionality tests. It's my first Django project, my first web app, and my first time using anything related to Javascript. It's a god drat mess.

ErIog fucked around with this message at 00:02 on Mar 19, 2010

nbv4
Aug 21, 2002

by Duchess Gummybuns
it doesn't really make sense as to why you'd want to do that. When you're editing objects, you want the page to reflect the current state of the object in the database.

SlightlyMadman
Jan 14, 2005

You've probably already considered this since it's so obvious, but just for the sake of stating it have you tried simply setting default values in the model?

nbv4
Aug 21, 2002

by Duchess Gummybuns

SlightlyMadman posted:

You've probably already considered this since it's so obvious, but just for the sake of stating it have you tried simply setting default values in the model?

those aren't dynamic

ErIog
Jul 11, 2001

:nsacloud:

nbv4 posted:

it doesn't really make sense as to why you'd want to do that. When you're editing objects, you want the page to reflect the current state of the object in the database.

It makes perfect sense. My app involves adding metadata to scans of magazine pages. So I've constructed a view that displays a page, and then gives links along the side to define the elements on the page. It also lists all the elements currently assigned to the page.

For magazine articles that span multiple pages, it uses a ManytoMany field that goes to the pages table. The current efficiency bottleneck is that there's no way to know which pages the article is going to be on when you first define the article in the system because it's going to be intermingled with full page ads. So you need to be able to add the page you're currently looking at to an article already defined for that issue within the system.

So my app will display all current elements associated with the page, and then all the articles in the issue under that with a little plus sign that will add the current page to that article.

Also, even without knowing what this is for, wanting to do this makes total sense. When you load the change form, you're right you do want it to reflect the current state of the object. However, you're loading the change form specifically so you can modify it. It makes sense that you might want to pre-fill modifications in order to automate as much of the data entry as possible as in my example here.

I think I may be overthinking this. Maybe I just need to make my little button submit an HTTP post request that is then handled in the views.py object that updates the specified object with the information I want. It's just that I've never used HTTP post requests before, and so I wanted to make sure I exhausted all the easy options before teaching myself something new.

ErIog fucked around with this message at 02:08 on Mar 19, 2010

Lamacq
Jun 15, 2001

Breezeblock RIP

ErIog posted:

I think I may be overthinking this. Maybe I just need to make my little button submit an HTTP post request that is then handled in the views.py object that updates the specified object with the information I want. It's just that I've never used HTTP post requests before, and so I wanted to make sure I exhausted all the easy options before teaching myself something new.

I think you are overthinking it, and/or you've reached the limit of what the admin site can do for you. The admin site is made of 100% pure awesome, but once you get beyond basic CRUD operations you're better off (in my experience) just coding it yourself.

That said, maybe you could do what you need by overriding the model change form template and adding some javascript to set the values after the page is loaded?

ErIog
Jul 11, 2001

:nsacloud:

Lamacq posted:

I think you are overthinking it, and/or you've reached the limit of what the admin site can do for you. The admin site is made of 100% pure awesome, but once you get beyond basic CRUD operations you're better off (in my experience) just coding it yourself.

That said, maybe you could do what you need by overriding the model change form template and adding some javascript to set the values after the page is loaded?

Just taught myself GET and POST requests in Django through JQuery. loving christ. This is like the 10th time I've wanted to do something in Django where I've spent about an hour combing the internet trying to implement something based on what I already know instead of just loving learning the new thing which only takes 5 minutes because Django and JQuery make it so god drat easy. I don't have the functionality in there yet, but my hack case of sending bullshit data over POST, catching it in my view, printing it to the console, and putting up a JS success alert with data from my view works great. Now all I need to do is put the template hooks in and write the simple logic in views.py.

I agree with you about the admin interface, by the way. I was going to post a question about it here, but I kept thinking "Surely, this can't be the way it is. I must have something wrong." Turns out I was right.

The big problem with the admin interface is that there's so much hidden magic going on with it behind the scenes that it becomes this impenetrable wall of finding out how to override it to get it to do what you want. They could have made all this junk a lot simpler by simply copying the admin site and logic into your project directory when you initialize your project. Then you could modify the admin code all you like without feeling like you're breaking a core rule of using Django which is to extend it through supported means rather than go off into uncharted territory rewriting it, thus making your projects not at all portable.

I find it to be clever in a way that borders on un-pythonic. I don't understand how the Django developers could feel alright about having something in their framework that is basically, "You loving do it our way, and deal with it or you could start from scratch..."

Also, I would advise anyone trying to save time by downloading the VirtualBox Bitnami Django stack to not do so. Just roll your own VirtualBox installation in the Linux of your choice. Bitnami puts all the packages in /opt/ which means you have to learn their wacky directory structure in order to know where your httpd conf files are stored.

I should note that this post probably comes off a lot angrier than I actually am. I'm not actually angry. It's just minor annoyances here and there that have stacked up from working on this thing full time for the past week. As I get more used to Django everything will be hunky dorey. Django makes lots of things so easy that when you start having little problems they seem a lot bigger than if everything was a giant slog.

ErIog fucked around with this message at 02:45 on Mar 19, 2010

Captain Capacitor
Jan 21, 2008

The code you say?

ErIog posted:

Just taught myself GET and POST requests in Django through JQuery. loving christ. This is like the 10th time I've wanted to do something in Django where I've spent about an hour combing the internet trying to implement something based on what I already know instead of just loving learning the new thing which only takes 5 minutes because Django and JQuery make it so god drat easy. I don't have the functionality in there yet, but my hack case of sending bullshit data over POST, catching it in my view, printing it to the console, and putting up a JS success alert with data from my view works great. Now all I need to do is put the template hooks in and write the simple logic in views.py.

I agree with you about the admin interface, by the way. I was going to post a question about it here, but I kept thinking "Surely, this can't be the way it is. I must have something wrong." Turns out I was right.

The big problem with the admin interface is that there's so much hidden magic going on with it behind the scenes that it becomes this impenetrable wall of finding out how to override it to get it to do what you want. They could have made all this junk a lot simpler by simply copying the admin site and logic into your project directory when you initialize your project. Then you could modify the admin code all you like without feeling like you're breaking a core rule of using Django which is to extend it through supported means rather than go off into uncharted territory rewriting it, thus making your projects not at all portable.

I find it to be clever in a way that borders on un-pythonic. I don't understand how the Django developers could feel alright about having something in their framework that is basically, "You loving do it our way, and deal with it or you could start from scratch..."

Also, I would advise anyone trying to save time by downloading the VirtualBox Bitnami Django stack to not do so. Just roll your own VirtualBox installation in the Linux of your choice. Bitnami puts all the packages in /opt/ which means you have to learn their wacky directory structure in order to know where your httpd conf files are stored.

I should note that this post probably comes off a lot angrier than I actually am. I'm not actually angry. It's just minor annoyances here and there that have stacked up from working on this thing full time for the past week. As I get more used to Django everything will be hunky dorey. Django makes lots of things so easy that when you start having little problems they seem a lot bigger than if everything was a giant slog.

Just looking through your problem I thought of a few things that might help you out. For the pre-filling of forms, I would suggest the ModelForm (at least I think it's called that) system, a form that you pass an instance and it pre-fills everything out, just like when you edit an object. The second would be Admin actions, which give you a way of doing some sort of custom operations just by receiving a few selected objects from the admin panel.

Hanpan
Dec 5, 2004

I can't for the life of me figure this out.

I'm trying to get the results of a ManyToMany field, so I can avoid doing this:

code:
{% for tag in article.tags.all %}
Which results in an extra query to the database. What I was hoping for was some kind of join query to help optimize my query.

I've been told that you simply can't do this with Django, short of using select_related() which doesn't really cater for what I want. My only other option is to use raw SQL but I'm not too keen on that because I'd have to manually enter the table names and such which is never good.

I was linked to this: http://github.com/lilspikey/django-batch-select/ which seems to do exactly what I want, but I'm struggling to accept that the only way to achieve this is with an addon.

Does anyone have any better suggestions?

Jonnty
Aug 2, 2007

The enemy has become a flaming star!

Hanpan posted:

I can't for the life of me figure this out.

I'm trying to get the results of a ManyToMany field, so I can avoid doing this:

code:
{% for tag in article.tags.all %}
Which results in an extra query to the database. What I was hoping for was some kind of join query to help optimize my query.

I've been told that you simply can't do this with Django, short of using select_related() which doesn't really cater for what I want. My only other option is to use raw SQL but I'm not too keen on that because I'd have to manually enter the table names and such which is never good.

I was linked to this: http://github.com/lilspikey/django-batch-select/ which seems to do exactly what I want, but I'm struggling to accept that the only way to achieve this is with an addon.

Does anyone have any better suggestions?

It sounds like select_related is sort of what you want, why don't you want to use it? I remember something about it being dodgy with M2M fields though. I've come to terms with the fact that if you want any sort of db efficiency in django, it's going to be messy, be that adding an extra call to select_related here and there or going to the lengths of installing an addon. Of course, I might be missing some easy way of doing this, so do wait a bit.

Hanpan
Dec 5, 2004

Jonnty posted:

It sounds like select_related is sort of what you want, why don't you want to use it? I remember something about it being dodgy with M2M fields though.

I'm pretty sure it doesn't work with M2M any at all, or at least it isn't for me. It's picking up my ForeignKey Field ok, but that's about it :smith:

Edit: Also, if I was to use select_related on a ForeignKey to my auth table, it will return the users password as part of the query set. Could this be considered a security risk or should I not worry about it?

Hanpan fucked around with this message at 10:54 on Mar 19, 2010

nbv4
Aug 21, 2002

by Duchess Gummybuns

Hanpan posted:

I'm pretty sure it doesn't work with M2M any at all, or at least it isn't for me. It's picking up my ForeignKey Field ok, but that's about it :smith:

Edit: Also, if I was to use select_related on a ForeignKey to my auth table, it will return the users password as part of the query set. Could this be considered a security risk or should I not worry about it?

when i first started using django, I used to think select_related was a big inside joke because no matter what I did, it never seemed to change anything. Then I noticed this part in the docs:

quote:

Note that, by default, select_related() does not follow foreign keys that have null=True.

then I changed all my models to be null=False and now all my queries are nice and efficient :c00l:

edit: also, don't worry about it returning the passwords, they are hashed anyways

Jonnty
Aug 2, 2007

The enemy has become a flaming star!

Hanpan posted:

I'm pretty sure it doesn't work with M2M any at all, or at least it isn't for me. It's picking up my ForeignKey Field ok, but that's about it :smith:

Edit: Also, if I was to use select_related on a ForeignKey to my auth table, it will return the users password as part of the query set. Could this be considered a security risk or should I not worry about it?

It shouldn't. If you're worried, though, there's docs somewhere on limiting which fields are actually pulled in.

e: near the bottom of their section here http://docs.djangoproject.com/en/dev/ref/models/querysets/#id4

Hanpan
Dec 5, 2004

nbv4 posted:

then I changed all my models to be null=False and now all my queries are nice and efficient :c00l:

Thanks for the tip, I tried it but still no luck. I guess it's because its a M2M as opposed to a ForeignKey.

I don't mind resorting to custom SQL... if only there was a way to inject the table names

tankadillo
Aug 15, 2006

I'm running a simple site with Django and I'm having a problem. The MySQL database is frequently (sometimes several times a day) exceeding its max user connections. I've looked over the logs and this only has ever happened when the user agent was a web spider. Google analytics reports that the site get about 10 real visitors a day, with occasional spikes of several hundred. This leads me to believe that the bots are crawling too fast or something, since my site can seem to handle regular traffic fine. I've looked over the Django code (which I wrote a year ago) and I don't see any especially inefficient parts, at least within my knowledge of the ORM, and I have indexes on my database tables. There's also not one particular Django view or line of ORM code that seems to be more or less likely to choke the database. The site is pretty simple, just a blog format with an Entry model and Django's comments framework.

Is there a magically easy way to find and fix the problem here? Or do I just need to do some more in-depth debugging. I only checked the error log very quickly, there could be more to it that I didn't notice yet.

tankadillo fucked around with this message at 19:59 on Mar 19, 2010

Yay
Aug 4, 2007
Seems largely irrelevant to Django, unless the ORM is creating some ridiculously grungey queries.

Mostly, its the kind of thing you want to handle at the server/proxy/firewall level, rather than the application server (wsgi, fcgi, whatever) level.

Put rules in your robots.txt and see if any of the bots abide them. If they don't, figure out the IP ranges of badly behaved robots and chuck them in a block rule in iptables. If they change IP, throw some rules in your apache to block the user agents. Maybe also appropriate to use mod_security, I've not used it extensively enough. Welcome to the endless battle.

Beyond that, you could probably aggressively cache querysets, templates, all that jazz, and serve up apropos http caching headers. Might help alleviate the problem, but its not an answer per se.

(Actual edit: its also worth noting that some spiders (google, yahoo) provide facilities for crawl delays. Yahoo through robots.txt rules, and Google through their webmaster tools)

(Actual edit part deux: If you're really only getting about 10 visitors a day, I'd think you'd rarely need to actually hit the database, and could just use the inbuilt caching and a non-mySQL backend)

Yay fucked around with this message at 20:16 on Mar 19, 2010

ErIog
Jul 11, 2001

:nsacloud:
Okay so I have my JQuery POST request working, but I can't figure out how to update a ManytoMany field. The Django docs explain how to do a plain field just fine where you do:

code:
object = Model.objects.get(id)
object.field += 1
object.save()
How do I do this for a ManytoMany field? It doesn't seem to work the same way. I read a workaround for just deleting the object, and then recreating it with the new properties but that seems kludgey.

Captain Lou
Jun 18, 2004

buenas tardes amigo

Jonnty posted:

It sounds like select_related is sort of what you want, why don't you want to use it? I remember something about it being dodgy with M2M fields though. I've come to terms with the fact that if you want any sort of db efficiency in django, it's going to be messy, be that adding an extra call to select_related here and there or going to the lengths of installing an addon. Of course, I might be missing some easy way of doing this, so do wait a bit.
AFAIK select_related only works on forward relationships, where it's used on a model that contains foreignkeys.

nbv4
Aug 21, 2002

by Duchess Gummybuns

ErIog posted:

Okay so I have my JQuery POST request working, but I can't figure out how to update a ManytoMany field. The Django docs explain how to do a plain field just fine where you do:

code:
object = Model.objects.get(id)
object.field += 1
object.save()
How do I do this for a ManytoMany field? It doesn't seem to work the same way. I read a workaround for just deleting the object, and then recreating it with the new properties but that seems kludgey.

if it's a many2many field, it won't just be an object, it'll be a list of objects (because there are many of them). To get those objects do object.m2m_set.all() or object.m2m.all() (depending on where you defined the relationship), which will return a queryset of all your many2many objects which you can then iterate over and edit/save each one individually. If you want to add an object to the many2many list, you do this: object.m2m.add(obj)

Lamacq
Jun 15, 2001

Breezeblock RIP

ErIog posted:

Okay so I have my JQuery POST request working, but I can't figure out how to update a ManytoMany field. The Django docs explain how to do a plain field just fine where you do:

code:
object = Model.objects.get(id)
object.field += 1
object.save()
How do I do this for a ManytoMany field? It doesn't seem to work the same way. I read a workaround for just deleting the object, and then recreating it with the new properties but that seems kludgey.

code:
relatedObject = RelatedModel.objects.get(pk=1)
object.relatedmodel_set.add(relatedObject)
object.save()
See http://docs.djangoproject.com/en/dev/ref/models/relations/#ref-models-relations

ATLbeer
Sep 26, 2004
Über nerd
So.. The OP is WAY out of date by now and I was going to restart the thread with a brand new OP but, thought about all going web 2.0 twitter feedy with it..

I figured I should transfer the OP to a blog and completely rewrote it for Django >1.0

Here's what I have now: http://degizmo.com/series-django-tutorials/

Suggestions? Comments? Criticisms? Needs more Ponies?

Mulozon Empuri
Jan 23, 2006

ATLbeer posted:

So.. The OP is WAY out of date by now and I was going to restart the thread with a brand new OP but, thought about all going web 2.0 twitter feedy with it..

I figured I should transfer the OP to a blog and completely rewrote it for Django >1.0

Here's what I have now: http://degizmo.com/series-django-tutorials/

Suggestions? Comments? Criticisms? Needs more Ponies?

Tests? And perhaps something about fancy stuff like virtualenv?

ATLbeer
Sep 26, 2004
Über nerd

Mulozon Empuri posted:

Tests? And perhaps something about fancy stuff like virtualenv?

Virtualenv is probably out of scope in a basic tutorial but, tests... drat right, adding it to my to-do list

deimos
Nov 30, 2006

Forget it man this bat is whack, it's got poobrain!

ATLbeer posted:

Virtualenv is probably out of scope in a basic tutorial but, tests... drat right, adding it to my to-do list

A good overview to a decent directory organization skeleton would be awesome so people get used to a sane directory structure. Something like this (which I think is based off of ericflo's work, but if it's his I can't find it anywhere else).

king_kilr
May 25, 2007

deimos posted:

A good overview to a decent directory organization skeleton would be awesome so people get used to a sane directory structure. Something like this (which I think is based off of ericflo's work, but if it's his I can't find it anywhere else).

ericflo had a public skeleton project, but deleted it after he realized he wasn't going to maintain it at all

Profane Obituary!
May 19, 2009

This Motherfucker is Dead
I tend to use lincoln loops' start project, but it does do things a little bit differently (particularly the settings).

http://github.com/lincolnloop/django-startproject

No Safe Word
Feb 26, 2005

Yeah buddy, theonion.com is now Django-powered according to Adrian's latest tweet: http://twitter.com/adrianholovaty/status/10930210800

Adrian Holovaty posted:

This is easily the most proud and excited I've ever been about a Django development: theonion.com is now Django-powered.

Adbot
ADBOT LOVES YOU

king_kilr
May 25, 2007

No Safe Word posted:

Yeah buddy, theonion.com is now Django-powered according to Adrian's latest tweet: http://twitter.com/adrianholovaty/status/10930210800

There's a better source than adrian: http://twitter.com/TheOnion/status/10921296161 :) If anyone has any questions about it I can probably answer them, I interned there last summer

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply