Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
As far as not having to blow away your DB when you are playing around, you could use something like South. I find myself using the "iteratively working on a migration" workflow a lot when I'm playing around with things.

You don't have to use that though, you could simply just make a little python script that creates all your data, so after you blow away your database you can just run your little "generate some data" script.

Adbot
ADBOT LOVES YOU

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
This logging thing is driving me mad. I don't even want to actually log SQL queries, I just want to know why it's not working.

Added my logger:
code:
LOGGING = {
...
    'loggers': {
        'django.db.backends': {
            'handlers': ['file'],
            'level': 'DEBUG'
        }
    }
}
Now pop into the python shell
code:
$ ./manage.py shell
>>> import logging
>>> logging.getLogger('django.db.backends').handlers[0].baseFilename
'/var/log/django/www.myawesomesite.com.log'
Ok, so it's got my file handler, try writing to it...
code:
>>> logging.getLogger('django.db.backends').debug('hello world')
That works fine - I'm tailing the log and I saw that entry get added.

When I hit my Django app through the webserver, however, nothing from django.db.backends gets written to the logs. If I change DEBUG=False to DEBUG=True in my settings, lots of stuff from django.db.backends gets written to the logs. I can't find anything that would suggest the logging is tied to that DEBUG setting though.

edit: ughhhh how did I miss this:

quote:

For performance reasons, SQL logging is only enabled when settings.DEBUG is set to True, regardless of the logging level or handlers that are installed.

fletcher fucked around with this message at 02:43 on Dec 19, 2013

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
code:
>>> MyAwesomeObject.objects.filter(some_field='').count()
1673
>>> change_some_data()
>>> MyAwesomeObject.objects.filter(some_field='').count()
1673
How come the count never changes until I disconnect from the shell and fire up a new one? When I run the COUNT(*) queries manually before and after change_some_data() the counts are different as expected, but not when I do it in the manage.py shell like above.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Dominoes posted:

Anyone know how to run South? I did 'from south.db import db', and was greeted with "ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings."

I tried south.settings.configure() and django.settings.confgure(), but neither are valid commands.

I've been using South but I've never had to manually write an import statement like that, all I ever do is:

code:
$ ./manage.py schemamigration my_app --auto
$ ./manage.py migrate my_app
What is it that you are trying to do?

Also, you can specify that environment variable at the top of your script (none of my migrations actually do this though):
code:
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'my_project.settings'
Did you follow the tutorial? http://south.readthedocs.org/en/latest/tutorial/part1.html

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Dominoes posted:

I was going off this Stack overflow thread

I'm trying to rename a column. I don't understand why Databases (and therefore Django models) are so finicky about changing things, but apparently South is the solution.

I want to run this: 'db.rename_column('app', 'old', 'new')', which requires importing db. I don't need to migrate anything, I just want to be able to change database columns without starting from scratch each time.

I see, you want a data migration then.

First create an empty data migration:
code:
(myapp)greg@greg-mint15 ~/code/my_app $ ./manage.py datamigration myapp rename_thing1_to_thing2
Created 0018_rename_thing1_to_thing2.py.
Now open up the ####_rename_thing1_to_thing2.py file it created and add your db.rename_column stuff from that stackoverflow post.

Then run the migration:
code:
$ ./manage.py migrate myapp

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Cool! Glad you got it working.

My turn for a question - Is there a better way I should be doing something like this? I want it to handle the case where last_id might not be specified, or if it is, making sure it's the right data type.

code:
try:
    filter_criteria['id__gt'] = int(request.GET.get('last_id'))
except (TypeError, ValueError):
    pass

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Megaman posted:

Any ideas?

What are you using to transfer the files from your django app server to S3?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Megaman posted:

Files are uploaded to the actual servers, then uploaded to s3. This seems like a step too many? Do people usually use forms to have users directly upload to s3 to get around using the servers? Then the issue is uuid/keys of said files.

That sounds fine, your servers probably have to be the proxy to s3 to avoid cross domain security stuff. But what are you using to communicate with S3? If you eliminate the S3 step and just save them to your servers, does the upload still timeout?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Megaman posted:

Nope, the timeout happens between the server and s3. We also can't store on the server, we have tons of data, thus s3. I was thinking the client could upload from our app directly to s3, wouldn't that be a valid solution to this? We could just cut out the server.

Wasn't suggesting it as a long term solution, just as a way to debug and isolate the issue.

So for the third time...what are you using to transfer the files to S3?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Megaman posted:

Boto, but then again you're question is insanely vague so Boto, Python, S3, Django, Apache. One of those should answer your question however you meant to ask it.

Yeah but just because Apache and Django are in your stack doesn't mean they are the components that send the files to s3. You also listed s3 in there...you're using s3 to send files to s3? That doesn't make any sense.

Anyways, are you able to reproduce the issue with a tiny little python script that just uses Boto to send a file to s3? It looks like others have this issue too. The issue is for aws-cli but it uses boto under the covers.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
I've got a method on one of my models:

code:
def do_the_thing(self):
    if self.some_counter_field < 5:
        self.some_counter_field += 1
        self.save()
        return True
    else:
        return False
The only place I call do_the_thing is in one of my templates. Worked exactly as intended on my local development server. Pushed it to production and I noticed that when I hit the Django Admin page for an instance of this particular model, it's incrementing some_counter_field. Tried the same thing again on my local dev server and wasn't able to reproduce. Any idea what might be happening here?

fletcher fucked around with this message at 03:30 on Feb 25, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

MonkeyMaker posted:

fletcher: I'm not sure I understand. Your code increments the counter and, in production, it's incrementing the counter. How is that not working correctly? Nevermind, I'm stupid.

So that's just a method on the model? Nothing in the model or model admin calls that method?

Yup that is correct. I have no idea why the admin interface would be calling that method, I am certain the only caller of the method is a single spot in one of my templates. Even if it was using some sort of serializer, it wouldn't invoke my custom methods on models right? It's got me really stumped.

The field is defined in my model as:
code:
some_counter_field = models.IntegerField(blank=True, default=0)

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
I switched to DEBUG = True in the production clone environment, and viewing the admin page no longer increments that value. So I thought great, I can set DEBUG = False on my local dev server to debug...but I still can't reproduce it locally. :(

edit: had the 2nd true/false backwards

fletcher fucked around with this message at 19:31 on Feb 25, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

MonkeyMaker posted:

Wait, switching on DEBUG changed the way it acts?

Can you show the entire model (at least the methods of the model) and the model's custom admin if it has one?

Yup, that appears to be the case, as strange as it may be. Unfortunately, I can't share the whole model and methods. Your question about the custom admin has got me thinking though - this one is an MPTT model, and that does custom admin stuff for rendering the hierarchy picklist options. I wonder if that's the culprit. Don't see any reference to the DEBUG setting in there, so doesn't really explain that behavior, but I'm gonna play around with it some more.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Pollyanna posted:

I'm at the point where I want to handle editing and deleting object models from the user side. I get how to create and read objects from the back-end, but is there a Django-ish way to edit and delete things from the front-end?

Also, I'm deploying on Heroku, and I keep running into problems with the database. I use SQLite on my end, but Heroku needs PostgreSQL. I added the Heroku database and wsgi settings, and it works fine on Heroku, but running it locally with Foreman gives me this issue:

code:
ImproperlyConfigured at /tracker/
settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.
when I try and check if a user is authenticated. Why does this happen? It works online, so why doesn't it work locally?

Plus, I thought Heroku was supposed to handle the migration from SQLite to PostgreSQL itself, so I shouldn't have to change settings myself?

Your CRUD operations (create, read, update, delete) always happen on the back-end. The only way for somebody to do it through the front should be via the back-end (:butt:). The Django way of doing this is through forms.

It sounds like you may need two settings.py files, one that heroku will use, and one that you will use locally. My settings.py is all setup for the server to use, but at the bottom of it I have:
code:
try:
    from settings_local import *
except ImportError, e:
    pass
Then locally I create a settings_local.py file which contains:
code:
DEBUG = True

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3'
    }
}
That allows me to override some of the things I defined in settings.py.

I do not check in settings_local.py to source control or deploy it anywhere, it only needs to be used on my local machine.

fletcher fucked around with this message at 22:44 on Feb 25, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

NtotheTC posted:

If I may sing the praises of Two Scoops of Django for a moment, they have a very good section on why you should avoid the settings_local.py anti pattern. (the short version is: unchecked-in code means it can be lost, also means all team members will have different settings_local.py files which leads to "It doesn't do that for me" errors that take hours to debug.)

Instead, have a settings folder in your project that contains all your different settings_*.py files (i.e. production, testing, development) and check them all in. To avoid checking in sensitive information like SECRET_KEY, leave it out of settings_production.py and instead pass it in via an environment variable on the server. In your settings_development.py you can have a throwaway SECRET_KEY variable that you do not use in production just to stop you having to faff around with it.

This won't really affect you when you're working alone but it's a good habit to get into.

This seems like a great idea, but I guess I should mention the other piece of my puzzle: Chef. I already have all my environments defined in Chef (prod, testing, dev, etc), and they all use the same settings_local.py.erb template file. So while my settings_local.py doesn't come from the same repo as my django app, it does come from a central repo and all devs would have the same copy of it. Not saying this is a great setup, but that's why the settings_local.py pattern appealed to me in the first place.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Currently I'm running collectstatic on each server the app is deployed on, and django-pipeline handles the less & uglifyjs and all that stuff. It can be kinda time consuming though, I was thinking I should do collectstatic on just 1 server, package it up, and then deploy the packaged version. How should I go about doing that?

fletcher fucked around with this message at 19:29 on Mar 11, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
I wrote a python script for my Django app that is intended to be scheduled as a cron job. What folder should this file live in? Or should it be a Custom management command?

edit: and a followup question since I think a custom admin command is the right answer:

My admin command uses self.stdout.write, but it calls other methods that use logger.info('butts'). Can my custom admin command adjust the logger config so those messages are included if they are called by the admin command?

fletcher fucked around with this message at 02:38 on Apr 1, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Awesome, thanks for the tip!

Is there any way to get the test database to stick around after a failed run? I wanna poke around in the data to diagnose why it failed.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
From the Model.clean docs:

quote:

This method should be used to provide custom model validation, and to modify attributes on your model if desired. For instance, you could use it to automatically provide a value for a field

So I've got:

Python code:
class Person(models.Model):
    first_name = models.CharField(max_length=100)
    last_name = models.CharField(max_length=100)
    full_name = models.CharField(max_length=201)

    def clean(self):
        self.full_name = self.first_name + ' ' + self.last_name
Then in the admin console:
code:
>>> joe_schmoe = Person.objects.get(full_name='Joe Schmoe')
>>> joe_schmoe.first_name = 'Joseph'
>>> joe_schmoe.save()
>>> Person.objects.get(full_name='Joseph Schmoe')
DoesNotExist: Person matching query does not exist.
What's the point of using a Model.clean() override to set full_name if Model.save() doesn't invoke it?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Ahz posted:

Does anyone know the best go-to library or whatever to generate custom QR codes?

I'm looking to generate and then store the GIF or whatever they create as a binary in postgres.

Out of curiosity, what do you want to use the QR codes for? I only ask because I've never seen somebody point their phone at one before.

PNG (or even SVG?) would be a much better format for QR codes than GIF I would think.

There seems to be no shortage of libraries for QR codes on PyPi:

https://pypi.python.org/pypi/qrcode
https://pypi.python.org/pypi/django-qrcode

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
What does your logging config look like in Django?

I have this in mine and it sends me an email with the full stack trace and all kinds of useful information if somebody hits a 500 error:

code:
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
    }
}

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Put it in your settings.py file

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Aw crap I thought it would be easy to send email on Heroku but apparently it's not, you have to use a third party SMTP provider: https://devcenter.heroku.com/articles/smtp

Django can then be configured to use that third party SMTP server: https://docs.djangoproject.com/en/dev/topics/email/

So maybe not the route you want to go down right now if we're just trying to debug this 500 error. Regardless, to answer your logger functions in the views question, you don't have to do that stuff for the email alerts I was talking about.

I'm actually not familiar with foreman at all, so not sure on that one. Sorry :(

I think your solution is probably in the logging configuration though.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Python code:
def generate_photo_filename(instance, filename):
    image = Image.open(???)
    ext = image.format
    filename = '%s.%s' % (uuid.uuid4(), ext)
    return os.path.join('myproject/photos', filename)


class Person(models.Model):
    name = models.CharField(max_length=75)
    profile_photo = models.ImageField(upload_to=generate_photo_filename)
    cover_photo = models.ImageField(upload_to=generate_photo_filename)
I wanted to handle the scenario where somebody uploads whatever.jpg but it's actually a gif file.

What goes in the question marks? How does generate_photo_filename know if it's handling profile_photo or cover_photo? I could just make separate functions for upload_to, wondering if there is a better way to handle this though.

edit: and I suppose .jpg is just slightly more correct than .jpeg, but whatever we can ignore that for now

fletcher fucked around with this message at 02:47 on Apr 17, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Is it necessary to make a South migration for something like python-social-auth? If you just do a pip install python-social-auth and then ./manage.py schemamigration social.apps.django_app.default --initial, I think the migration file ends up somewhere inside my virtualenv (and thus outside of my version controlled repo). So I'm thinking ./manage.py syncdb is sufficient, no need to use South?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

MonkeyMaker posted:

Don't make migrations for software you don't control.

Ok cool, thanks!

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

supermikhail posted:

Re-bruteforcing my way through Django: I have an (aforementioned) Entry, and a Rating which refers to an entry via ForeignKey. I've been working on getting the average of an entry's ratings after the user has voted (with jQuery). On the view side:

return HttpResponse(serializers.serialize("json", entry.rating_set.all() ))

On the javascript side:

var parsed_input = $.parseJSON(input);
var numerator = 0;
var denominator = 0;
parsed_input.forEach(function(element){
numerator += element.fields.rating;
denominator++;
});
var replacement = (numerator/denominator).toFixed(2);

Except, of course, Entry has a perfectly good function for this purpose, and I look forward to the day I find out that Python and JavaScript have different precision and reloading the page from django and from javascript gives numbers embarrassingly differing by just 0.01.

Are my concerns valid? And is there a way to pass the average rating from an Entry in a more direct way?

Return the result of the Entry function that calculates the average as the HTTP result instead of returning all the ratings to calculate it client side

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
I replaced django-openid-auth with python-social-auth and got confused when I started seeing exceptions like this:

quote:

ImproperlyConfigured: Error importing authentication backend django_openid_auth.auth.OpenIDBackend: "No module named django_openid_auth.auth"

Couldn't figure out why it was still trying to use it even though there were no references to django_openid_auth in the code and then it hit me: it was probably storing it with the session.

code:
>>> from django.contrib.sessions.models import Session
>>> s = Session.objects.get(pk='dslrhofmnk5f69m7py7j2e5ihw8zmopp')
>>> s.get_decoded()
{'_auth_user_id': 689L, '_auth_user_backend': 'django.contrib.auth.backends.ModelBackend'}
Durrrr what a stupid mistake on my part!

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

supermikhail posted:

So now I intend to have javascript linked in the template it's used in. I'm a bit confused by the fact that I have to put load staticfiles every time, and also am wondering if this is the right way, because I haven't seen any recommendations re: arranging javascript files and it seems to be the usual practice to have everything in the same file. I think I'm remembering jQuery, in fact.

It's not the most efficient way to go about it but I just use django-pipeline to combine & minify all javascript files into one single file, and that gets included in my base.html (which all other templates extend). While DEBUG=True they all get included as separate un-minified files. There's still a somewhat loose correlation between the names of some javascript files and the names of some templates, but it's not very strict and there's some overlap here and there.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

supermikhail posted:

I don't see what the point of this is. Is it a performance boost if upon deployment everything is in a single file?

Minimizing the number of HTTP requests is definitely something you can do to improve performance. There are tradeoffs though, since you don't want to load a ton of code that you aren't going to use. The site I do it on doesn't have a ton of JavaScript so it's not a huge deal.

It keeps things simple though, I don't have to think about what JavaScript is loaded for which page, it's just all there all the time on every page.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Is there a standard convention for where you declare all your signal handlers?

edit: found this on the signals page https://docs.djangoproject.com/en/dev/topics/signals/

quote:

Where should this code live?
Strictly speaking, signal handling and registration code can live anywhere you like, although it’s recommended to avoid the application’s root module and its models module to minimize side-effects of importing code.

In practice, signal handlers are usually defined in a signals submodule of the application they relate to. Signal receivers are connected in the ready() method of your application configuration class. If you’re using the receiver() decorator, simply import the signals submodule inside ready().
Changed in Django 1.7:

Since ready() didn’t exist in previous versions of Django, signal registration usually happened in the models module.

I'm using 1.5. In the end of my models.py do I just do an import myapp.signals or something?

fletcher fucked around with this message at 23:29 on May 16, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Awesome! Thanks for the tips.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Ahz posted:

If this was a standard SQL recordset, I would make a single big query and pass back a large recordset with all the data I needed in a single obj so that I only hit the DB once.

You can do that in Django as well through the use of select_related and prefetch_related. Just use those with your original query for the order and then you don't have to worry about your loops in templates generating additional queries.

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Should I be avoiding the use of auto_now and auto_now_add?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

MonkeyMaker posted:

nope. I don't see any warnings or deprecations, do you?

Yeah I figured it was probably ok. I was googling around and saw some people had pretty strong opinions about how they were evil and should be avoided, but the posts were quite a few years old.

Aaaaand I just realized that I've had auto_now and auto_now_add backwards in all my field definitions this whole time :downs:

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Let's say I have a couple models: Blog and BlogComment. BlogComment has a ForeignKey to Blog. How to I use the ORM to find all the Blogs that have at least 1 comment? Or should I add a comment_count field to Blog and just filter on that?

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Haven't used annotate or Count yet. That's pretty sweet! Thanks Blinkz0rz!!

Here's the SQL it generated:
code:
 SELECT blog.id,
       blog.author_id,
       blog.publish_date,
       blog.title,
       blog.slug_title,
       blog.text,
       Count(blogcomment.id) AS num_comments
FROM   blog
       LEFT OUTER JOIN blogcomment
                    ON ( blog.id = blogcomment.blog_id )
GROUP  BY blog.id
HAVING Count(blogcomment.id) > 0  

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
Is there a way to programatically set an ImageField to an existing local file and have it run through my upload_to override?

Adbot
ADBOT LOVES YOU

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb
This seemed to work for me: http://stackoverflow.com/questions/3723220/how-do-you-convert-a-pil-image-to-a-django-file

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply