Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I usually copy/paste my logging config from one project to the next so I'm always a little hazy on the details of how logging works because I don't usually think about it.

But I think you need to set propagate to False for django.db.backends because otherwise that handler logs it and then the django logger logs it.

Adbot
ADBOT LOVES YOU

NtotheTC
Dec 31, 2007


What do you guys turn to for single sign on in Django these days? We're looking to be moving over to an oauth2 so solution for our various apps. Django all auth was my first thought but I wondered if that was mainly aimed at social auth

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I've never been happy with any of the pre-packaged things so I roll my own.

It's not terribly difficult if you understand how SSO works.

NtotheTC
Dec 31, 2007


Thermopyle posted:

I've never been happy with any of the pre-packaged things so I roll my own.

It's not terribly difficult if you understand how SSO works.

Ah that's interesting. Do you have something I could look at? I understand the process but my implementation knowledge is lacking.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

NtotheTC posted:

Ah that's interesting. Do you have something I could look at? I understand the process but my implementation knowledge is lacking.

Not really as every time I've done it it's been contract work that is kinda tied to the clients needs because I didn't have the time or money budget to abstract it out into its own thing that I could put out there for the world to use.

Just jump in to implementing it and it won't be too bad.

epswing
Nov 4, 2003

Soiled Meat

NtotheTC posted:

What do you guys turn to for single sign on in Django these days? We're looking to be moving over to an oauth2 so solution for our various apps. Django all auth was my first thought but I wondered if that was mainly aimed at social auth

I recently (late last year) implemented SSO using social-auth-app-django. It wasn't too bad to implement. My company uses Google for Business (G-Suite) and now we can just log into our company email, and also have access to our internal (django-backed) portal without having to log in again.

NtotheTC
Dec 31, 2007


epalm posted:

I recently (late last year) implemented SSO using social-auth-app-django. It wasn't too bad to implement. My company uses Google for Business (G-Suite) and now we can just log into our company email, and also have access to our internal (django-backed) portal without having to log in again.

The issue is we won't be using any type of social auth, it'll be our own implementation so social-auth and all-auth are a bit heavyweight when I'm not using 90% of their functionality.

I'll probably try and implement our own version if I can't find a package I like the look of

E: that said social auth does a good job of separating out the various bits so maybe it's not a bad starting point

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

The problem with social-auth-app-django is that it and python-social-auth are not maintained very well so its risky to base a business upon what they do. There's not really anyone there taking the lead and managing pull requests, issues, etc.

I stopped using them too because of that.

Implementing social auth yourself is not hard...once you've figured out the mess that is oauth(2). The benefit social-auth-app-django really brings to the table is the wide social provider support. On the other hand, because of the bad project management it can be months before a fix gets pushed when one of those social providers changes something.

Data Graham
Dec 28, 2009

📈📊🍪😋



HOLY JESUS GOD

I didn't know you could undo a migration by doing ./manage.py migrate <app> <previous_migration>

THIS CHANGES EVERYTHING



(e. I don't know what I thought reverse_code was for then!)

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Migrations are cool and good.

However, it's still fricking irritating managing database state between different branches of your code repo.

duck monster
Dec 15, 2004

Thermopyle posted:

Migrations are cool and good.

However, it's still fricking irritating managing database state between different branches of your code repo.

Its a ridiculous problem, and a source of unending teeth gnashing and pain. I've worked on big teams where we had to take an approach of maintaining a branch called "database_migrations", and whenever you wanted to mod the schema, you told the project manager your plan, and he'd let you check out the branch, update the schema migrations, test it and unit test it, and check it back in, THEN, you merge that branch into your own one , and proceed from there. that way your schema is in everyone elses hands for breakage testing/compatibility before you need to merge your branch back upstream.

Don't even get me started on schema.rb on ruby on rails. That loving file fails constantly in merges, because migrations love to rewrite it, all of which is a cosmic punishment for using that God forsaken ORM. God I'm glad I'm not "on rails" anymore.

IAmKale
Jun 7, 2007

やらないか

Fun Shoe
Any Graphene/GraphQL experts here? I'm starting a small Django side project after getting up to speed with Graphene and need some insight on how I might construct my models to allow for a certain structure of GraphQL query.

Basically, I'm working on a API in which you can see if your theme park membership allows you to go "today". As far as queries go, I want to make a request like this:

code:
query {
  today {
    location1
    location2
  }
}
And get back a data structure like this:

code:
{
  "data": {
    "today": {
      "location1": {
        "passes": [
          { name: "pass1", canGo: true },
          { name: "pass2", canGo: true}
          { name: "pass3", canGo: false}
        ]
      },
      "location2": {
        "passes": [
          { name: "pass4", canGo: true },
          { name: "pass5", canGo: false}
        ]
      }
    }
  }
}
Right now I've got the following couple of Django models in mind to store this data:

Python code:
class LocationModel:
    name = CharField

class PassModel:
    name = CharField
    date = ForeignKey(DateModel)
    location = ForeignKey(LocationModel)
    can_go = Boolean
Would this be sufficient for filtering by location name? I'm really new to GraphQL and the flexibility of the fields that can be specified in queries is making me think I have to model my data a little differently than I would with a typical REST API.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.
I've made a Python Selenium script that scrapes my bank accounts and stores the information in a database. I've decided I want to build a cross platform frontend, and some reading suggests I can do that with Django. I'm envisaging a simple website that's basically shows an image for each account, and underneath the most recent balance and when it was scraped. There would then be a tick box for each bank, and a button for 'audit' which would audit anything that's ticked by running the appropriate Python code.

Am I completely wrong that something like that is a sensible project to do in Django? I'm perfectly happy to have my idea shot down, and find a better way to make the GUI (PyQT maybe?).

Hed
Mar 31, 2004

Fun Shoe
Django would work and is great, but if you don’t need a DB with modeled objects (the ORM) it would probably be easier to just use bottle or flask IF you don’t already have experience with Django.

To the second part I really think web sites are easier to design and dork around with than any GUI toolkit these days.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.

Hed posted:

Django would work and is great, but if you don’t need a DB with modeled objects (the ORM) it would probably be easier to just use bottle or flask IF you don’t already have experience with Django.

To the second part I really think web sites are easier to design and dork around with than any GUI toolkit these days.

The DB is just a pretty simple mySQL database right now with a couple of tables and the code itself was written without any thought to OOP. How complicated would something like that be to implement? If that project works, then there's a separate but similar project that I'd be wanting to implement. It has a lot more functionality (calendar, email, cron jobs). That makes me tempted to learn Django, as long as it's not ridiculous.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

That would be very simple to implement in Django.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.

Thermopyle posted:

That would be very simple to implement in Django.

Great to hear. In the OP the suggested beginner resource is a book. https://wsvincent.com/django-for-beginners/ seems to have some decent enough reviews on Amazon. Any other suggestions, especially if related to my imminent project, would be greatly appreciated.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Sad Panda posted:

Great to hear. In the OP the suggested beginner resource is a book. https://wsvincent.com/django-for-beginners/ seems to have some decent enough reviews on Amazon. Any other suggestions, especially if related to my imminent project, would be greatly appreciated.

I always find it hard to recommend stuff to beginners because it's hard for me to remember the state of knowledge back when I was at that point.

That being said, I'm fairly sure I got started just fine with the official django tutorial on the django site. I think I've been hearing good things about Mozilla's Django tutorial as well.

There's a great book called 2 Scoops of Django that is all about Django best practices. It's not a tutorial or exactly aimed at beginners, but I think maybe it's a good thing to have and peruse to help you get the lay of the land whilst going through beginners tutorials.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.
Edit - I'm an idiot. Of course there was no data being shown. I didn't move the data over.

My question however comes to database design. Should I be using separate databases for each application? Or a separate schema for each application? Googling about implementing multiple schema seems to get some confused answers.

Sad Panda fucked around with this message at 14:05 on Jul 25, 2018

Data Graham
Dec 28, 2009

📈📊🍪😋



Do you mean application in the sense of what Django considers a "project" (i.e. a top-level single web application), or a django "app" (i.e. a module within a django project)?

If the former, you want a separate database (/schema) per project, because each project maintains its own auth, migrations, content types, and other global structures.

If the latter, Django keeps all the apps (/modules) within the DB schema automatically, like in the "appname" module you would get tables like "appname_widget" and so on.

Sad Panda
Sep 22, 2004

I'm a Sad Panda.
Good clarification. So a DB for each project sounds good.

Current bizarreness is I guess related to caching?

I have /static/images/ with some files in. There used to be a header.jpg, but that got renamed however on stopping and starting the server it is still there. It seems like if there's ever been a file named that, it loads it. which is an absolute pain for debugging. How do I stop that caching? If I swap browser, then it won't 'remember' from the previous ones, so it's storing in the browser cache. But I'd like to stop that.

Python code:
{% load static %}
<div>
    {% for account in accounts %}
        <h3>{{ account.id}} </h3>
        <img src="/static/images/sally.jpg" alt="Typed">
        <img src="{% static 'images/header.jpg' %}" alt="{{ account.bankid}}">
    {% endfor %}
</div>

Data Graham
Dec 28, 2009

📈📊🍪😋



https://docs.djangoproject.com/en/2.0/ref/contrib/staticfiles/#manifeststaticfilesstorage

It changed my life

Sad Panda
Sep 22, 2004

I'm a Sad Panda.
Thank you for that.

OK, so given I've got a currently functional console-app, how am I supposed to import this in so I can call it? Do I use startapp and then copy and paste files into it? Would I then call that from my main views.py file? For example so I set it up so that if the user goes to localhost.com/audit it triggers the running of my audit script and displays a page?

NtotheTC
Dec 31, 2007


Sad Panda posted:

Thank you for that.

OK, so given I've got a currently functional console-app, how am I supposed to import this in so I can call it? Do I use startapp and then copy and paste files into it? Would I then call that from my main views.py file? For example so I set it up so that if the user goes to localhost.com/audit it triggers the running of my audit script and displays a page?

If you package up your python script you can install it as a requirement in your django project and then import it like any other module and run it from the view code

epswing
Nov 4, 2003

Soiled Meat
I'm having a hard time understanding why python manage.py dumpdata > backup.json followed by python manage.py loaddata backup.json doesn't ever Just Work on the first try. There's a ton of noise on Stack Overflow of people trying and failing to do this.

This time, the error is

quote:

django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'django_admin_log' with primary key '133' has an invalid foreign key: django_admin_log.content_type_id contains a value '22' that does not have a corresponding value in django_content_type.id.

I just want to dump all the data on the server, transfer it to my development machine, start with a fresh SQLite database, and load all the data. It's a small database, less than 100mb (actually closer to 15mb). I've tried various combinations of --exclude contenttypes and --exclude auth.Permission and get similar errors.

What's the best way to get what I'm after?
Is there a sure-fire way to do this using dumpdata and loaddata?
Is there some other tool I should be using?

Edit: the prod database is Postgres, if that matters.

epswing fucked around with this message at 17:08 on Jul 30, 2018

Data Graham
Dec 28, 2009

📈📊🍪😋



Try using --natural-primary on your dumpdata. That will prevent it from trying to reuse the same pk IDs, and it'll create its own new ones when you import.

epswing
Nov 4, 2003

Soiled Meat

Data Graham posted:

Try using --natural-primary on your dumpdata. That will prevent it from trying to reuse the same pk IDs, and it'll create its own new ones when you import.

Using --natural-primary I still get foreign key errors like

quote:

File "C:\Blah\lib\site-packages\django\db\backends\sqlite3\base.py", line 295, in check_constraints
bad_row[1], referenced_table_name, referenced_column_name,
django.db.utils.IntegrityError: Problem installing fixtures: The row in table 'mservice_servicereport' with primary key '6829' has an invalid foreign key: mservice_servicereport.user_id contains a value '24' that does not have a corresponding value in mhome_user.id.

Edit: Might this have something to do with the fact that I recently followed https://code.djangoproject.com/ticket/25313 to migrate from a built-in User model to a custom User model? Everything seems to be working but... do I need to do anything with the contenttypes table?

Regardless, I'm dumping all the data, then loading all the data. Maybe things being loaded out of order, i.e. the mservice_servicereport table is being loaded before the mhome_user table?

epswing fucked around with this message at 22:05 on Jul 30, 2018

Ahz
Jun 17, 2001
PUT MY CART BACK? I'M BETTER THAN THAT AND YOU! WHERE IS MY BUTLER?!

epalm posted:

Using --natural-primary I still get foreign key errors like


Edit: Might this have something to do with the fact that I recently followed https://code.djangoproject.com/ticket/25313 to migrate from a built-in User model to a custom User model? Everything seems to be working but... do I need to do anything with the contenttypes table?

Regardless, I'm dumping all the data, then loading all the data. Maybe things being loaded out of order, i.e. the mservice_servicereport table is being loaded before the mhome_user table?

Life got easier for me when I pretend these dump and load data commands don't exist and I just use direct DB tooling and then manage odd data schema changes via custom code + migrations.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Yeah, dump data poo poo has always been way more difficult than its worth when you can just write a quick script or use native sql tools to do the job.

epswing
Nov 4, 2003

Soiled Meat
drat. I was hoping that someone would tell me I'm dumb and to just use dumpdata/loaddata like this. It makes a lot of sense to use the middle-tier language and framework (in this case, Python and Django) to deal with the discrepancies between databases.

From what I'm reading, looks like I have to use pg_dump to generate the sql insert statements, then manually remove the SET statements, and lines starting with SELECT pg_catalog.setval, then replace true and false with "t" and "f" (or should that be 1 and 0?), and wrap the whole thing in BEGIN; and END; statements. I'd love to automate this, but I'm a little stuck on the replacement of true/false, because "what if the data contains a string with the word true in it?"

I guess the pain threshold for just installing postgres locally is lower than automating this into a tool.

(Unless there's a tool I'm not finding, other than some 2004 Java throwback on sourceforge? ;) )

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

epalm posted:

drat. I was hoping that someone would tell me I'm dumb and to just use dumpdata/loaddata like this. It makes a lot of sense to use the middle-tier language and framework (in this case, Python and Django) to deal with the discrepancies between databases.

From what I'm reading, looks like I have to use pg_dump to generate the sql insert statements, then manually remove the SET statements, and lines starting with SELECT pg_catalog.setval, then replace true and false with "t" and "f" (or should that be 1 and 0?), and wrap the whole thing in BEGIN; and END; statements. I'd love to automate this, but I'm a little stuck on the replacement of true/false, because "what if the data contains a string with the word true in it?"

I guess the pain threshold for just installing postgres locally is lower than automating this into a tool.

(Unless there's a tool I'm not finding, other than some 2004 Java throwback on sourceforge? ;) )

Installing postgres locally is super easy and it's a good idea to use the same database engine in development as in production.

docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres!

If you want to get fancy here's a quickly-googled guide to setting up your whole stack via docker: https://wsvincent.com/django-docker-postgresql/

Also, I agree that it makes sense to use the man who is already in the middle to do the inter-database translation. I've just never found the dump data stuff to work well without a lot of shenanigans. What I have done in the past is define two databases in my settings and then write a management command to dump all my data from one to the other.

epswing
Nov 4, 2003

Soiled Meat

Thermopyle posted:

Installing postgres locally is super easy and it's a good idea to use the same database engine in development as in production.

docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres!

If you want to get fancy here's a quickly-googled guide to setting up your whole stack via docker: https://wsvincent.com/django-docker-postgresql/

Interesting, yeah I might look into all this container jazz I keep reading about.

Thermopyle posted:

Also, I agree that it makes sense to use the man who is already in the middle to do the inter-database translation. I've just never found the dump data stuff to work well without a lot of shenanigans.

Yeah, likewise. Too bad, really.

Thermopyle posted:

What I have done in the past is define two databases in my settings and then write a management command to dump all my data from one to the other.

This sounds interesting. The management command, does it just do something like
Python code:
for author in Author.objects.using('default').all():
    author.save(using='other_database')
for all models in the correct order?

Edit: ^^Actually that can't be right, because the foreign keys wouldn't point to the right thing.

epswing fucked around with this message at 18:39 on Jul 31, 2018

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

epalm posted:



This sounds interesting. The management command, does it just do something like
Python code:
for author in Author.objects.using('default').all():
    author.save(using='other_database')
for all models in the correct order?

Edit: ^^Actually that can't be right, because the foreign keys wouldn't point to the right thing.

Well it's been years since I did it, but I just wrote whatever logic was needed to do the stuff needed.

Data Graham
Dec 28, 2009

📈📊🍪😋



When I've had to do a wholesale migration from one database to another via management command, I've had to go model by model, in dependency order, and set the id of each object after saving it, then saving it again.

I also save a temporary "old_id" field on every model so I can import the related models by looking up the old ID of the parent object and linking the new related object to it via the new ID.

epswing
Nov 4, 2003

Soiled Meat
I just folded and installed Postgres locally.

Then wrote a batch file to roughly do this, so I can one-click restore the prod database to my dev environment:
  • dropdb django
  • createdb django
  • ssh blah@host pg_dump django | psql django
Should've done this weeks ago :slick:

epswing fucked around with this message at 19:40 on Aug 1, 2018

Data Graham
Dec 28, 2009

📈📊🍪😋



This is more a rabbitmq/Celery question than a Django question, but I'm hoping someone has experience with the former because of the latter.

I'm trying to add a second vhost to my rabbitmq. I currently have one Django project (running through Apache) with a bunch of Celery workers connecting successfully to my local rabbitmq via vhost1. However, as soon as I go

rabbitmqctl add_vhost vhost2

Then the tasks that my project sends to vhost1 no longer work. The celery logs show no tasks being accepted. But then as soon as I "rabbitmqctl delete_vhost vhost2", tasks work again.

Anyone see anything like this? This is on FreeBSD, rabbitmq 3.7.7.

Dominoes
Sep 20, 2007

Hey dudes. Looking for wisdom on creating a non-saved subclass in a model. Eg I don't want to save a separate entry; just using it to reduce code.

Python code:
class Address(models.Model):
    country = models.CharField(max_length=3)  # ISO 3166-1 alpha-3
    address1 = models.CharField(max_length=150)
    address2 = models.CharField(max_length=150)
    town = models.CharField(max_length=100)
    postal = models.IntegerField()
    phone = models.IntegerField()


class Order(models.Model):
    datetime = models.DateTimeField()

    shipping_address = models.ForeignKey(Address, on_delete=CASCADE)
    billing_address = models.ForeignKey(Address, on_delete=CASCADE)
Ie: I don't want to save Address entries to the DB; I just don't want to repeat the code, and want their data to be in the Order table, self-contained. OneToOneFields ?

Eg without the DB tie-in:

Python code:
@dataclass
class Address:
    country: str
    address1: str
    address2: str
    #...

@dataclass
class Order:
    shipping_address: Address
    billing_address: Address

Dominoes fucked around with this message at 07:45 on Aug 3, 2018

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Look up abstract models.

They don't let you do what you described exactly, but it's the closest you'll get.

FWIW, there's nothing wrong with having an Address model with its own table.

Thermopyle fucked around with this message at 14:15 on Aug 3, 2018

Ahz
Jun 17, 2001
PUT MY CART BACK? I'M BETTER THAN THAT AND YOU! WHERE IS MY BUTLER?!

Dominoes posted:

Hey dudes. Looking for wisdom on creating a non-saved subclass in a model. Eg I don't want to save a separate entry; just using it to reduce code.

Python code:
class Address(models.Model):
    country = models.CharField(max_length=3)  # ISO 3166-1 alpha-3
    address1 = models.CharField(max_length=150)
    address2 = models.CharField(max_length=150)
    town = models.CharField(max_length=100)
    postal = models.IntegerField()
    phone = models.IntegerField()


class Order(models.Model):
    datetime = models.DateTimeField()

    shipping_address = models.ForeignKey(Address, on_delete=CASCADE)
    billing_address = models.ForeignKey(Address, on_delete=CASCADE)
Ie: I don't want to save Address entries to the DB; I just don't want to repeat the code, and want their data to be in the Order table, self-contained. OneToOneFields ?

Eg without the DB tie-in:

Python code:
@dataclass
class Address:
    country: str
    address1: str
    address2: str
    #...

@dataclass
class Order:
    shipping_address: Address
    billing_address: Address

If you're talking about a real ordering system, shits more complicated than that anyways. Address is historical information for point-in-time anyways, so you need a way to save both the current addresses (ship/bill) but also the historical for each order.

Nothing wrong with denormalizing a little bit in the order and keeping separate tables for current address data.

Adbot
ADBOT LOVES YOU

velvet milkman
Feb 13, 2012

by R. Guyovich
I'm going insane here trying to sort out an issue I'm having with serving gzipped files.

I have a web application where users upload fairly large compressed text files (the typical extension is *.fastq.gz), among other things. Right now I'm in the development stage so I'm serving the media locally. Serving anything except these *.fastq.gz files is working fine.

When I retrieve the *.fastq.gz file, it only downloads a partial (e.g. 50kb of 50mb) uncompressed version of the file. I know it's uncompressed because I can view this file in a text editor, and the contents are there, but only up to the first 50kb that I was able to download. Bizarrely, if I change the extension of the file in my media storage folder to something like *.fastq.zip, and then retrieve that file, everything works as expected and the full 50mb is downloaded. Of course I have to manually rename the file on my file system from .zip to .gz to decompress it properly, but it works.

What the hell is going on here? I have an inkling this has something to do with content encoding, but I'm so outside of my wheelhouse here I don't even know what to google at this stage, let alone how to implement the solution in Django. Any ideas?

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply