Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
porksmash
Sep 30, 2008

velvet milkman posted:

I'm going insane here trying to sort out an issue I'm having with serving gzipped files.

Question: Are you serving them with Django's static file system? If so it's definitely not correctly setting the headers, specifically Content-Disposition header. The browser is trying to automatically decode and display as text, since serving up gzip content is extremely common. Not sure why it's failing though.

I know this problem can be easily solved with a real web server, but in production it might be tricky. You'd have to have a custom view and URL to serve up the files that sets Content-Disposition correctly to something like
pre:
Content-Disposition: attachment; filename="filename.gz"

Adbot
ADBOT LOVES YOU

huhu
Feb 24, 2006
Is there a way to write this monstrosity more clearly/efficiently? I've cobbled this together to get thumbnails.
code:
 
im_thumb_small = Image.open(input_full_path)
im_thumb_small.thumbnail((200,200))
thumb_io = BytesIO()
im_thumb_small.save(thumb_io, format='JPEG')
thumb_small_file = InMemoryUploadedFile(
    thumb_io,
    None,
    '{}.{}'.format(sequence, file_extension),
    'image/jpeg',
    None,
    None
)

src_thumbnail_small = File(
    name = os.path.join('small', year, location, '{}.{}'.format(sequence, file_extension)),
    file = thumb_small_file
)

photo = Photo(
    ...
    src_thumbnail_small  = src_thumbnail_small,
    
)
photo.save()

huhu fucked around with this message at 19:58 on Sep 8, 2018

Lumpy
Apr 26, 2002

La! La! La! Laaaa!



College Slice
Sanity check:

Assume we have a model "Thing"

Python code:
class Thing(models.Model):
   name = models.CharField(max_length=50, unique=True)
   some_other_field = models.CharField(max_length=10, unique=True)
   ...
Requirements have changed so that a Thing can have multiple some_other_fields now. I'm guessing I should create a SomeOtherField model and give it a ForeignKey to a Thing, then write a script that will go through all existing Things, grab their some_other_fields, create SomeOtherField objects that tie to the Thing in question, then delete the some_other_field on Thing once I have re-written huge parts of the app do deal with this change.

What dumb thing am I missing / what smart thing should I do instead?

Average Lettuce
Oct 22, 2012


Lumpy posted:

Sanity check:

Assume we have a model "Thing"

Python code:
class Thing(models.Model):
   name = models.CharField(max_length=50, unique=True)
   some_other_field = models.CharField(max_length=10, unique=True)
   ...
Requirements have changed so that a Thing can have multiple some_other_fields now. I'm guessing I should create a SomeOtherField model and give it a ForeignKey to a Thing, then write a script that will go through all existing Things, grab their some_other_fields, create SomeOtherField objects that tie to the Thing in question, then delete the some_other_field on Thing once I have re-written huge parts of the app do deal with this change.

What dumb thing am I missing / what smart thing should I do instead?

I would do this on a single Migration with Migration Operations. You'd have to have three steps, first you have to store in memory a dict relating the string to the specific object it belongs to, then you run the migrations, then you go through the stored dict and create the new models and relations. I'm not sure if the Migration Operations have a cleaner way of doing this.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Lumpy posted:

Sanity check:

Assume we have a model "Thing"

Python code:
class Thing(models.Model):
   name = models.CharField(max_length=50, unique=True)
   some_other_field = models.CharField(max_length=10, unique=True)
   ...
Requirements have changed so that a Thing can have multiple some_other_fields now. I'm guessing I should create a SomeOtherField model and give it a ForeignKey to a Thing, then write a script that will go through all existing Things, grab their some_other_fields, create SomeOtherField objects that tie to the Thing in question, then delete the some_other_field on Thing once I have re-written huge parts of the app do deal with this change.

What dumb thing am I missing / what smart thing should I do instead?

Like that ^ guy mentions, you should do anything that modifies your db and data in a migration. They're pretty easy to do.

1. First add some_other_other_field as a FK to a model with your data and create a migration.
2. Add an empty migration and insert your code to migrate your data from some_other_field to your new some_other_other_field FK'ed model.
3. Add another migration removing some_other_field
4. Add another migration renaming some_other_other_field to some_other_field.

Ahz
Jun 17, 2001
PUT MY CART BACK? I'M BETTER THAN THAT AND YOU! WHERE IS MY BUTLER?!
Opinion Poll:

Camel Case for model field names for an API/DRF Django project?

I think I am getting sick of snake casing fields then deciding whether I delivery snake case metadata to my api, or convert case before sending data and just say gently caress convention and camelCase the entire models.py!

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Snake case because most of your coding should be done on the backend, not the front.

Data Graham
Dec 28, 2009

📈📊🍪😋



Unless you WANT all your clients to get access to your super cool business logic because it's in client-side JS

Dominoes
Sep 20, 2007

Thermopyle posted:

Snake case because most of your coding should be done on the backend, not the front.
That depends on what you're doing. There are lots of things that can be calculated in-browser without having to pass data over the internet.

Ahz
Jun 17, 2001
PUT MY CART BACK? I'M BETTER THAN THAT AND YOU! WHERE IS MY BUTLER?!
I still don't see it. At the end of the day, an API dumps a bunch of JSON to the client for whatever reason and after a dozen or so projects, I haven't seen the benefit of a whole bunch of field name conversions on the front-end.

Unless you're saying do the field name conversion pre-response from the server, i.e. a whole bunch of metadata manipulation on the server either in the response or the serializer.

Either way whether I convert field names in the client or the server, this conversion seems like a waste of effort when I can just sync names across front/backend for trivial poo poo like fName, lName, email etc. right in the models.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Dominoes posted:

That depends on what you're doing. There are lots of things that can be calculated in-browser without having to pass data over the internet.

If you're not passing it over the internet then it's not applicable to this question.


Ahz posted:

I still don't see it. At the end of the day, an API dumps a bunch of JSON to the client for whatever reason and after a dozen or so projects, I haven't seen the benefit of a whole bunch of field name conversions on the front-end.

Unless you're saying do the field name conversion pre-response from the server, i.e. a whole bunch of metadata manipulation on the server either in the response or the serializer.

Either way whether I convert field names in the client or the server, this conversion seems like a waste of effort when I can just sync names across front/backend for trivial poo poo like fName, lName, email etc. right in the models.

Yes, there's not a benefit of a whole bunch of field name conversions. My point is that you shouldn't be doing as much with the field names on the front end anyway, so if you're going to break the conventions on the frontend or the backend, its better to do it on the frontend since the amount of times you're touching the field names is less there.

Thermopyle fucked around with this message at 20:47 on Sep 14, 2018

Data Graham
Dec 28, 2009

📈📊🍪😋



Another consideration is that who says the backend's conventions should bend only to JavaScript's? Your API should be consumable by mobile apps too, which means you also have to think about clients written in Objective-C, Swift, and Java. Or really a client might be anything, including python. If you want your API to be portable it should be agnostic.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Also, I think it shouldn't be that big of a deal to use your frontends data access layer to auto convert snake case to camelcase and then not ever think about it again.

Cock Democracy
Jan 1, 2003

Now that is the finest piece of chilean sea bass I have ever smelled
It's now over, but was anyone else at DjangoCon US this year? I really had a great time. What an awesome community.

porksmash
Sep 30, 2008
Sadly I couldn't make it. Impatiently waiting for the videos to come up on youtube.

Cock Democracy
Jan 1, 2003

Now that is the finest piece of chilean sea bass I have ever smelled

porksmash posted:

Sadly I couldn't make it. Impatiently waiting for the videos to come up on youtube.
Yeah, it'll probably be awhile. In prior years, the wait has been about a month. This year they are adding closed captions, so it will probably be even longer. My favorite talks were: elasticsearch, how to give a drat, and autism devices. I heard that these were really good too, but I missed them: it's about time, easier classes, state of django panel, your web framework needs you. Also heard some rumors that they're considering Denver for next year.

MonkeyMaker
May 22, 2006

What's your poison, sir?

Cock Democracy posted:

It's now over, but was anyone else at DjangoCon US this year? I really had a great time. What an awesome community.

I was there (you saw me on stage a few times, since I was the co-chair). Videos should be about 2 weeks, even with captioning. Next year is probably San Diego again (I haven't heard of any hosting proposals from Denver) but we don't have a confirmed location yet.

Cock Democracy
Jan 1, 2003

Now that is the finest piece of chilean sea bass I have ever smelled

MonkeyMaker posted:

I was there (you saw me on stage a few times, since I was the co-chair). Videos should be about 2 weeks, even with captioning. Next year is probably San Diego again (I haven't heard of any hosting proposals from Denver) but we don't have a confirmed location yet.
Ah ok, I know who you are. Thanks for info, and great job on the conf this year. I'd be thrilled if it were in San Diego again.

Data Graham
Dec 28, 2009

📈📊🍪😋



I have Celery+RabbitMQ running happily on several FreeBSD and Linux servers. A few days ago I stood up a new one on FreeBSD, and for some unknowable reason the Celery clients keep disconnecting themselves, seemingly due to not being able to send heartbeats.

code:
2018-10-31 09:08:33.166 [error] <0.2210.0> closing AMQP connection <0.2210.0> (127.0.0.1:48360 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s
2018-10-31 09:08:34.176 [error] <0.2223.0> closing AMQP connection <0.2223.0> (127.0.0.1:21796 -> 127.0.0.1:5672):
missed heartbeats from client, timeout: 60s
Poking around issue threads seems to turn up that this is just "one of those things" that isn't due to any one particular causal factor, and nobody has any obvious solution despite several putative fixes to this very issue having been put into Celery over the past few years. I spent two evenings trying without success to disable heartbeats and finding only people mumbling in comment sections that you shouldn't do that and here's how you would if you ever wanted to but it isn't supported and just crashes if you try so :shrug:

Found that if I was in this disconnected state, the first async task I tried to send to Celery would always fail, but then on the second try the workers would wake up again and it would start working, so maybe I needed to build some kind of retry thing at the JS level? Urg

I was halfway through writing my own keepalive task to run through celery-beat when I thought "Hmm, can't you also use redis for the broker?"

Installed redis, was up and running with no issues in literally less than a minute. No bizarre config tools, no mutant erlang libraries, no documentation written in broken english, no heartbeat errors. Why is this not the recommended solution

Data Graham fucked around with this message at 03:24 on Nov 1, 2018

NtotheTC
Dec 31, 2007


I've always used celery + redis with django projects simply because I'm generally running redis for caching anyway.

I think I remember a discussion by the maintainer of Celery complaining that he had too much work to do and if he dropped support for redis as a broker he could focus entirely on rabbitMQ and then quite a lot of people spoke up saying that if he dropped redis support they'd move on to huey or rq or something that DID support redis rather than switch to rabbitmq

BaronVonVaderham
Jul 31, 2011

All hail the queen!

NtotheTC posted:

I've always used celery + redis with django projects simply because I'm generally running redis for caching anyway.

Same, and I've never had an issue just installing redis and sticking "redis://" as my CELERY_RESULT_BACKEND settings var. Bonus is then it works if I just run a local server in my terminal or boot up my full docker cluster without having to have any alternate configs as long as I expose the right port on the redis container.


While we're on the topic, running into a problem with celery myself. I've never actually written unit tests for celery tasks that kick off a group of other tasks themselves. These are coded such that I run through some logic to gather the tasks into a list while looping through a queryset and comparing it to a dump file from an external API (in the test I mock the response with a truncated version saved locally). Outside the loop I then do my_task_list_var.apply_async(), since I want these running in parallel asynchronously across my cluster of workers.

My problem is inside of the test this doesn't work, I suspect because I need to somehow spawn a worker when I set this up. I've never done this before, my tasks have always been standalone so I could just call explicitly as .apply() in the test method...but these subtasks are coded as async within the parent task. Anyone ever set up testing of a scenario like this before have a tip to get this going?

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

The problem with celery is that its too broad and "supports" too much for the number of core developers that work on it.

It's good for what it is, but you have to know which areas to avoid because of their neglected status.

Nowadays, 90% of the time I use python-rq because it's more narrowly focused and doesn't suffer from this particular type of problem to the degree that celery does. Of course, celery does more stuff and when I need that stuff I use it to great effect.

epswing
Nov 4, 2003

Soiled Meat
I have a few workflows that I know could benefit from queues. My site is pretty low-traffic, so it's not urgent, but e.g. I'm
  • processing a payment (3rd party request)
  • then generating an image
  • then generating a pdf
  • then adding the image to the pdf (along with some text)
  • then attaching the pdf to an email (along with some text)
  • and then sending the email (3rd party request)
This all happens inline, and takes a couple of seconds to execute, during which time the user is looking at a frozen browser tab, until it rolls over to the success page. I know that after the payment is processed, I should just show the success page right away and queue up the rest of the work. I've looked at celery, and rabbitmq, with e.g. a redis backend, but it all seems so heavyweight, all these extra processes and dependencies (and the knowledge required to run and debug it). I've heard devs mention that an in-memory db like redis should be used because Django ORM is "so very painfully slow", but honestly it's a low traffic site, and I think storing the work in a local postgres db is probably fast enough for my purposes. Also I'm allergic to over-engineering.

hitze
Aug 28, 2007
Give me a dollar. No, the twenty. This is gonna blow your mind...

epalm posted:

I have a few workflows that I know could benefit from queues. My site is pretty low-traffic, so it's not urgent, but e.g. I'm
  • processing a payment (3rd party request)
  • then generating an image
  • then generating a pdf
  • then adding the image to the pdf (along with some text)
  • then attaching the pdf to an email (along with some text)
  • and then sending the email (3rd party request)
This all happens inline, and takes a couple of seconds to execute, during which time the user is looking at a frozen browser tab, until it rolls over to the success page. I know that after the payment is processed, I should just show the success page right away and queue up the rest of the work. I've looked at celery, and rabbitmq, with e.g. a redis backend, but it all seems so heavyweight, all these extra processes and dependencies (and the knowledge required to run and debug it). I've heard devs mention that an in-memory db like redis should be used because Django ORM is "so very painfully slow", but honestly it's a low traffic site, and I think storing the work in a local postgres db is probably fast enough for my purposes. Also I'm allergic to over-engineering.

I used Django-Q for my low traffic stuff that needed queues. https://github.com/Koed00/django-q

NtotheTC
Dec 31, 2007


Huey is quite a lightweight little library that integrates well with Django... it defaults to using Redis at the backend and I don't think it has an option to use a postgres DB out of the box though.

I'm heavily biased on Redis because I use it everywhere but honestly it's pretty easy to install and use with Django just using the defaults. The complexity increases massively when load balancers get involved and you need redis clusters etc but if you're just using it as a K:V store or broker for task queues you can have it running in 3 commands.

epswing
Nov 4, 2003

Soiled Meat

hitze posted:

I used Django-Q for my low traffic stuff that needed queues. https://github.com/Koed00/django-q

Thanks, I'll check this out.

NtotheTC posted:

Huey is quite a lightweight little library that integrates well with Django... it defaults to using Redis at the backend and I don't think it has an option to use a postgres DB out of the box though.

I'm heavily biased on Redis because I use it everywhere but honestly it's pretty easy to install and use with Django just using the defaults. The complexity increases massively when load balancers get involved and you need redis clusters etc but if you're just using it as a K:V store or broker for task queues you can have it running in 3 commands.

If the server is shutdown/rebooted, and there are items in the queue (i.e. in redis), do they all just get vaporized?

NtotheTC
Dec 31, 2007


epalm posted:

Thanks, I'll check this out.


If the server is shutdown/rebooted, and there are items in the queue (i.e. in redis), do they all just get vaporized?

I think redis by default persists things to a file as well as stored in memory. So if the server is rebooted redis will just reload whatever was in the file to memory and pick it up again

https://redis.io/topics/persistence

NtotheTC fucked around with this message at 07:08 on Nov 7, 2018

MonkeyMaker
May 22, 2006

What's your poison, sir?
They're not all uploaded yet (about 50%) but here are the DjangoCon US 2018 talks

CarForumPoster
Jun 26, 2013

⚡POWER⚡
Any tips for hiring someone to develop a Django based site when I personally don’t know much about django and am just becoming handy in Python?

NtotheTC
Dec 31, 2007


Someone who can demonstrate their ability with full stack development (I. E. has websites live) would be the safe bet.

You could always cross post in the small jobs thread (if it's a small job).

If it's a permanent hire and you're looking for hints that they know their stuff then maybe set a small project to prescreen applicants? Just a page or two that's vaguely relevant to the job at hand to get an idea whether they're using the right tools etc

porksmash
Sep 30, 2008
Question to those who have upgraded a project from Django 1.x to 2.0+ - I have added the requisite on_delete kwarg to all my ForeignKey fields, however there are several old migrations that do not have on_delete specified. Squashing is not an option just yet, as the upgrade process has been a bitch and I have had to make several migration steps to resolve issues/upgrade packages/etc that are not yet live and can't be squashed OR deployed until the Django 2.1 upgrade is completed.

Is the best solution to just edit the migrations to include on_delete?

BaronVonVaderham
Jul 31, 2011

All hail the queen!
Anyone in here have any experience integrating a third party Vue template/theme/library/package thing into a django app?

I found one that I really like (https://themeforest.net/item/cover-dashboard-ui-kit-for-modern-web-application/19869018?s_rank=3) from a company where a previous employer had acquired our theme library (https://themeforest.net/item/unity-html-responsive-multipurpose-template/16177004).

I was not involved in the acquisition or translating that into our internal theme library module, I just imported that, inherited everything, and used it, but it was mostly CSS. For the JS I used, I basically cherry-picked the pieces I want and copied the source into template fragments, as I was building a custom fein-cms system, so I had to build out these fragments anyway to define my content types.

But for my current app I'm working on, the goal is to build the front end using Vue.js with django rest framework on the back end serving up the data.

I can't seem to figure out the best way to go about building this from that library package. Their documentation seems rather sparse, but also seems to assume I'm building this as a 100% JS app, not integrating it with any other framework. I really don't want to repeat what I did with that other theme and import things component by component, the entire point of getting a third-party theme was to be able to just bulk import the lot in my base template and just override the CSS with my site's theming.

NtotheTC
Dec 31, 2007


E: sorry I completely misread your post, ignore me

For myself, I wouldnt try to combine the vuejs into the django. There have been any number of libraries that try to integrate frontend frameworks with django forms and they all work great for the cookie cutter hello world example blog app but the moment you try to do anything real with it it becomes horrible and complicated to work with.

If you're happy with writing the django app with Rest framework then do that, and have the vuejs be an entirely separate SPA that calls it. Then you get nice nest codebases instead of one convoluted one

BaronVonVaderham
Jul 31, 2011

All hail the queen!

NtotheTC posted:

E: sorry I completely misread your post, ignore me

For myself, I wouldnt try to combine the vuejs into the django. There have been any number of libraries that try to integrate frontend frameworks with django forms and they all work great for the cookie cutter hello world example blog app but the moment you try to do anything real with it it becomes horrible and complicated to work with.

If you're happy with writing the django app with Rest framework then do that, and have the vuejs be an entirely separate SPA that calls it. Then you get nice nest codebases instead of one convoluted one

I'm pretty comfortable combining them now, it's something we use extensively at my current job (which is where I learned Vue exists, my last jobs used React and I wanted to die). It's not too bad to use.

It's just a matter of figuring out how the hell to import this pre-built stuff and use it. I'm starting to think it's more trouble than it's worth trying to use this as a fancy npm module, all I can do is import the .css and .js files in the /lib/ directory of this package and build it component by component..... I guess not the worst thing, since that lets me tack on my own styling as I go.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

I just came across this image that illustrates something I've explained a million times over the years. Perhaps someone else will find it useful.

Lumpy
Apr 26, 2002

La! La! La! Laaaa!



College Slice
DRF question time (again!):

Given this:

Python code:
class Organization(models.Model):
    name=models.CharField(...)
    ...

class OrgFlag(models.Model):
    name=models.CharField(...)
    organizations=models.ManyToManyField(Organization, related_name="orgflags")
    ...

class Person(models.Model):
    name=models.CharField(...)
    organization=models.ForeignKey(Organization)
    ...
A Person belongs to an Organization, and an Organization can have different flags associated with them. What I want to have happen is when a Person gets their own profile, I want it to have a list of the names of the flags their associated organization has. So the return would look like:

code:
{ id: 1,
  name: 'Lumpy',
  orgflags: ['A_FLAG', 'ANOTHER_ONE']
}
My current thinking is that I have to make a special Organization serializer that only contains orgflags with many=True, that calls an OrgFlag serializer that just has the name as a field, and then the Person serializer uses that to serialize their organization field.

Does that sound correct, and is my placement of the organization flags sane, or should the field linking them be on Organizations? I always get confused by this, and do it so rarely that I get re-confused every time....

IAmKale
Jun 7, 2007

やらないか

Fun Shoe

Lumpy posted:

DRF question time (again!):

Given this:

Python code:
class Organization(models.Model):
    name=models.CharField(...)
    ...

class OrgFlag(models.Model):
    name=models.CharField(...)
    organizations=models.ManyToManyField(Organization, related_name="orgflags")
    ...

class Person(models.Model):
    name=models.CharField(...)
    organization=models.ForeignKey(Organization)
    ...
A Person belongs to an Organization, and an Organization can have different flags associated with them. What I want to have happen is when a Person gets their own profile, I want it to have a list of the names of the flags their associated organization has. So the return would look like:

code:
{ id: 1,
  name: 'Lumpy',
  orgflags: ['A_FLAG', 'ANOTHER_ONE']
}
My current thinking is that I have to make a special Organization serializer that only contains orgflags with many=True, that calls an OrgFlag serializer that just has the name as a field, and then the Person serializer uses that to serialize their organization field.

Does that sound correct, and is my placement of the organization flags sane, or should the field linking them be on Organizations? I always get confused by this, and do it so rarely that I get re-confused every time....
My hunch is that it'll be easier (that is, DRF intends) for you to specify a depth in your PersonSerializer, then access org_flags as a nested value of the organization property on your Person instance's JSON representation:

code:
{
  id: 1,
  name: 'Lumpy',
  organization: {
    name: 'SA Inc',
    org_flags: ['A_FLAG', 'ANOTHER_ONE']
  }
}
That said, if you don't want the entire organization represented in the payload, you might be able to use the Serializer.to_representation() method to flatten org_flags into the main user object that gets returned.

NtotheTC
Dec 31, 2007


Is there a go-to e-commerce library for django that you guys use? Theres any number of them on django packages and django-oscar appears to be winning but popularity does not always trump experience

epswing
Nov 4, 2003

Soiled Meat
This is probably a stupid question but, who makes the decision that Django should use python2 or python3? I'm not sure where in the collection of [ubuntu, nginx, wsgi, gunicorn] this decision is made.

Adbot
ADBOT LOVES YOU

NtotheTC
Dec 31, 2007


epalm posted:

This is probably a stupid question but, who makes the decision that Django should use python2 or python3? I'm not sure where in the collection of [ubuntu, nginx, wsgi, gunicorn] this decision is made.

Your webserver (gunicorn, uwsgi) is what points to the python environment

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply