Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
I set up Django to use Celery with RabbitMQ. IIRC I'm using the older Celery 3.x series instead of 4.x because my server runs on Windows, and Celery 4.x won't support Windows. I got a very basic asynchronous task to run through Django, but I have a lot of questions about how to work with Celery at a more administrative level. I am assuming Celery exists in part to handle a lot of this kind of stuff, but I don't even know where to start digging to figure out some of this. I am planning to use Celery in particular to run a script to build something on different machines. I want the results of this to get back to Django and alert the end users that things are ready. My questions:

How can I show my users which agents are up and running? How can I manage them?
How can I tell if an agent went down?
What happens to tasks that are in-progress when an agent goes down? Will they try to run them again if the agent comes back up?
Can I serialize tasks on one particular machine?
How might I send live updates back to the end users sitting on a rich JavaScript front end? In particular, I want stream of log messages a la the Observer pattern. I was pondering using RabbitMQ directly for this.

Edit: For most of this, it appears the answer is to "Use Celery to its fullest and set up Flower for the monitoring and administration." I'm not sure how I might do the observer pattern, but that's a stretch right now anyways.

Rocko Bonaparte fucked around with this message at 00:16 on Feb 8, 2017

Adbot
ADBOT LOVES YOU

Count Thrashula
Jun 1, 2003

Death is nothing compared to vindication.
Buglord
I'm having trouble getting static images to show up in a for loop in my template.

code:
{% for item in mainthing.subthing_set.all %}
	<td class="text-center" style="padding-right: 10px;">
		<img class="img-rounded" style="border:1px solid black;" src="{% static 'app/images/items/'|add:item.item_type|add:'.gif' %}" />
	</td>
{% endfor %}
But I keep getting...
Invalid block tag on line 67: 'static', expected 'empty' or 'endfor'. Did you forget to register or load this tag?
...no matter what I put in the static tag, or even if I try to do a with block, I get "expected 'endwith'". Am I doing static wrong?

a witch
Jan 12, 2017

Did you include
code:
{% load static %}
in your template?

Count Thrashula
Jun 1, 2003

Death is nothing compared to vindication.
Buglord

a witch posted:

Did you include
code:
{% load static %}
in your template?

WHOOPS

Count Thrashula
Jun 1, 2003

Death is nothing compared to vindication.
Buglord
I'm having trouble piecing something together in my head.

So, say I have a game, and each scenario in that game requires certain maps. A user can have certain maps in their collection, and I'm trying to show if a scenario is playable or not based on if the user has those maps in their collection.

So...
model Map has a property "mapnumber" that is tied by foreignkey to model Scenario
model UserMap has a property "mapnumber" that is tied by foreignkey to model User

What I'm having trouble with is checking "do the Scenario.Maps exist in User.UserMaps".

I'm toying with something along the lines of (very abridged and off the top of my head)

code:
scen = Scenario.object.get(pk=1)
maps = scen.map_set.all()
user = User.object.get(pk=1)
user_maps = UserMap.object.filter(user=user)

playable = True
for map in maps:
   if map not in user_maps:
      playable = False
...but the syntax is escaping me based on working with the model objects. Something like "if map.name not in user_maps.name_set" doesn't work, but that's the sort of functionality I need, I guess. Do I just need two nested for loops?

Count Thrashula fucked around with this message at 01:27 on Feb 15, 2017

Hed
Mar 31, 2004

Fun Shoe
Is there an app that will plug in advanced search? I basically want this:
  • if someone puts "asdf" in the search field, do an icontains for asdf across several model fields and OR them (can do this pretty easily with Q objects in the query set)
  • if someone puts something like "remark:asdf" in the search field, just do an icontains in the model's remarks field

It would be straightforward enough to code in a mixin or whatever, but I'm surprised I haven't found anything searching so far.

Lumpy
Apr 26, 2002

La! La! La! Laaaa!



College Slice

COOL CORN posted:

I'm having trouble piecing something together in my head.

So, say I have a game, and each scenario in that game requires certain maps. A user can have certain maps in their collection, and I'm trying to show if a scenario is playable or not based on if the user has those maps in their collection.

So...
model Map has a property "mapnumber" that is tied by foreignkey to model Scenario
model UserMap has a property "mapnumber" that is tied by foreignkey to model User

What I'm having trouble with is checking "do the Scenario.Maps exist in User.UserMaps".

I'm toying with something along the lines of (very abridged and off the top of my head)

code:
scen = Scenario.object.get(pk=1)
maps = scen.map_set.all()
user = User.object.get(pk=1)
user_maps = UserMap.object.filter(user=user)

playable = True
for map in maps:
   if map not in user_maps:
      playable = False
...but the syntax is escaping me based on working with the model objects. Something like "if map.name not in user_maps.name_set" doesn't work, but that's the sort of functionality I need, I guess. Do I just need two nested for loops?

If you are only creating the UserMaps to tie the two together, why not just put a maps property on the User? Regardless, you should be able to do this all in one query by filtering User maps based on Scenario maps, then you are only returning "playable" ones.

Maluco Marinero
Jan 18, 2001

Damn that's a
fine elephant.

Hed posted:

Is there an app that will plug in advanced search? I basically want this:
  • if someone puts "asdf" in the search field, do an icontains for asdf across several model fields and OR them (can do this pretty easily with Q objects in the query set)
  • if someone puts something like "remark:asdf" in the search field, just do an icontains in the model's remarks field

It would be straightforward enough to code in a mixin or whatever, but I'm surprised I haven't found anything searching so far.

If you want a use a search engine you're probably looking for Haystack which has heaps of different options to plug in.

porksmash
Sep 30, 2008
I started using django-color-field for a couple model fields, but then later removed it. I now have a migration that depends on the django-color-field package, so I cant ever remove it and have to install it in every new python environment. Is the best way to get rid of this stupid dependency to squash my migrations?

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
How might I stream log messages from a celery task to an end user through a REST API? I wanted to centralize these logs in one spot just to know what went on, but it looks like I could go an extra step and start streaming it as it comes in to any users that care. As it stands, I figured I'd set up the built-in Python logger as Celery uses it to stream logging messages using the rabbitmq message broker and have this central location receive these and dump them to disk. After that, I have no idea.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Rocko Bonaparte posted:

How might I stream log messages from a celery task to an end user through a REST API? I wanted to centralize these logs in one spot just to know what went on, but it looks like I could go an extra step and start streaming it as it comes in to any users that care. As it stands, I figured I'd set up the built-in Python logger as Celery uses it to stream logging messages using the rabbitmq message broker and have this central location receive these and dump them to disk. After that, I have no idea.

Well, REST isn't really a streaming sort of thing. You'll probably want websockets. Or I don't understand what you're asking and you can forget what I say.

Also, you might be going about it the wrong way. When I think streaming centralized logging, I think ELK or a hosted solution like logentries/papertrail/loggly. Kind of depends on exactly what type of users you have and what your goals here are.

Dominoes
Sep 20, 2007

Hey dudes. Looking to send data from javascript to Django, concluding with a page refresh. I've been sending data like this:

JavaScript code:
    $("#ScheduleTraining").click(function () {
        $.ajax({
            method: "POST",
            headers: {"X-CSRFToken": getCookie('csrftoken')},
            url: "/ground-training/",
            data: {type: 'scheduled', selected: findSelected(), date: $('#dateVal').val()}
        })
I can then parse this (super-mangled data) with python in my view, but can't get the page to refresh after, which I need to do since these are database updates. I think AJAX is the wrong tool, but googling always points me to it. Haven't found an obvious way to make the page refresh, even though the view points to the same info after handling the post it would to load a fresh page.

edit: Looks like I can get a better result by modifying elements with Jquery to look like how they would after a page refresh; should be more responsive, although with more/clumsier code.

Dominoes fucked around with this message at 00:42 on Feb 25, 2017

Data Graham
Dec 28, 2009

📈📊🍪😋



Add a success: callback function in your ajax call, and do a document.location.reload() in it.

Dominoes
Sep 20, 2007

Data Graham posted:

Add a success: callback function in your ajax call, and do a document.location.reload() in it.
Thx homes.

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!

Thermopyle posted:

Well, REST isn't really a streaming sort of thing. You'll probably want websockets. Or I don't understand what you're asking and you can forget what I say.

Also, you might be going about it the wrong way. When I think streaming centralized logging, I think ELK or a hosted solution like logentries/papertrail/loggly. Kind of depends on exactly what type of users you have and what your goals here are.

I was hoping to have them all at a REST URL even if it isn't a REST application under the hood. I don't know if that's worth the effort. I wanted me front end to have a text box that showed all the messages the Celery task was generating--including ones that were generated before the user first connected to it. So if the user looks at the status well after it completed, they'd see all the messages for that task. If they connect midway, it would show what was already written to a file and then receive new messages in real time.

ELK looks like it can be hooked up to Django as a separate app. I could compromise on using a different URL.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Rocko Bonaparte posted:

I was hoping to have them all at a REST URL even if it isn't a REST application under the hood. I don't know if that's worth the effort. I wanted me front end to have a text box that showed all the messages the Celery task was generating--including ones that were generated before the user first connected to it. So if the user looks at the status well after it completed, they'd see all the messages for that task. If they connect midway, it would show what was already written to a file and then receive new messages in real time.

ELK looks like it can be hooked up to Django as a separate app. I could compromise on using a different URL.

Hmm. I think I'd consider using django-channels with websockets.

I think I saw a tutorial for doing something similar come across my rss feeds earlier today, give me a few minutes...

edit: https://realpython.com/blog/python/getting-started-with-django-channels/

Thermopyle fucked around with this message at 00:31 on Mar 1, 2017

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!

Thermopyle posted:

Hmm. I think I'd consider using django-channels with websockets.

I think I saw a tutorial for doing something similar come across my rss feeds earlier today, give me a few minutes...

edit: https://realpython.com/blog/python/getting-started-with-django-channels/

Of course it uses Redis when I already have rabbitmq.

I was thinking about this some more and the most concise way to put it is I want to stream a tail -f to the end user. It looks like websockets is part of that if I want to get that stream going. I have realized I have a few other issues here now too, like how I intend to capture the messages from the different agents to a prescribed file on my host. They're coming in from different machines, and I have to be able to disambiguate them. I figured I would just use RabbitMQ for that since I already have that up for Celery's sake. So I guess like anything else, I'll just have to solve it in parts.

a witch
Jan 12, 2017

django-channels is not very good, for a lot of reasons that have been covered at great length on django-developers.

You'll be better off running websockets in a separate process entirely than trying to make them fit into django. There are still great python options for this like Twisted or asyncio, you can even use continue to use the django ORM with them.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

As-is It's probably bad as the default Django deployment story going forward, but I don't think it's a bad solution for an easy websockets solution.

a witch
Jan 12, 2017

It's usable as an easy solution as long as you're able/willing to run asgi and wsgi in separate processes. Having your http requests go over asgi is a bad time.

Once you've set up the appropriate routing, it's easy enough to replace the asgi half with something better.

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
Given I haven't done a Django deployment yet and am still just trying the site from runserver, you all are scaring the poo poo out of me.

a witch
Jan 12, 2017

Rocko Bonaparte posted:

Given I haven't done a Django deployment yet and am still just trying the site from runserver, you all are scaring the poo poo out of me.

This is exactly the level of traffic and complexity that django-channels handles well fwiw, so it may suit your needs.

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

a witch posted:

django-channels is not very good, for a lot of reasons that have been covered at great length on django-developers.

You'll be better off running websockets in a separate process entirely than trying to make them fit into django. There are still great python options for this like Twisted or asyncio, you can even use continue to use the django ORM with them.

I'm not sure that "not very good" is what the dev list discussions awhile back really concluded. More like "not very good as a default deployment option for django going forward". But I read those discussions while they were on-going so it's been quite awhile...

django-channels seems fine for probably most people wanting to do websockets.

Feral Integral
Jun 6, 2006

YOSPOS

Having trouble with some django migrations. I have this pretty simple baseball app that models a 'player', 'stat', and 'pitch', and one of the fields on the player is
code:
bbref_id = models.CharField(max_length=30, default=fake.first_name())
. To try and work out the problem, I dropped my postgres database, recreated it, deleted the migrations folder and all .pyc files in the app directory and ran:

code:
python manage.py makemigrations baseball

Migrations for 'baseball':
  0001_initial.py:
    - Create model Match
    - Create model Pitch
    - Create model Player
    - Create model Stat
    - Add field batter to pitch
    - Add field pitcher to pitch


which turns out fine. Then I run migrate and it tells me I still have some migrations so I do that:

code:
python manage.py makemigrations

Migrations for 'baseball':
  0002_auto_20170301_2040.py:
    - Alter field bbref_id on player
    - Alter field birth_city on player
    - Alter field birth_country on player
    - Alter field birth_state on player
    - Alter field death_city on player
    - Alter field death_country on player
    - Alter field death_state on player
    - Alter field first_name on player
    - Alter field last_name on player
    - Alter field middle_name on player
    - Alter field player_id on player
    - Alter field retro_id on player

then I try migrate and I get:

code:
python manage.py migrate

...
django.db.utils.ProgrammingError: column "bbref_id" of relation
"baseball_player" does not exist
But in the above makemigrations it says that this field was altered, so why doesn't the column exist? I think I'm suffering from a lack of understanding of what the migration is actually doing, anyone know what I'm loving up here?

Dominoes
Sep 20, 2007

-

Dominoes fucked around with this message at 21:56 on Mar 1, 2017

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Migrations are easy...if you have a solid understanding of databases and a solid understanding of how an ORM works.


In other words, they're not easy.


edit: this post made more sense in relation to the previous post that was removed! I'll leave it as commiseration.

Lumpy
Apr 26, 2002

La! La! La! Laaaa!



College Slice

Feral Integral posted:

Having trouble with some django migrations. I have this pretty simple baseball app that models a 'player', 'stat', and 'pitch', and one of the fields on the player is
code:
bbref_id = models.CharField(max_length=30, default=fake.first_name())
. To try and work out the problem, I dropped my postgres database, recreated it, deleted the migrations folder and all .pyc files in the app directory and ran:

code:
python manage.py makemigrations baseball

Migrations for 'baseball':
  0001_initial.py:
    - Create model Match
    - Create model Pitch
    - Create model Player
    - Create model Stat
    - Add field batter to pitch
    - Add field pitcher to pitch


which turns out fine. Then I run migrate and it tells me I still have some migrations so I do that:

code:
python manage.py makemigrations

Migrations for 'baseball':
  0002_auto_20170301_2040.py:
    - Alter field bbref_id on player
    - Alter field birth_city on player
    - Alter field birth_country on player
    - Alter field birth_state on player
    - Alter field death_city on player
    - Alter field death_country on player
    - Alter field death_state on player
    - Alter field first_name on player
    - Alter field last_name on player
    - Alter field middle_name on player
    - Alter field player_id on player
    - Alter field retro_id on player

then I try migrate and I get:

code:
python manage.py migrate

...
django.db.utils.ProgrammingError: column "bbref_id" of relation
"baseball_player" does not exist
But in the above makemigrations it says that this field was altered, so why doesn't the column exist? I think I'm suffering from a lack of understanding of what the migration is actually doing, anyone know what I'm loving up here?

I am probably wrong, but doing 'makemigrations' on a new DB could be it... just do 'migrate' the first time. "makemigrations" in a nutshell says "create a diff between the existing DB and what the DB would look like with the current Models" So if you run 'makemigrations' on an empty DB, it will create a "migration" from nothing. So don't run 'makemigrations' until you've run 'migrate' for the first time.

The second thing it could be may be related to trying to alter everything due to the function calls in the default??

tl;dr: drop your DB, then run 'migrate' first.


I have never had issues with migrations, but maybe I'm just lucky.

\/\/ Hey, I'm only half as dumb as I think I am!

Lumpy fucked around with this message at 22:19 on Mar 1, 2017

Pumpkin Pirate
Feb 2, 2005
???????
Makemigrations doesn't look at your current DB. It compares the existing migrations to the current state of the model definitions, so it will work fine even if the actual database isn't migrated. I think you're right about the defaults causing the second round of migrations though. Feral Integral, If you want each new object to get a new random name by default, use:
code:
bbref_id = models.CharField(max_length=30, default=fake.first_name)
Your current code is actually calling first_name when the field is defined, and setting whatever is returned at that point as a database level default for all new objects. Then the next time you start up your code, a new default is chosen, and the migration sees that there's a new default, and it tells you that you should make a new migration to reflect that.

Feral Integral
Jun 6, 2006

YOSPOS

Thanks Lumpy and Pumpkin Pirate!!

Yeah it was a combination of not running the 'migrate' command before makemigrations, and then also a stupid goof where I thought the database I was dropping with pgadmin3 was actually being dropped, not just removed from the list of databases..doh. That will teach me to not just use the command line to drop the db (like I usually do..) in the future like a dumbass.

Feral Integral fucked around with this message at 01:10 on Mar 2, 2017

Dominoes
Sep 20, 2007

Thermopyle posted:

edit: this post made more sense in relation to the previous post that was removed! I'll leave it as commiseration.
My workflow's like this:

-Use migrations until they break. Adding tables and fields is usually fine; modifications are a gamble.
-If there's no real data in the db, ie it's early in the dev process, wipe the DB and migrations; start fresh. Always fixes it.
-If there's real data saved, update the DB with SQL from a console for the app that's broken (Ie other apps might still work with migrations). Migrations will never work again for that app, but your website and database will work.

Is there a way to unbreak migrations, once they're broken?

Dominoes fucked around with this message at 18:08 on Mar 2, 2017

Eleeleth
Jun 21, 2009

Damn, that is one suave eel.

Dominoes posted:

-If there's real data saved, update the DB with SQL from a console for the app that's broken (Ie other apps might still work with migrations). Migrations will never work again for that app, but your website and database will work.

Wouldn't it be better to add this to a custom migration with RunSQL? That way you'll at least have a record of what was run in version control.

e: this is also a good way to add indexes that the orm doesn't natively support!

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Tomie knows me posted:

Wouldn't it be better to add this to a custom migration with RunSQL? That way you'll at least have a record of what was run in version control.

e: this is also a good way to add indexes that the orm doesn't natively support!

Yes, this is the right way to do it.


Dominoes posted:

-Use migrations until they break. Adding tables and fields is usually fine; modifications are a gamble.

Also, this makes it sound like migrations breaking is something to be expected. In one sense that's true because we're programmers and as we know, programmers are bad. In another sense it's inaccurate. You just have to think about every model modification and what it's going to mean for how the migration system works. Then migrations don't break.

So, like if you're going to change something = models.CharField() to something = models.IntegerField(), don't just change it and run make a migration.

Add another field something2 = models.IntegerField(null=True, blank=True), create a migration. Then make a blank migration with RunPython to migrate data from something to something2. Then remove something and make a migration. Then rename something2 to something and you're done.

Dominoes
Sep 20, 2007

And then you get errors because django can't find so and so field because DUH YOU DELETED IT!


Root cause is probably I'm not making proper migration files, and relying too much on makemigrations. I feel like this is a problem that could be solved by Django telling you "Whoah I'm not sure what you're doing, please make a migration file", rather than irreversibly* breaking.

* Irreversibly meaning like when a car's totaled; not that it's broken beyond repair, but when it's easier to back everything up and start fresh, or do everything in SQL, than troubleshoot. I feel like I'm not out of line saying this is an unusually counterintuitive process for Python.

Tell me if I'm off-base here: makemigrations is how you change your database, after editing, adding, or removing model fields. Unless your modification is unsuitable for makemigrations... And there's no official guidance about what unsuitable means, and django doesn't warn you, or provide an undo option if it doesn't work. So yes, I expect them to break. And I expect there's room for improvement, ie this will be fixed within a few years.

Dominoes fucked around with this message at 23:14 on Mar 2, 2017

Maluco Marinero
Jan 18, 2001

Damn that's a
fine elephant.

Dominoes posted:

And there's no official guidance about what unsuitable means, and django doesn't warn you, or provide an undo option if it doesn't work. So yes, I expect them to break. And I expect there's room for improvement, ie this will be fixed within a few years.

This is only true if you're not running a quality assurance site and you run the resulting migration on a database blind.

Schema changes can be handled just fine by makemigration. Data migrations are NEVER handled automatically, so if you think you need to alter a field and that alteration requires changes to the data, then you make a data migration and think about it in a flow that doesn't violate the schema while doing so. I have NEVER had problems with schema migrations automatically created, and I've made at least 10-15 Django Projects in the last 3-4 years. It is also possible to roll back migrations as long as they have a backwards behaviour, at which point you can then delete a misconfigured migration and set things up properly again.

Django migrations honestly feel at this point like one of the best solutions out there for this kind of thing, the data migration model works well and there really is no reason to NEED to run raw SQL without being in a migration container. If you DO get in this state, your last resort can be running 'fake' migrations to skip over a particular migration while still letting Django think it has been done.

All of this is documented, and I've very very rarely used something other than makemigrations. That said, we also spend a lot of time early in the project nailing down as much of the data model as possible, so typically our migrations rarely require major remodelling, usually it's adding fields and tables exclusively, which is where Django migrations is easiest to reason about.

Maluco Marinero fucked around with this message at 23:51 on Mar 2, 2017

Data Graham
Dec 28, 2009

📈📊🍪😋



Hey guys. I'm getting myself embroiled in a bit of a philosophical war with a business associate over whether I can use Django for production apps, or (his preference) that we use Java for everything.

His objections to Django are that it's a "toy", that it's hardly better than WordPress from an architectural standpoint, and (probably the biggest sticking point right now) that there's no way to protect secret strings like the database password.

In his ideal world there's no way for a local user on the server to be able to extract those secret strings from the web app, because they're compiled into an executable binary or archive. In Django you have to have your secret strings either sitting in plaintext in the settings.py file, or imported from some other file that's readable by the www user anyway, which means anyone who's able to hijack a login shell of a user who has access to those files would get the keys to the kingdom.

I know this is kind of a wacky irrational position to take and that best practices are such that basically if anyone gets local shell access you should consider your system totally compromised, but that's not good enough for him. I want to see if there's any possible way I can harden a Django/Apache/mod_wsgi installation to his satisfaction.

The best solution I've been able to come up with has been to set the secret strings in the httpd.conf (in the vhost), setting them as environment variables with SetEnv statements, and then importing them into the application with a custom call to get_wsgi_application() in wsgi.py. Theoretically that was good because you could then make your httpd.conf readable only by root and it would pass the variables at startup and be inaccessible to no non-root users; but turns out that's not good enough either because he doesn't want to run apache as root either, for security reasons. I'm back to having the strings in a file readable by the www user, which there doesn't seem to be a way around.

Has anyone been able to figure out an ingenious way around this? I would very very very much like to continue using Django for our projects and not have to use ColdFusion (which he loves because all the sensitive strings are encrypted at the app server level and a local user has no access to it), or some JSP platform which I'm not sure why he thinks would be any more secure because to my knowledge all the JSP frameworks involve storing passwords and such in cleartext just like in Django. I'd rather not fall back on that argument either; I'd want to know if there's some way of locking down Django itself.

PT6A
Jan 5, 2006

Public school teachers are callous dictators who won't lift a finger to stop children from peeing in my plane
I might be totally off-base, but wouldn't a password still be readable as a string in a compiled binary, unless it's purposefully obfuscated by somehow constructing it in code? And, following from that, wouldn't using .pyc files exclusively and making .py files inaccessible to the www user provide an equivalent level of security?

EDIT: Which, to be sure, is "not a lot of security" if someone has access to the server already.

PT6A fucked around with this message at 17:07 on Mar 4, 2017

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

Data Graham posted:

Hey guys. I'm getting myself embroiled in a bit of a philosophical war with a business associate over whether I can use Django for production apps, or (his preference) that we use Java for everything.

His objections to Django are that it's a "toy", that it's hardly better than WordPress from an architectural standpoint, and (probably the biggest sticking point right now) that there's no way to protect secret strings like the database password.

In his ideal world there's no way for a local user on the server to be able to extract those secret strings from the web app, because they're compiled into an executable binary or archive. In Django you have to have your secret strings either sitting in plaintext in the settings.py file, or imported from some other file that's readable by the www user anyway, which means anyone who's able to hijack a login shell of a user who has access to those files would get the keys to the kingdom.

I know this is kind of a wacky irrational position to take and that best practices are such that basically if anyone gets local shell access you should consider your system totally compromised, but that's not good enough for him. I want to see if there's any possible way I can harden a Django/Apache/mod_wsgi installation to his satisfaction.

The best solution I've been able to come up with has been to set the secret strings in the httpd.conf (in the vhost), setting them as environment variables with SetEnv statements, and then importing them into the application with a custom call to get_wsgi_application() in wsgi.py. Theoretically that was good because you could then make your httpd.conf readable only by root and it would pass the variables at startup and be inaccessible to no non-root users; but turns out that's not good enough either because he doesn't want to run apache as root either, for security reasons. I'm back to having the strings in a file readable by the www user, which there doesn't seem to be a way around.

Has anyone been able to figure out an ingenious way around this? I would very very very much like to continue using Django for our projects and not have to use ColdFusion (which he loves because all the sensitive strings are encrypted at the app server level and a local user has no access to it), or some JSP platform which I'm not sure why he thinks would be any more secure because to my knowledge all the JSP frameworks involve storing passwords and such in cleartext just like in Django. I'd rather not fall back on that argument either; I'd want to know if there's some way of locking down Django itself.

You business associate is a dummy.

Anyway, you're probably wanting to google "secrets management". The most well-known product is probably Vault by HashiCorp. I'd also ask this question in maybe the General Programming thread and/or the devops thread.

Also, I don't know how ColdFusion works, but I don't know how it's possible that your application can have access to the key, but not a local user.

epswing
Nov 4, 2003

Soiled Meat

I get the feeling that, even if you convince your associate of a solution to the secret-hiding problem, he/she is just going to find more "reasons" to use the thing they've already decided they want to use.

Data Graham
Dec 28, 2009

📈📊🍪😋



epalm posted:

I get the feeling that, even if you convince your associate of a solution to the secret-hiding problem, he/she is just going to find more "reasons" to use the thing they've already decided they want to use.

No doubt.

It's been a long slog, just getting to the point where I'm able to get Django a fair hearing. I'm sure the only real answer is "just keep having a good track record for another decade or two".

(He honestly believes in security through obscurity, is the thing. If an encryption routine is hidden in an imported script fragment called "snrbpt.cfm" instead of done inline in the main file, that to him is enough of a pain in the rear end to a potential intruder for him to get bored and give up or something.)

I'll see if Vault can do anything for me, thanks.

Data Graham fucked around with this message at 00:02 on Mar 5, 2017

Adbot
ADBOT LOVES YOU

Rocko Bonaparte
Mar 12, 2002

Every day is Friday!
I started playing with WebSockets in Django based on recommendations in the thread. I started off the chat tutorial and was able to shovel it into my development environment as a separate app, which is probably actually a good thing given it is doing something quite peculiar.

I am trying to figure out how to shove the WebSocket up to something tailing my log files. I saw that the channel API dragged in Twisted, so I figured I must have some kind of reactor running or whatever on which I can plug in coroutines. I found this semi-popular FollowTail source code that people like to reference to tail a file in Twisted. I figured I would just fuse that up to my worker and it would trickle the poo poo down. No go. I am assuming the reactor isn't actually running or something; I manually echoed some test stuff to append to my dummy file and nothing came down the pipe to my browser. I know I get into ws_message, and I know FollowTail is constructed and start() is called.

Here is what my consumer looked like:
code:
import os
from django.conf import settings

from logtail.FollowTail import FollowTail


def ws_message(message):
    log_file_name = message.content['text']
    log_path = os.path.join(settings.GET_CONSUMER_LOG_DIR(), log_file_name)

    if os.path.exists(log_path):
        tailer = FollowTail(log_path)

        def line_received(line):
            message.reply_channel.send({
                "text": line,
            })

        tailer.lineReceived(line_received)
        tailer.start()

    else:
        message.reply_channel.send({
            "text": "File not found: %s" % log_file_name,
        })
Any ideas?

Note: I didn't think it was up to me to start the reactor so I am assuming that the Django channels support is running it already. I suspect that is the main problem.

Edit: I wasn't attaching the callback correctly. The reactor wasn't running. Also, that implementation doesn't even read anything. It's hosed up.

Edit Edit: Got it working, but then realized it's just polling anyways. I mean, it's doing it in a polite way, but whatever. I guess there isn't really a good (standardized) way to get notified about file change messages across OSes that plays nice with all this stuff.

Rocko Bonaparte fucked around with this message at 19:39 on Mar 5, 2017

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply