Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

PYPI is flaky/down in case anyone is wondering why all their builds are failing.

# ? Jan 12, 2021 16:58

Adbot: ADBOT LOVES YOU

# ? May 26, 2024 12:58

Wallet: Jun 19, 2006

Famethrowa posted:

Thank you for the feedback everyone. I was working on some other classes so this went on backburner. This is what I'm focusing on right now, just need to grok how to offset in a way that will output correctly for each key/cipher length tested. Thinking perhaps I hardcode a brute-force function to test a set of possible keys rather then use an input key.

The "key" here is just the line length you want to try, isn't it? If the point is to learn python you may want to do this with loops instead of loving with comprehensions as it will make it easier to follow the indexing, but it's probably worth writing a function to test a solution with a given key instead of brute forcing.

For any given range of possible keys you can discard a significant number as impossible in an unmodified scytale using the modulo operator (%) which will make your life easier.

As far as the valid offsets for slicing it's probably worth reading up on the floor division (//) operator if you haven't.

Wallet fucked around with this message at 17:49 on Jan 12, 2021

# ? Jan 12, 2021 17:46

Mursupitsku: Sep 12, 2011

Mursupitsku posted:

After running some tests it seems that the classifier is the only thing taking a significant amount of time to run. I haven't yet tested if dropping features would increase the performance. In any case there is only 17 features total and my gut tells I cant really drop any of them.

Would just throwing more computing power at it work? Atm I'm running it on an semi old laptop.

What are my options to optimize the classifier itself?

So after some more research it seems that using

code:

sklearn.calibration.CalibratedClassifierCV

made the

code:

predict_proba()

take a lot of time. Removing the calibration made the code run at least 10 times faster.

However without the calibration the classifier skews the probabilities towards 0 and 1. Ill have to do some back testing to see if the results of the monte carlo simulation are acceptable without the calibration. Anyone have any insight into this?

# ? Jan 13, 2021 07:09

Sulla Faex: May 14, 2010; No man ever did me so much good, or enemy so much harm, but I repaid him with ENDLESS SHITPOSTING

Wallet posted:

The "key" here is just the line length you want to try, isn't it? If the point is to learn python you may want to do this with loops instead of loving with comprehensions as it will make it easier to follow the indexing, but it's probably worth writing a function to test a solution with a given key instead of brute forcing.

For any given range of possible keys you can discard a significant number as impossible in an unmodified scytale using the modulo operator (%) which will make your life easier.

As far as the valid offsets for slicing it's probably worth reading up on the floor division (//) operator if you haven't.

I'm not the OP but I am also teaching myself python at home and I thought I'd try this with a variable rod width (i.e. not just 4, as it is here).. Decoding is easy, but encoding I handled in a way that I'm not sure matches what you're hinting at.

I don't know what the rules are about homework etc etc, I have code I could share via PM if it's a problem posting directly here

Sulla Faex fucked around with this message at 16:22 on Jan 14, 2021

# ? Jan 14, 2021 16:10

Wallet: Jun 19, 2006

Sulla Faex posted:

I'm not the OP but I am also teaching myself python at home and I thought I'd try this with a variable rod width (i.e. not just 4, as it is here).. Decoding is easy, but encoding I handled in a way that I'm not sure matches what you're hinting at.

I don't know what the rules are about homework etc etc, I have code I could share via PM if it's a problem posting directly here

I think you're fine�people just like to let people solve problems for themselves if they want to instead of giving them the answer. I'm guessing the assignment was already due by now.

# ? Jan 15, 2021 23:43

Sulla Faex: May 14, 2010; No man ever did me so much good, or enemy so much harm, but I repaid him with ENDLESS SHITPOSTING

That's what I was thinking, but I wasn't sure so I thought I'd play it safe.

This is what I have: https://pastebin.com/PTEnQLMZ

It seems to work and it makes sense to me but that doesn't really mean much at the end of the day.

To solve the division issue I just pad the original string with empty spaces, since that's what would happen on the physical parchment anyway.. but I'm not sure if that's what you were hinting at?

# ? Jan 16, 2021 12:57

Wallet: Jun 19, 2006

Sulla Faex posted:

To solve the division issue I just pad the original string with empty spaces, since that's what would happen on the physical parchment anyway.. but I'm not sure if that's what you were hinting at?

Padding is probably the sane solution, now that you mention it. I was thinking about something like this (which could be implemented more elegantly, but whatever):

Python code:

def decode(original_text: str, max_step: int = 10) -> dict:
    length = len(original_text)
    possible_solutions = {}
    for s in range(2, max_step + 1):
        text = original_text
        report_key = str(s)
        if not (remainder := length % s) == 0:
            partial_positions = [(i * (length // s)) - 1 for i in range(1, remainder+1)]
            partial_line = "".join([text[i] for i in partial_positions])
            text = [c for i, c in enumerate(text) if i not in partial_positions]
            report_key = f'incomplete {s + 1}'
        else:
            partial_line = None
        solve = ["".join(text[i::s]) for i in range(s)]
        if partial_line:
            solve.append(partial_line)
        possible_solutions[report_key] = solve
    return possible_solutions

divisible = 'Iryyatbhmvaehedlurlp'
indivisible = 'iryymatbhemvaehedlurlp'

assert decode(divisible)['4'] == ['Iamhu', 'rtver', 'ybadl', 'yhelp']
assert decode(indivisible)['incomplete 5'] == ['iamhu', 'rtver', 'ybadl', 'yhelp', 'me']

Which (if I did it correctly) will solve a scytale with a number of characters not cleanly divisible by the key without padding.

# ? Jan 17, 2021 02:18

Sulla Faex: May 14, 2010; No man ever did me so much good, or enemy so much harm, but I repaid him with ENDLESS SHITPOSTING

Ahh ok, I was thinking of doing something similar at first but the more I thought about it, the more I figured that padding was a cleaner representation of what was happening in the real world scytale. I'm glad to see I didn't at least fall into any obvious traps

Thank you for typing it up and posting!

# ? Jan 17, 2021 10:24

Famethrowa: Oct 5, 2012

Well, I may have ended up doing this by hand (since coding wasn't a requirement) but I'm glad I could inspire!

My attempts were so spectacularly off from what you all managed. I think I was far too green with Python to understand all the options available to me. I don't think I fully understood utilizing padding and modulus. I think I need a lot more time with the language.

# ? Jan 19, 2021 03:51

punk rebel ecks: Dec 11, 2010; A shitty post? This calls for a dance of deduction.

So this is the first time I'm really trying to learn how to code a major language and I'm finding Python fun.

# ? Jan 22, 2021 09:22

Mursupitsku: Sep 12, 2011

code:

import concurrent.futures
from XGB import simulate_match
import time

matchesToSimulate = [match1, match2, match3, match4, match5, match6, match7, match8, match9, match10, match11, match12, match13,
			match14, match15, match16, match17, match18, match1, match19, match20, match21, match22, match23, match24, match25, match26]



def main():
    start = time.time()
    logLosses = []
    correctPredictions = []
    with concurrent.futures.ProcessPoolExecutor() as executor:
        homeWinPercentages = executor.map(simulate_match, matchesToSimulate)
        for hwp in homeWinPercentages:
            logLosses.append(hwp[2])
            correctPredictions.append(hwp[1])
            
    print(correctPredictions)
    rawAccuracy = sum(correctPredictions) / len(correctPredictions)
    print("Model raw accuracy: " + str(rawAccuracy))
    LogarithmicLoss = sum(logLosses) * (-1)/ len(logLosses)
    print("Model Log Loss: " + str(LogarithmicLoss))
    end = time.time()
    print('It took ' + str(end - start) + ' seconds to run.')
    
    
if __name__ == '__main__':
    main()

Running the above code on a 4cpu laptop took 2 hours and 9 minutes. I then tried to run the same code on a 12 cpu workstation laptop and it took 1 hour and 46 minutes to run. I was expecting a lot bigger decrease in runtime. Running the same simulations on the laptop without concurrent.futures and with jupyter notebook takes 6 hours and 46 minutes. Why am I not seeing a bigger decrease in runtime when running this on the 12cpu workstations laptop compared to the 4 cpu laptop?

Specs of the laptops are:
Workstation:
Operating System: Windows 10 Enterprise 64-bit (10.0, Build 18363) (18362.19h1_release.190318-1202)
Processor: Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz (12 CPUs), ~2.6GHz
Memory: 65536MB RAM

Laptop:
Operating System: Windows 10 Home 64-bit (10.0, Build 18363) (18362.19h1_release.190318-1202)
Processor: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz (4 CPUs), ~2.9GHz
Memory: 8192MB RAM

Mursupitsku fucked around with this message at 12:27 on Jan 25, 2021

# ? Jan 25, 2021 12:24

Loving Africa Chaps: Dec 3, 2007; We had not left it yet, but when I would wake in the night, I would lie, listening, homesick for it already.

Have you looked at how many cores the script is actually using?

In any case parallel computing often has rapidly diminishing returns when I've tried it with python.

# ? Jan 25, 2021 12:27

Mursupitsku: Sep 12, 2011

Loving Africa Chaps posted:

Have you looked at how many cores the script is actually using?

In any case parallel computing often has rapidly diminishing returns when I've tried it with python.

Task manager shows that there are 12 individual python processes running. Does this mean that 12 cpus are actually used? I'm not sure about this.

# ? Jan 25, 2021 12:31

QuarkJets: Sep 8, 2008

It's really hard to say without having hands on the hardware and running a lot more tests. But here are some thoughts off the top of my head:

Those CPUs have half as many physical cores, and depending on what you're using them for you may be overloading them. Hyperthreading allows each core to work on 2 tasks simultaneously, but that does not mean that the core is getting twice as much work done. This is a problem-specific matter

There are also going to be differences in per-core performance between the two CPUs, which you can't do anything about

The systems could also have differences in resource competition, normally in longer-running processes you can observe variations in performance due to all kinds of reasons. Often benchmarking entails taking several measurements to smooth out fluctuations; if the system decided to kick off some critical process in the middle of your computation time then everything will get bogged down because you're trying to use literally all of the cores. Windows is notorious for having poo poo like this happen.

Then there are the external resources that may be used by those jobs, which can bottleneck your performance. As an extreme hypothetical, if each process is just doing very light analysis on 100 GB of data from an old platter drive then the hard drive read speed imposes a performance bottleneck and adding more simultaneously-running processes doesn't help

I'm assuming that match1, match2, etc. are pretty small objects and probably aren't to blame

# ? Jan 25, 2021 13:02

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

I'm wondering how other people might be dealing with a situation of scratchwork classes and rigid type checking. I have some classes that are getting progressively fed information so their fields start out optional. When they are in-use, these will be filled.

Type checking hates this if I don't null check everything once the instance is in-use. One of the situations was a good candidate to use a builder because there was a lot of logic associated with generating defaults. However, another one is just some scratchwork.

So right now I'm trying a dataclass that has the fields optional while it's being internally messed around, but it basically returns a version of itself with fields not optional and not null. If I add a property, I will have to remember to put it in both places. I'm not sure about doing reflection stuff to auto-populate because I have to specify the typing information. For as much as I'm using it, that's fine, but it smells a little bit. Has anybody else worked through a problem like this before?

These containers are areas where people love to put the wrong thing in the wrong spot so I specifically want type checking to get involved for them.

# ? Jan 27, 2021 01:44

mystes: May 31, 2006

Oh I thought this was the .net thread for some reason but I suspect that getting static type checking for that situation will be difficult in python as well. The only real alternative I can think of is code generation but that's pretty ugly as well.

mystes fucked around with this message at 02:32 on Jan 27, 2021

# ? Jan 27, 2021 02:10

Rocko Bonaparte: Mar 12, 2002; Every day is Friday!

mystes posted:

Oh I thought this was the .net thread for some reason but I suspect that getting static type checking for that situation will be difficult in python as well. The only real alternative I can think of is code generation but that's pretty ugly as well.

Yeah fair but can you entertain me with more about what you mean with code generation in this case? I have a notion and it's not what I want to do but I'm up for knowing shenanigans for Python internals reasons.

# ? Jan 27, 2021 15:53

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

Question about web scraping. I'm using Selenium to try to log in to a website. If I simply open the site in the ChromeDriver/EdgeDriver version of the browser and manually try to log in, it attempts and then auto logs out with the URL 'thedomain/Account/Login?ReturnUrl=%2Faccount%2Flogout'. Does anyone have any suggestion on what might be causing this and ways around it? Alternatives to Selenium to try? There's a bunch of information that I'd love to scrape but if it can't login then we have issues!

# ? Jan 27, 2021 16:03

mystes: May 31, 2006

Rocko Bonaparte posted:

Yeah fair but can you entertain me with more about what you mean with code generation in this case? I have a notion and it's not what I want to do but I'm up for knowing shenanigans for Python internals reasons.

I think you would literally just have to make a script that outputs python code that you run whenever you need to update the classes, because python doesn't have any built in way of doing this automatically and trying to parse/modify your existing code would just be even more pointlessly complicated, so I probably wouldn't recommend it. (I really just mentioned it because other languages have various ways of doing things like this that are more integrated into the language like c# source generators, or template haskell for haskell and so code generation would be a more obvious solution / less ugly in those languages.)

I guess if you're using mypy you could theoretically make a plugin for it so you could typecheck some sort of wrapper that would generate versions of classes with optional members or something wacky like that, but it seems like way too much work and I assume that wouldn't work with pycharm, etc. anyway.

Sad Panda posted:

Question about web scraping. I'm using Selenium to try to log in to a website. If I simply open the site in the ChromeDriver/EdgeDriver version of the browser and manually try to log in, it attempts and then auto logs out with the URL 'thedomain/Account/Login?ReturnUrl=%2Faccount%2Flogout'. Does anyone have any suggestion on what might be causing this and ways around it? Alternatives to Selenium to try? There's a bunch of information that I'd love to scrape but if it can't login then we have issues!

You mean that you're interactively doing the exact same thing you would do in a normal browser, not using any code or anything, right? Do you know if it's an http redirect or is there javascript that's logging you out? If the former, maybe start by looking at the requests/responses to see any differences in your headers and try to fix them to match a normal browser.

If it's javascript there may be some selenium detection thing going on which you may have to override somehow..

mystes fucked around with this message at 16:20 on Jan 27, 2021

# ? Jan 27, 2021 16:04

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

mystes posted:

You mean that you're interactively doing the exact same thing you would do in a normal browser, not using any code or anything, right? Do you know if it's an http redirect or is there javascript that's logging you out? If the former, maybe start by looking at the requests/responses to see any differences in your headers and try to fix them to match a normal browser.

If it's javascript there may be some selenium detection thing going on which you may have to override somehow..

I wrote code to login and it had this problem so I'm now just using Selenium to open an instance of the browser. I then manually type in username + password and press login. It flashes to show it logs in and then seems to automatically log me out. Any advice on determining if it is a redirect or JS? Not sure how to look at requests/responses. Never had websites that kick me out like this before.

edit - tested with a bad username/password and it gives a login failed error so it is after authenticating me.

Sad Panda fucked around with this message at 16:37 on Jan 27, 2021

# ? Jan 27, 2021 16:35

mystes: May 31, 2006

Sad Panda posted:

I wrote code to login and it had this problem so I'm now just using Selenium to open an instance of the browser. I then manually type in username + password and press login. It flashes to show it logs in and then seems to automatically log me out. Any advice on determining if it is a redirect or JS? Not sure how to look at requests/responses. Never had websites that kick me out like this before.

edit - tested with a bad username/password and it gives a login failed error so it is after authenticating me.

Use the network tab of the developer tools in the browser.

# ? Jan 27, 2021 16:43

Phobeste: Apr 9, 2006; never, like, count out Touchdown Tom, man

mystes posted:

I guess if you're using mypy you could theoretically make a plugin for it so you could typecheck some sort of wrapper that would generate versions of classes with optional members or something wacky like that, but it seems like way too much work and I assume that wouldn't work with pycharm, etc. anyway.

Yeah this is pretty much the only way to do it if you want static typechecking. Ironically if all you want is runtime typechecking I'm pretty sure you could do it with dynamic classes, but mypy does only AST level evaluation so it can't see it.

# ? Jan 27, 2021 16:46

Dr Subterfuge: Aug 31, 2005; TIME TO ROC N' ROLL

Sad Panda posted:

I wrote code to login and it had this problem so I'm now just using Selenium to open an instance of the browser. I then manually type in username + password and press login. It flashes to show it logs in and then seems to automatically log me out. Any advice on determining if it is a redirect or JS? Not sure how to look at requests/responses. Never had websites that kick me out like this before.

edit - tested with a bad username/password and it gives a login failed error so it is after authenticating me.

Possibly your account has been flagged. Does it still kick you if you do the whole process manually? What if you automate it with a new account?

# ? Jan 27, 2021 16:52

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

mystes posted:

Use the network tab of the developer tools in the browser.

Not exactly sure what I'm looking for.

The list of names is ..

code:

Login?ReturnUrl=%2Faccount%2Flogout
css?family=Poppins:300,400,500,600,700
plugins.bundle.css?v=whacpO8gO-MNYd4aGkbHh3vu39CVDVGZDpjGJkfvtCQ
style.bundle.css?v=WMsMRjfd6Rkd7mps5JHHOW4tutMZjCjdE_RhO-NF5Vw
light.css?v=j1ZYFsOCxl6ZDYoefg-JEB9gVNj8BEj3ayvp1xygqNg
light.css?v=Tz51Yr0MQmX-EtPADFVfsCsHRDSUlVQhEi2eUoL7w6o
cookieconsent.min.css?v=inJzCCygaRjJq7ylfc9_UEuHTXJKe1nR2JPhekUm5J4
smbannerlogo.png
plugins.bundle.js?v=adXeC0dmDs1jgX2hAPWZDmUbgt6lzn6S38YRgNrQRAA
scripts.bundle.js?v=GAk1ChNCkkQVf_hpRVohiwyVjhpWi3_n6IoOAcJfCX0
site.min.js?v=jSKxiFYwh3pHBHNN6TtH_tnDSbCnHAyQfLTLkXJr9L8
jquery.validate.min.js
jquery.validate.unobtrusive.min.js
customvalidation.min.js?v=g4Q3UT1vJRIr_tqNd_y8F8voH7xIRYhj03q4l_GAaDc
account.login.min.js?v=L2tAkIuJff8wJbiz2TJGb6GuY9o16m_mUWOehYAFe7A
cookieconsent.min.js?v=0VbRdXiHPQdSNQIcnaIy7xoZ7c70GJ8ugW7v34eZ8FA
ai.2.min.js

with ai.2.min.js being red. Seems to be the 'Application Insights JavaScript SDK 2.5.11' so I'm guessing some JS detection.

# ? Jan 27, 2021 17:01

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

Dr Subterfuge posted:

Possibly your account has been flagged. Does it still kick you if you do the whole process manually? What if you automate it with a new account?

I can login just fine if I open up a browser as normal. This is only if done in Selenium.

Sad Panda fucked around with this message at 17:22 on Jan 27, 2021

# ? Jan 27, 2021 17:01

mystes: May 31, 2006

Sad Panda posted:

Not exactly sure what I'm looking for.

The list of names is ..

code:

Login?ReturnUrl=%2Faccount%2Flogout
css?family=Poppins:300,400,500,600,700
plugins.bundle.css?v=whacpO8gO-MNYd4aGkbHh3vu39CVDVGZDpjGJkfvtCQ
style.bundle.css?v=WMsMRjfd6Rkd7mps5JHHOW4tutMZjCjdE_RhO-NF5Vw
light.css?v=j1ZYFsOCxl6ZDYoefg-JEB9gVNj8BEj3ayvp1xygqNg
light.css?v=Tz51Yr0MQmX-EtPADFVfsCsHRDSUlVQhEi2eUoL7w6o
cookieconsent.min.css?v=inJzCCygaRjJq7ylfc9_UEuHTXJKe1nR2JPhekUm5J4
smbannerlogo.png
plugins.bundle.js?v=adXeC0dmDs1jgX2hAPWZDmUbgt6lzn6S38YRgNrQRAA
scripts.bundle.js?v=GAk1ChNCkkQVf_hpRVohiwyVjhpWi3_n6IoOAcJfCX0
site.min.js?v=jSKxiFYwh3pHBHNN6TtH_tnDSbCnHAyQfLTLkXJr9L8
jquery.validate.min.js
jquery.validate.unobtrusive.min.js
customvalidation.min.js?v=g4Q3UT1vJRIr_tqNd_y8F8voH7xIRYhj03q4l_GAaDc
account.login.min.js?v=L2tAkIuJff8wJbiz2TJGb6GuY9o16m_mUWOehYAFe7A
cookieconsent.min.js?v=0VbRdXiHPQdSNQIcnaIy7xoZ7c70GJ8ugW7v34eZ8FA
ai.2.min.js

with ai.2.min.js being red. Seems to be the 'Application Insights JavaScript SDK 2.5.11' so I'm guessing some JS detection.

"Login?ReturnUrl=%2Faccount%2Flogout" sounds like the logout page is being passed to the login page as the page to redirect to after log in. Are you trying to access the logout page at the beginning as the first page you access in order to get the login page to appear? Otherwise maybe figure out why this is being sent when it's not when you use a normal browser.

Or just try manually opening the login page without that returnurl parameter.

mystes fucked around with this message at 17:17 on Jan 27, 2021

# ? Jan 27, 2021 17:14

Sad Panda: Sep 22, 2004; I'm a Sad Panda.

mystes posted:

"Login?ReturnUrl=%2Faccount%2Flogout" sounds like the logout page is being passed to the login page as the page to redirect to after log in. Are you trying to access the logout page at the beginning as the first page you access in order to get the login page to appear? Otherwise maybe figure out why this is being sent when it's not when you use a normal browser.

Or just try manually opening the login page without that returnurl parameter.

Thanks for that. I used it and googled a bit. Stack Overflow had some discussions and suggestions of flags to send. I've combined that with a random-user-agent plugin and it seems to have got me in. Working out how to scrape what I want rather than logging in again to risk being logged out.

code:

chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument("--disable-blink-features")
chrome_options.add_argument("--disable-blink-features=AutomationControlled")

# ? Jan 27, 2021 17:23

Dominoes: Sep 20, 2007

Rocko Bonaparte posted:

I'm wondering how other people might be dealing with a situation of scratchwork classes and rigid type checking. I have some classes that are getting progressively fed information so their fields start out optional. When they are in-use, these will be filled.

Type checking hates this if I don't null check everything once the instance is in-use. One of the situations was a good candidate to use a builder because there was a lot of logic associated with generating defaults. However, another one is just some scratchwork.

So right now I'm trying a dataclass that has the fields optional while it's being internally messed around, but it basically returns a version of itself with fields not optional and not null. If I add a property, I will have to remember to put it in both places. I'm not sure about doing reflection stuff to auto-populate because I have to specify the typing information. For as much as I'm using it, that's fine, but it smells a little bit. Has anybody else worked through a problem like this before?

These containers are areas where people love to put the wrong thing in the wrong spot so I specifically want type checking to get involved for them.

Whenever you use one of the optional values, explicitly handle both the null and non-null case. Verbose, but robust.

Your builder or two-class system could work, if you confirm your workflow change states from nullable to non-nullable cleanly.

# ? Jan 27, 2021 18:23

mystes: May 31, 2006

By the way, it's a bit off-topic but I think that typescript is currently the only mainstream language that has a powerful enough type system to do what Rocko Bonaparte wants directly right now, and in fact this is the exact example they give for the "mapped types" feature in the documentation: (https://www.typescriptlang.org/docs/handbook/release-notes/typescript-2-1.html#mapped-types)

quote:

Mapped Types

One common task is to take an existing type and make each of its properties entirely optional. Let�s say we have a Person:

interface Person {
name: string;
age: number;
location: string;
}

A partial version of it would be:

interface PartialPerson {
name?: string;
age?: number;
location?: string;
}

with Mapped types, PartialPerson can be written as a generalized transformation on the type Person as:

type Partial<T> = {
[P in keyof T]?: T[P];
};

type PartialPerson = Partial<Person>;

I really hope that typescript helps popularize this type of stuff so it gets added to more languages.

# ? Jan 27, 2021 20:18

mr_package: Jun 13, 2000

I'm deploying Flask via Gunicorn/nginx. Right now this all runs under an actual user (CentOS) e.g. everything is running in ~/specialuser/app1 ~/specialuser/app2 for that user. I am assuming this is incorrect and I should have deployed as some kind of system user, with permissions to read/write state in /var/lib/appname/ and things like that. Is there a good guide to doing this, or understanding it better? I want to take a look and see if there are tradeoffs, understand how it should have been done, what it will take to migrate, etc.

Right now configurations and other things are in dot-files and dot-dirs in the home dir for this user and I want to migrate over to the "right" way. e.g. nginx runs as nginx:nginx, is it correct to have a similar separate user:group to deploy my Python web apps? Should each app be its own user, or should I create a group for webapps e.g. should it be app1:webapps app2:webapps, things like that. I've never had to deal with that aspect of Linux before.

# ? Jan 27, 2021 23:20

Wallet: Jun 19, 2006

mr_package posted:

I'm deploying Flask via Gunicorn/nginx. Right now this all runs under an actual user (CentOS) e.g. everything is running in ~/specialuser/app1 ~/specialuser/app2 for that user. I am assuming this is incorrect and I should have deployed as some kind of system user, with permissions to read/write state in /var/lib/appname/ and things like that. Is there a good guide to doing this, or understanding it better? I want to take a look and see if there are tradeoffs, understand how it should have been done, what it will take to migrate, etc.

Unless you're doing this because you want to dig into all of this stuff, the easy way to handle deploying your Flask apps is usually going to be containerizing them using Docker. There's probably a shitload of images for that purpose but this uwsgi/nginx image and this meinheld/gunicorn image are very easy to set up.

# ? Jan 28, 2021 01:25

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Wallet posted:

Unless you're doing this because you want to dig into all of this stuff, the easy way to handle deploying your Flask apps is usually going to be containerizing them using Docker. There's probably a shitload of images for that purpose but this uwsgi/nginx image and this meinheld/gunicorn image are very easy to set up.

If you wanna be lazy AND cheap, deploy your Flask app with Zappa to AWS lambda. gently caress servers.

# ? Jan 28, 2021 09:06

OnceIWasAnOstrich: Jul 22, 2006

CarForumPoster posted:

If you wanna be lazy AND cheap, deploy your Flask app with Zappa to AWS lambda. gently caress servers.

Because a have a ton of free GCP credits I do this with Cloud Run and its been working great for a quickie Flask app this past week I didn't want to bother a server with. It feels a bit more like a natural fit for Flask containers than Lambda with the configurable per-container concurrency, although Lambda can probably do that too by now.

OnceIWasAnOstrich fucked around with this message at 16:19 on Jan 28, 2021

# ? Jan 28, 2021 16:16

mr_package: Jun 13, 2000

I know it's old school but yeah, I want to run it as a boring old Linux service style thing. I do not use Docker so there's a learning curve either way, but given that I have everything already working fine (it's been deployed ~24 months like this) I was hoping there'd be an easy way to package it up in the standard Linux way. Whatever that means.

Maybe I need to find something not Python-specific, but more about how to deploy a Linux service in general, e.g. when you install nginx on CentOS it's user/group nginx/nginx. I'm assuming it's good practice to do same with individual apps if they are network services like Flask. But maybe it gets tricky when it's actually a shared hosting environment, and creating regular users is the right thing here. I remember back in the web hosting days a long time ago it was always like that: virtualhosted users would get their own dir where Apache would serve files from even if they weren't allowed on the system via ssh.

# ? Jan 28, 2021 19:36

mystes: May 31, 2006

mr_package posted:

I know it's old school but yeah, I want to run it as a boring old Linux service style thing. I do not use Docker so there's a learning curve either way, but given that I have everything already working fine (it's been deployed ~24 months like this) I was hoping there'd be an easy way to package it up in the standard Linux way. Whatever that means.

Maybe I need to find something not Python-specific, but more about how to deploy a Linux service in general, e.g. when you install nginx on CentOS it's user/group nginx/nginx. I'm assuming it's good practice to do same with individual apps if they are network services like Flask. But maybe it gets tricky when it's actually a shared hosting environment, and creating regular users is the right thing here. I remember back in the web hosting days a long time ago it was always like that: virtualhosted users would get their own dir where Apache would serve files from even if they weren't allowed on the system via ssh.

If you just want to run a service in the normal linux way that's easy with systemd now. Just create a service file, create a user for it, and enable it.

# ? Jan 28, 2021 19:47

Macichne Leainig: Jul 26, 2012; by VG

mystes posted:

If you just want to run a service in the normal linux way that's easy with systemd.

Basically this, here's an example systemd service file I use for running a Python service. Maybe it's useful.

code:

[Unit]
Description=Python Web API

[Service]
ExecStart=/usr/bin/python3 /home/username/api/main.py
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=pyapi
Restart=on-failure
User=username

[Install]
WantedBy=default.target

Basically implemented this to reboot the API if it ever crashed, has worked wonderfully. I did not bother creating a new user, but I am the only person with access to this environment anyway.

# ? Jan 28, 2021 19:50

mystes: May 31, 2006

Btw, if you create a user for the service just use adduser --system so it can't be used to log in.

# ? Jan 28, 2021 19:54

Pie in the Sky: Apr 16, 2009; whoops here we go again

Wallet posted:

Padding is probably the sane solution, now that you mention it. I was thinking about something like this (which could be implemented more elegantly, but whatever):

Python code:

def decode(original_text: str, max_step: int = 10) -> dict:
    length = len(original_text)
    possible_solutions = {}
    for s in range(2, max_step + 1):
        text = original_text
        report_key = str(s)
        if not (remainder := length % s) == 0:
            partial_positions = [(i * (length // s)) - 1 for i in range(1, remainder+1)]
            partial_line = "".join([text[i] for i in partial_positions])
            text = [c for i, c in enumerate(text) if i not in partial_positions]
            report_key = f'incomplete {s + 1}'
        else:
            partial_line = None
        solve = ["".join(text[i::s]) for i in range(s)]
        if partial_line:
            solve.append(partial_line)
        possible_solutions[report_key] = solve
    return possible_solutions

divisible = 'Iryyatbhmvaehedlurlp'
indivisible = 'iryymatbhemvaehedlurlp'

assert decode(divisible)['4'] == ['Iamhu', 'rtver', 'ybadl', 'yhelp']
assert decode(indivisible)['incomplete 5'] == ['iamhu', 'rtver', 'ybadl', 'yhelp', 'me']

Which (if I did it correctly) will solve a scytale with a number of characters not cleanly divisible by the key without padding.

This may be a silly question, but I've never seen this sort of inline type declaration in Python. Is this for a linter? Or docs? Something else?

# ? Jan 28, 2021 19:55

mystes: May 31, 2006

Pie in the Sky posted:

This may be a silly question, but I've never seen this sort of inline type declaration in Python. Is this for a linter? Or docs? Something else?

It's part of the python standard now but the normal python interpreter doesn't check it. You can use it with mypy or an ide for type checking.

https://docs.python.org/3/library/typing.html

# ? Jan 28, 2021 19:57

Adbot: ADBOT LOVES YOU

# ? May 26, 2024 12:58

susan b buffering: Nov 14, 2016

mystes posted:

It's part of the python standard now but the normal python interpreter doesn't check it. You can use it with mypy or an ide for type checking.

https://docs.python.org/3/library/typing.html

Also used for dataclasses and namedtuples(via typing.NamedTuple).

I think attrs also supports using them in class definitions now.

# ? Jan 28, 2021 20:02

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »