Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Jose Cuervo posted:

What does this process look like? Get everyone in the class to download Anaconda on day one of the class, then...?

Yep, this is what I do.

# ? Sep 27, 2018 20:34

Adbot: ADBOT LOVES YOU

# ? May 14, 2024 00:35

Mister Fister: May 17, 2008; D&D: HASBARA SQUAD
KILL-GORE

I love the smell of dead Palestinians in the morning.
You know, one time we had Gaza bombed for 26 days
(and counting!)

I'm learning about regular expressions. With the below expression, i understand the stuff inside the brackets will look to match any upper case/lowercase a-z character, periods, and whitespace while the * is a wildcard. What do the ^ and $ characters indicate? I'm having trouble finding documentation on this. Thanks!

code:

pattern = '^[A-Za-z\.\s]*$'

# ? Sep 29, 2018 23:21

The March Hare: Oct 15, 2006; _{Je r�ve d'un}
Wayne's World 3; Buglord

Mister Fister posted:

I'm learning about regular expressions. With the below expression, i understand the stuff inside the brackets will look to match any upper case/lowercase a-z character, periods, and whitespace while the * is a wildcard. What do the ^ and $ characters indicate? I'm having trouble finding documentation on this. Thanks!
code:
pattern = '^[A-Za-z\.\s]*$'

Start and end of line.

https://regex101.com/

This website will try to help you out, but it defaults to PHP so you will want to switch to Python on the left.

# ? Sep 29, 2018 23:25

Gangsta Lean: Dec 3, 2001; Calm, relaxed...what could be more fulfilling?

* isn�t wildcard, it means the previous thing should match zero or more times.

# ? Sep 30, 2018 00:57

Mister Fister: May 17, 2008; D&D: HASBARA SQUAD
KILL-GORE

I love the smell of dead Palestinians in the morning.
You know, one time we had Gaza bombed for 26 days
(and counting!)

Thanks guys!

# ? Sep 30, 2018 01:06

Cute n Popular: Oct 12, 2012

I've been learning Python for about a year. I think I have a pretty good grasp of the basics butI'm trying to move beyond that. I'm reading up on design patterns but most of the information out there isn't great.

Is there anything out there that goes a little bit deeper than this is a class, this is how you use it. Preferably someone going over small projects and explaining why the author choose to design something a certain way and the benefits/cons?

# ? Sep 30, 2018 01:11

bob dobbs is dead: Oct 8, 2017; I love peeps; Nap Ghost

Cute n Popular posted:

I've been learning Python for about a year. I think I have a pretty good grasp of the basics butI'm trying to move beyond that. I'm reading up on design patterns but most of the information out there isn't great.

Is there anything out there that goes a little bit deeper than this is a class, this is how you use it. Preferably someone going over small projects and explaining why the author choose to design something a certain way and the benefits/cons?

"fluent python"
"effective python"
and then go actually work for a few python shops
also try "the architecture of open source applications"

# ? Sep 30, 2018 01:37

Slimchandi: May 13, 2005; That finger on your temple is the barrel of my raygun

Fluent Python is a great suggestion, that is one dense book that I find myself coming back to again and again. The essays help break up the coding examples.

I was in a similar position to you a year or so ago, having a large project, writing code and refactoring/improving when you find out it sucks/doesn't work/isn't extensible/testable is effective, if slightly demoralising.

http://www.diveintopython3.net is another good free read, doesn't discuss much about patterns but covers lots of different topic areas.

# ? Sep 30, 2018 08:18

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Mister Fister posted:

I'm learning about regular expressions. With the below expression, i understand the stuff inside the brackets will look to match any upper case/lowercase a-z character, periods, and whitespace while the * is a wildcard. What do the ^ and $ characters indicate? I'm having trouble finding documentation on this. Thanks!
code:
pattern = '^[A-Za-z\.\s]*$'

Tangential but... To avoid escaping certain characters use raw strings for regex like this:

code:

pattern = r'^[A-Za-z\.\s]*$'

# ? Sep 30, 2018 11:15

huhu: Feb 24, 2006

Does anyone know of a good technique for extracting XMP Data from Photos? I used Lightroom to add a bunch of data to Photos and I'm trying to extract it all in a batch script for Django. Currently I'm using libxmp which doesn't seem to want to install on my hosting due to a bunch of non Python dependencies.

Edit: Ended up editing the library and hardcoding a path to the dependency.

huhu fucked around with this message at 23:23 on Oct 1, 2018

# ? Oct 1, 2018 21:49

Cute n Popular: Oct 12, 2012

bob dobbs is dead posted:

"fluent python"
"effective python"
and then go actually work for a few python shops
also try "the architecture of open source applications"

Thanks! Picked up a copy of Effective Python to start off with since it seems like it goes over the basics and fills in the gaps of anything I might have missed self-learning.

# ? Oct 3, 2018 02:10

Turtlicious: Sep 17, 2012; by Jeffrey of YOSPOS

So I wanted to create an open source python thing to help people with scheduling issues.

I've read, and am trying to understand this:

https://developers.google.com/optimization/scheduling/employee_scheduling

and I'm wondering where I would need to look next to see about adding an array, so that you can input names and shift preferences.

Like, just, I want to be able to input a name, and a bunch of numbers for shift preferences, and then it take all of those preferences and try to generate the schedule with the highest number of points.

# ? Oct 7, 2018 09:37

cinci zoo sniper: Mar 15, 2013

Turtlicious posted:

So I wanted to create an open source python thing to help people with scheduling issues.

I've read, and am trying to understand this:

https://developers.google.com/optimization/scheduling/employee_scheduling

and I'm wondering where I would need to look next to see about adding an array, so that you can input names and shift preferences.

Like, just, I want to be able to input a name, and a bunch of numbers for shift preferences, and then it take all of those preferences and try to generate the schedule with the highest number of points.

What are the custom preferences you want to add?

code:

solver.Add(shifts[(j, i)] != n)  # Nurse j cannot be assigned shift n on day i

Something like this?

# ? Oct 7, 2018 09:53

Grundulum: Feb 28, 2006

I am an almost total python novice. I need math worksheets like the ones found here. I feel like I can handle randomization of numbers and operations on my own, but I have no clue how to go about getting the layout. The end product doesn�t need to be fancy�an MxN table with each cell holding a binary operation is fine by me. What are my options?

# ? Oct 7, 2018 13:20

cinci zoo sniper: Mar 15, 2013

Grundulum posted:

I am an almost total python novice. I need math worksheets like the ones found here. I feel like I can handle randomization of numbers and operations on my own, but I have no clue how to go about getting the layout. The end product doesn�t need to be fancy�an MxN table with each cell holding a binary operation is fine by me. What are my options?

Sounds like you could go with storing your operations as strings inside a pandas dataframe cell, and then visualise it with matplotlib (example - https://stackoverflow.com/a/45936469). If you want something considerably fancier for the visualiation of tables then this may become quite a pain in the rear end.

# ? Oct 7, 2018 13:45

Turtlicious: Sep 17, 2012; by Jeffrey of YOSPOS

cinci zoo sniper posted:

What are the custom preferences you want to add?
code:
solver.Add(shifts[(j, i)] != n)  # Nurse j cannot be assigned shift n on day i
Something like this?

Sort of, but with numbers, so like I want it to place all the 5s

then 4s, 3s, 2s, 1s, and so on, until it's full and everyone has the closest to what they asked for.

# ? Oct 7, 2018 20:11

cinci zoo sniper: Mar 15, 2013

Turtlicious posted:

Sort of, but with numbers, so like I want it to place all the 5s

then 4s, 3s, 2s, 1s, and so on, until it's full and everyone has the closest to what they asked for.

"Closest to arbitrary position they asked" for is going to be a way more complex problem than what's discussed in that article. Are there really no options to do this manually?

# ? Oct 7, 2018 20:15

fletcher: Jun 27, 2003; ken park is my favorite movie; Cybernetic Crumb

I'm trying to write a little python script that outputs something in a Java friendly format, similar to the conversion that native2ascii does.

I'm very close to just shelling out to native2ascii to get this thing working, but I was curious about the behavior I observed.

Considering the following snippet:

Python code:

with open('test.out', mode='w', encoding='raw_unicode_escape') as out:
    out.write('\u00fc')

The test.out file contains � but I wanted it to contain \u00fc (i.e. the encoding='raw_unicode_escape' doesn't seem to make a difference here).

It seems to work fine for the other special characters I've come across, but not this one. What's goin on here?

# ? Oct 8, 2018 23:51

NtotheTC: Dec 31, 2007

Are you allowed to touch the java code? Because it would probably be much simpler to add a JSON parser library and have the python output a JSON object

To answer your question though try prepending your write string with r or ur if it's python2

code:

out.write(r'\u00fc')

NtotheTC fucked around with this message at 09:10 on Oct 9, 2018

# ? Oct 9, 2018 08:56

unpacked robinhood: Feb 18, 2013; by Fluffdaddy

How nighmarish would it be to try and interpret a bunch of gps coordinates someone logged while moving around into a series of waypoints ?

For example it would finding the coordinates of the points I highlighted on this map:

# ? Oct 9, 2018 09:05

Grundulum: Feb 28, 2006

unpacked robinhood posted:

How nighmarish would it be to try and interpret a bunch of gps coordinates someone logged while moving around into a series of waypoints ?

How often are you planning to do this? How much automation are you doing? If you�re only looking to do this infrequently, there is a Java application called Plot Digitizer that does exactly what you�re describing: give it minimum/maximum values in X and Y, then click on points to get the coordinates.

http://plotdigitizer.sourceforge.net/

(I know this isn�t a Python solution, but I don�t see a need to reinvent the wheel if this is a small-scale problem.)

# ? Oct 9, 2018 10:09

unpacked robinhood: Feb 18, 2013; by Fluffdaddy

Grundulum posted:

How often are you planning to do this? How much automation are you doing?

It should be automated, there are hundreds of those. (It's from the microsoft geolife corpus)
e: thanks for the link, it doesn't hurt having a manual solution.

unpacked robinhood fucked around with this message at 12:02 on Oct 9, 2018

# ? Oct 9, 2018 11:19

Hadlock: Nov 9, 2004

I need to write a tiny tiny flask app with front end ui, users will need to login using ldap + some sql backend. Users in ldap group A can see all the things, users in ldap group b can only see a few things. We just got a new sso vendor and boss thinks everything must be ldap now.

It looks like there is flask-ldap and flask-simpleldap, maybe some others. What should I be looking at?

I'm almost certainly using sql alchemy and postgres

# ? Oct 13, 2018 00:12

bob dobbs is dead: Oct 8, 2017; I love peeps; Nap Ghost

Hadlock posted:

I need to write a tiny tiny flask app with front end ui, users will need to login using ldap + some sql backend. Users in ldap group A can see all the things, users in ldap group b can only see a few things. We just got a new sso vendor and boss thinks everything must be ldap now.

It looks like there is flask-ldap and flask-simpleldap, maybe some others. What should I be looking at?

I'm almost certainly using sql alchemy and postgres

flask rules, flask hyphen stuff usually drools

use python-ldap

http://www.python-ldap.org/en/latest/

# ? Oct 13, 2018 02:46

cinci zoo sniper: Mar 15, 2013

https://github.com/mahmoud/boltons Some cool stuff.

# ? Oct 16, 2018 21:07

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

cinci zoo sniper posted:

https://github.com/mahmoud/boltons Some cool stuff.

This, particularly remap, looks really useful.

# ? Oct 17, 2018 01:41

NtotheTC: Dec 31, 2007

Considering the "flask hyphen stuff sucks" mantra, is the best solution for storage based sessions on flask to roll your own or is there an actual serviceable lib solution out there?

# ? Oct 18, 2018 15:00

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

I've ended up rolling my own a couple times.

(this is why you should default to Django and only use Flask if you have a really good idea of the scope of your project and know for a fact that there's good support for your needed features)

# ? Oct 26, 2018 19:11

Jose Cuervo: Aug 25, 2004

Not sure if this is the right place to ask this, but I want to generate a time series which simulates consumption of a product. One way of doing this is to assume that the consumption has a particular distribution, say Triangular(lower=2, mode=4, upper=5), and then at each time step draw a random variate from that distribution to simulate the amount consumed during that time step. However, generating the time series in this way does not produce any correlation in consumption between successive time steps. That is, if the consumption of the product was on the higher end of the distribution at time t, then the consumption of the product at time t+1 should likely be on the higher end of the distribution as well, and vice versa.

How would I go about adding the correlation aspect to the simulated time series?

# ? Oct 26, 2018 20:34

CarForumPoster: Jun 26, 2013; â¡POWERâ¡

Jose Cuervo posted:

Not sure if this is the right place to ask this, but I want to generate a time series which simulates consumption of a product. One way of doing this is to assume that the consumption has a particular distribution, say Triangular(lower=2, mode=4, upper=5), and then at each time step draw a random variate from that distribution to simulate the amount consumed during that time step. However, generating the time series in this way does not produce any correlation in consumption between successive time steps. That is, if the consumption of the product was on the higher end of the distribution at time t, then the consumption of the product at time t+1 should likely be on the higher end of the distribution as well, and vice versa.

How would I go about adding the correlation aspect to the simulated time series?

If you�re generating based on random variables (0,1) that follow a triangular CDF, you could make an if/then that looks at the random variable from t-1 and keeps generating numbers until the random number is within x distance/percentage/etc of the previous

# ? Oct 27, 2018 04:14

Morek: Nov 23, 2006

Jose Cuervo posted:

Not sure if this is the right place to ask this, but I want to generate a time series which simulates consumption of a product. One way of doing this is to assume that the consumption has a particular distribution, say Triangular(lower=2, mode=4, upper=5), and then at each time step draw a random variate from that distribution to simulate the amount consumed during that time step. However, generating the time series in this way does not produce any correlation in consumption between successive time steps. That is, if the consumption of the product was on the higher end of the distribution at time t, then the consumption of the product at time t+1 should likely be on the higher end of the distribution as well, and vice versa.

How would I go about adding the correlation aspect to the simulated time series?

What is the usage scale you're actually trying to simulate? Hundreds of use sessions, millions, billions?
How often do you need to run this simulation?
Getting a pin on that will help tell you what kind of abstractions/simplifications of the problem will work for your purposes.

If the answer was "less than a few million" and "once", i'd probably take a swing at directly simulating users and the properties of their sessions.

Some data structure of active users and their session states, at simulation clock tick you walk your set and simulate the user doing things or ending their session and new users signing in.
Nice thing about this sort of approach is it gives you easy access to a bunch of knobs that make logical sense to the business ("How long do users stay logged in?") and also easily tune down fidelity/tune up scale as needed.

# ? Oct 27, 2018 05:33

Spime Wrangler: Feb 23, 2003; Because we can.

You don't really give enough info to specify the problem. Does the triangle distribution describe instantaneous total consumption at time t, or does it describe inter-arrival times of consumers, or per-instance time-required-to-consume?

If it's the first then maybe describe the usage you want to simulate as the combination of a deterministic function and a random function, e.g.: consumption(t) = sin(t) + gaussian(mu=0, sigma=1)

Or maybe some kind of random walk?

# ? Oct 27, 2018 06:05

AFancyQuestionMark: Feb 19, 2017; Long time no see.

So, I am currently taking my first Image Processing course and part of the first assignment is histogram equalization. I am using numpy, and for some reason I just can't get it to actually equalize the histogram, no matter how hard I try. I have no clue why the following doesn't work:

code:

def histogram_equalize_greyscale(im_orig):
    image = im_orig * 255
    image = image.astype(np.uint8)
    hist = np.histogram(image, bins = 256)[0]
    sum = np.cumsum(hist)
    sum = np.divide(sum, image.size)
    input_intensities = np.where(hist > 0)[0]
    m = input_intensities[0]
    q = input_intensities[-1]
    lookup_table = np.round((sum[np.arange(256)] - sum[m]) / (sum[q] - sum[m]) * 255).astype(np.uint8)
    print(lookup_table)
    image = lookup_table[image]
    image = image.astype(np.float64)
    image /= 255
    return image, hist, np.histogram(image, bins = 256)[0]

For some reason, this just pushes all of the grey levels towards the upper range, rather than stretching out the histogram.

Can anyone help me?

# ? Oct 28, 2018 12:01

cinci zoo sniper: Mar 15, 2013

AFancyQuestionMark posted:

So, I am currently taking my first Image Processing course and part of the first assignment is histogram equalization. I am using numpy, and for some reason I just can't get it to actually equalize the histogram, no matter how hard I try. I have no clue why the following doesn't work:
code:
def histogram_equalize_greyscale(im_orig):
    image = im_orig * 255
    image = image.astype(np.uint8)
    hist = np.histogram(image, bins = 256)[0]
    sum = np.cumsum(hist)
    sum = np.divide(sum, image.size)
    input_intensities = np.where(hist > 0)[0]
    m = input_intensities[0]
    q = input_intensities[-1]
    lookup_table = np.round((sum[np.arange(256)] - sum[m]) / (sum[q] - sum[m]) * 255).astype(np.uint8)
    print(lookup_table)
    image = lookup_table[image]
    image = image.astype(np.float64)
    image /= 255
    return image, hist, np.histogram(image, bins = 256)[0]
For some reason, this just pushes all of the grey levels towards the upper range, rather than stretching out the histogram.

Can anyone help me?

So, which variable is failing expectations in what way? Right now my best guess is that you return histogram that is np.histogram(im_orig*255).

# ? Oct 28, 2018 12:14

AFancyQuestionMark: Feb 19, 2017; Long time no see.

No, that's not it. The return variable failing expetations is image itself (as well as its histogram as a consequence of that). The image I return has all of its grey levels near the brighter end of the scale near 255 (or 1 when normalized) rather than spread around, like they need to be.

# ? Oct 28, 2018 12:23

Hed: Mar 31, 2004; Fun Shoe

I would recommend backing up a bit. Look at the original histogram for huge spikes or cumulative transfer function of pixel intensity--does it have any discontinuities? If so then histogram linearization is going to have some interesting effects.

# ? Oct 28, 2018 14:06

AFancyQuestionMark: Feb 19, 2017; Long time no see.

It's probably as you say, since it seems to work fine on most other images. Sorry for troubling you.

# ? Oct 28, 2018 16:21

Jose Cuervo: Aug 25, 2004

CarForumPoster posted:

If you�re generating based on random variables (0,1) that follow a triangular CDF, you could make an if/then that looks at the random variable from t-1 and keeps generating numbers until the random number is within x distance/percentage/etc of the previous

Morek posted:

What is the usage scale you're actually trying to simulate? Hundreds of use sessions, millions, billions?
How often do you need to run this simulation?
Getting a pin on that will help tell you what kind of abstractions/simplifications of the problem will work for your purposes.

If the answer was "less than a few million" and "once", i'd probably take a swing at directly simulating users and the properties of their sessions.

Some data structure of active users and their session states, at simulation clock tick you walk your set and simulate the user doing things or ending their session and new users signing in.
Nice thing about this sort of approach is it gives you easy access to a bunch of knobs that make logical sense to the business ("How long do users stay logged in?") and also easily tune down fidelity/tune up scale as needed.

Spime Wrangler posted:

You don't really give enough info to specify the problem. Does the triangle distribution describe instantaneous total consumption at time t, or does it describe inter-arrival times of consumers, or per-instance time-required-to-consume?

If it's the first then maybe describe the usage you want to simulate as the combination of a deterministic function and a random function, e.g.: consumption(t) = sin(t) + gaussian(mu=0, sigma=1)

Or maybe some kind of random walk?

Sorry for not providing enough information. The project involves simulating the amount of fuel consumed at a forward operating base each day. During periods of intense activity there are several or more days of high daily consumption, while during lull periods there are stretches of several days of low daily consumption. Historical consumption information is classified information, so I am have to generate a consumption time series on my own. The information I can get from the subject matter experts is a minimum, maximum, and mode for the amount of fuel consumed per day, which is why I mentioned the Triangular distribution (although I suppose I could also use a Beta distribution).

Right now a generate a time series of the daily fuel consumption (180 days of daily consumption) by drawing a random variate from a Triangular(minimum, mode, maximum) distribution 180 times. However, generating a time series in this way does not, in general, result in the stretches of correlated activity that would be in the real time series. And I was trying to figure out a way to achieve this correlation, while still being able to say that the underlying distribution of the daily consumption is Triangular(minimum, mode, maximum).

# ? Oct 29, 2018 14:36

OnceIWasAnOstrich: Jul 22, 2006

A triangular distribution doesn't sound like it would effectively approximate what really sounds like some sort of bimodal distribution. Maybe model using two or more distributions (one for high usage one for low usage) and a latent variable describing which state you are in. You could then either sequentially draw X numbers from your current state where X is Poisson or binomial or something, and then another random number of days for the other and keep switching. Alternatively you draw one day at a time and also draw a Bernoulli at some low probability that tells you whether to switch states.

# ? Oct 29, 2018 19:54

Adbot: ADBOT LOVES YOU

# ? May 14, 2024 00:35

bob dobbs is dead: Oct 8, 2017; I love peeps; Nap Ghost

if you see clustering, use a fat-tailled distribution. lognormal for choice. beta w/ certain params that make fat tail is good too.

my default guess for every business variable has degenerated to pareto-distributed, which has been remarkably fruitful

# ? Oct 29, 2018 19:58

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »