Bob Morales posted:

I'm trying to figure out what the least-dumb way to assign a service tech based on a map of highlighted states. Each rep has 1 or more states that they are responsible for, and i want to email the responsible tech when a ticket comes in from a store in their state. I'm new to Python.

code:

    states = ['TX', 'OK']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'jim'

    states = ['AR', 'LA', 'MS', 'AL', 'GA', 'TN']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'mark'

    states = ['FL']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'dave'

    states = ['MI', 'IL', 'IN', 'OH', 'NC', 'SC']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'tom'

    states = ['KY', 'WV', 'VA', 'MD', 'DE', 'NJ', 'CT', 'RI', 'MA', 'ME', 'NH', 'VT',
        'NY', 'PA']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'bill'

    states = ['WA', 'OR', 'ID', 'MT', 'WY', 'UT', 'CO', 'NM', 'ND', 'SD', 'NE', 'KS',
        'MN', 'IA', 'MO', 'WI']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'joe'

Would some kind of 2D array work better? Trying to not repeat myself as much as I can as well as make it easy to change techs/states later down the road.

Bob Morales posted:

I'm trying to figure out what the least-dumb way to assign a service tech based on a map of highlighted states. Each rep has 1 or more states that they are responsible for, and i want to email the responsible tech when a ticket comes in from a store in their state. I'm new to Python.

code:

    states = ['TX', 'OK']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'jim'

    states = ['AR', 'LA', 'MS', 'AL', 'GA', 'TN']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'mark'

    states = ['FL']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'dave'

    states = ['MI', 'IL', 'IN', 'OH', 'NC', 'SC']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'tom'

    states = ['KY', 'WV', 'VA', 'MD', 'DE', 'NJ', 'CT', 'RI', 'MA', 'ME', 'NH', 'VT',
        'NY', 'PA']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'bill'

    states = ['WA', 'OR', 'ID', 'MT', 'WY', 'UT', 'CO', 'NM', 'ND', 'SD', 'NE', 'KS',
        'MN', 'IA', 'MO', 'WI']
    if d.get('State Abbreviation') in states:
      responsibleTech = 'joe'

Would some kind of 2D array work better? Trying to not repeat myself as much as I can as well as make it easy to change techs/states later down the road.

Jose Cuervo posted:

This only works if each state is taken care of by a single tech.

Jose Cuervo posted:

You could create a dictionary with each state mapped to the tech responsible:

Python code:

state_to_tech = {'TX': 'jim,
        'OK': 'jim',
        'AR': 'mark', 
....}

and then use the following line

Python code:

responsible_tech = state_to_tech[d.get('State Abbreviation')]

This only works if each state is taken care of by a single tech.

SurgicalOntologist posted:

The only thing "wrong" with a functional style is it's not mainstream Python, so your typical Python programmer is not likely to have encountered it and will be confused.

Cingulate, I don't know about identifying a style (it's not really clear cut, even my version isn't functional in the more academic sense), but it's worth pointing out that there's no list comprehension, but a generator expression. You shouldn't have 75% list comprehensions, but a lot of generator expressions/comprehensions of all kinds is usually considered the best way to write Python code.

BigRedDot posted:

I haven't used map or filter in years. List comprehensions and generator expressions

• are declarative (almost always the best option when it is available)
  
• are non-trivially more efficient (dispatch immediately to Python C API)
  

As for the style, it's sometimes referred to as "declarative". In the manner of Prolog, the idea is to tell the computer, directly, what you want, rather than a bunch of steps for how to do it. The implication being that you always have to know what you want, but that giving the steps requires converting what you want into the sequence of steps, and that eliminating this conversion process leaves less room for errors.

Bob Morales posted:

We use 1 tech per state right now.

I thought about that, but then if we fire a tech and hire a new one I have to change the tech assigned to like, 15 states in some cases.

Dominoes posted:

Dominoes posted:

setuptools question.

Let's say I have a module called 'quick', and ran python setup.py develop on it. Inside the main directory is a subdirectory called quick, and a file in it called funcs.py.

funcs.py has some functions called 'mean', 'var', etc.

I want to use this like so:

Python code:

import quick

quick.var()

No worky. I have to do this currently:

Python code:

import quick.funcs

quick.funcs.var()

I looked at some setup.pys on github and found something like this in requests':
from .funcs import sum_ as sum, mean, var
No worky. Any idea?

Dominoes posted:

Did some homework: It looks like splitting loops does cause slowdowns (Although still much faster than equivalent numpy/scipy funcs), but you can still split up some of the funcs. Ie:

These basic funcs that work together can't really be made to share loops, so there's no benefit to mushing them together:

Python code:

@numba.jit(nopython=True)
def sum_(data):
    sum__ = 0.
    for i in range(data.size):
        sum__ += data[i]
    return sum__


@numba.jit(nopython=True)
def mean(data):
    M = data.size
    sum__ = sum_(data)
    return sum__ / M


@numba.jit(nopython=True)
def var(data):
    M = data.size

    mean_ = mean(data)
    var_sum = 0.
    for i in range(M):
        var_sum += (data[i] - mean_) ** 2
    return var_sum / M


@numba.jit(nopython=True)
def std(data):
    return var(data) ** .5

But this alternative to the pearson implementation I posted above that uses those funcs is more than twice as slow:

Python code:

@numba.jit(nopython=True)
def corr_broken(data1, data2):
    """Optimized pearson correlation test, similar to scipy.stats.pearsonr."""
    M = data1.size

    mean1 = mean(data1)
    mean2 = mean(data2)

    std1 = std(data1)
    std2 = std(data2)

    cross_sum = 0.
    for i in range(M):
        cross_sum += (data1[i] * data2[i])
    cross_mean = cross_sum / M

    return (cross_mean - mean1 * mean2) / (std1 * std2)

QuarkJets posted:

Have you actually compared the numpy mean() and sum() method against your compiled mean() and sum() functions? I would have guessed that the numpy functions are about the same speed in the single-array case, since it's using compiled Fortran (faster) with a bunch of extra features (slower). But once you start doing more complex things, like calculating correlations, numpy's tendency to create temporary arrays would bite you in the rear end and the numba implementation would become faster.

SurgicalOntologist posted:

Many tutorials/examples will instead recommend from .funcs import var but my understanding is since Python 3 relative imports are not recommended.

SurgicalOntologist posted:

It seems that if someone was really concerned about speed, and needed both the variance and the mean, they would end up implementing it themselves rather than use a library version which will inevitably compute the mean twice. In this day and age I can't see anyone re-implementing mean and variance as anything except wasted effort. What can library authors do about this?

QuarkJets posted:

e: I guess that you could write simple specific-case functions and just make their uses very specific ("mean_float32 returns the float32 flattened mean of a an array of float32 values", "mean_int64 returns the int64 flattened mean of an array of int64 values", etc). That's appealing to people solving hard computational problems due to the speed involved, but it's not very appealing to the computer scientists that create standard libraries (for maintainability and user friendliness reasons) or to the average user of these libraries. Going back to FFTW, that's why FFTW has so many separate but almost-equivalent functions (this one takes a complex array and modifies it in place, this one takes a real array and returns a complex array, etc). The maintainers want a library that is optimally fast, and they chose to sacrifice some user friendliness in order to maintain that speed.

SurgicalOntologist posted:

This brings up an interesting point. We often have situations where an intermediate result of one algorithm is useful on its own. Variance and mean is an obvious, and simple, example, but I can recall encountering the same issue before with more complex algorithms. Is there a standard for dealing with this?

It seems that if someone was really concerned about speed, and needed both the variance and the mean, they would end up implementing it themselves rather than use a library version which will inevitably compute the mean twice. In this day and age I can't see anyone re-implementing mean and variance as anything except wasted effort. What can library authors do about this?

I see two options, neither of which are very attractive (and neither of which I've seen done). You could have the variance function spit out the mean as well. Which would work but would create some awkward code. Alternatively, the variance function could take the mean as an optional input, which if passed causes the mean computation step to be skipped. This would also work but relies on an invariant that could break down.

Is there a more general term for this problem in computer science? Basically the phenomenon whereby clean encapsulation is at odds with doing each calculation only once.

BigRedDot posted:

BTW Dominoes I share a new office with most of the Numba devs. If you have any specific ideas you want me to pass on please feel free to let me know here or pm.

KICK BAMA KICK posted:

CODE AND poo poo

jimcunningham posted:

%d and the like are confusing to me for some reason.

BigRedDot posted:

There is another approach, deferred execution, and that is the path that Blaze (and dynd to some degree) are trying to follow. Basically, instead of immediately executing expressions, you keep track of all the expressions that are used and collect them into a big expression graph. The only when you actually want a "realized" result is the entire expression graph executed. But at this point you have a lot more information, you can optimize the expression graph to coalesce duplicate computations, remove unnecessary temporaries, use the most efficient access patterns, and only compute enough to actually provide what was asked for. If you are thinking that sounds like another compiler, well that's because basically it is. If you combine this with things like Numba, it becomes even more powerful. Our vision is for scientific and analytical compute is really about being able to spell things at a high level, but push efficient execution down to the metal and across clusters.

KICK BAMA KICK posted:

but that doesn't work in Python 3 because enumerate returns a generator or iterator or something like that instead of a plain old list.

BigRedDot posted:

You can "slice" generators using itertools:

SurgicalOntologist posted:

This brings up an interesting point. We often have situations where an intermediate result of one algorithm is useful on its own. Variance and mean is an obvious, and simple, example, but I can recall encountering the same issue before with more complex algorithms. Is there a standard for dealing with this?

It seems that if someone was really concerned about speed, and needed both the variance and the mean, they would end up implementing it themselves rather than use a library version which will inevitably compute the mean twice. In this day and age I can't see anyone re-implementing mean and variance as anything except wasted effort. What can library authors do about this?

I see two options, neither of which are very attractive (and neither of which I've seen done). You could have the variance function spit out the mean as well. Which would work but would create some awkward code. Alternatively, the variance function could take the mean as an optional input, which if passed causes the mean computation step to be skipped. This would also work but relies on an invariant that could break down.

Is there a more general term for this problem in computer science? Basically the phenomenon whereby clean encapsulation is at odds with doing each calculation only once.

jimcunningham posted:

Is the old style use a no-no. Or just preference?