GrAviTy84 posted:

I have a data set that has a lot of features. Each row is an event but each event is part of a larger event (it's power outage data, I have bulk outage events, broken down into restoration steps, as well as geographical categories, device categories, etc.) I want to be able to "collapse" different categories so that, say I just have a total time elapsed for one outage event by summing over the time taken in each restoration step, but maintain dataframe structure so that if I wanted to break it down further into GIS or month, or whatever sub categories, it would be easy to do later on. As such, I'm trying to develop a function that will take a df, a column name to categorize rows into, and a column name that is summed in each category. I'd like to keep it as standalone as I can so I can collapse these indices in different ways depending on what features I want to study. right now I have:

code:

def sumByFeature(df,sumIndex,sumValue):
    featureIndex = df.groupby(by=sumIndex)[sumValue].sum()
    df = pd.concat([df,featureIndex], axis=1, join_axes=[featureIndex['sumIndex'])
    return df.drop_duplicates([sumIndex],keep='first')

but it really doesnt like the way I'm passing the column name into the function. Is there a correct way to pass a column name of a dataframe into a function?

semi-related: is there a data science thread? I suppose it could be either in here or in SAL.

edit: there might be a much better way to do this, too. I'm new to doing this stuff with python so I'm still learning a lot of the methods that I can use stock.

Munkeymon posted:

Sounds like you want a rules engine.

Rosalind posted:

Rosalind posted:

I am about to give up on trying to learn Python.

I have completed every Python tutorial recommended to me. They all speak of this beautiful, elegant programming language and everything is so easy to do. Oh yes I know how to set strings to variables and make them all lower case and count the length of a string and add, divide, and subtract. Oh boy do I know how to do these things. But then I get to actually getting started with actually using Python to do anything useful and it turns into a poo poo show.

I spent 5 hours tonight just trying to do the following tonight and have failed miserably: import a CSV that I exported from R and then run basic descriptive statistics on it. I thought this would be a good first task as I have analyzed these data before in R and know them like the back of my hand.

I googled "Import csv from R and descriptive statistics python" (or some approximation thereof). I found people talking about something called "pandas." I then found people discussing Anaconda and Rodeo as "good environments" for data analysis in Python.

I installed Anaconda. It gave me a choice of several different things which did not seem relevant including RStudio??? I googled again and found that the one called Spyder was the one I needed. Every piece of code I tried to run in this Spyder thing gave me various error messages and no amount of googling got me any useful answers.

I then installed Rodeo. This looked like RStudio so I was happy at first, but again nothing seemed to work right. I found some code that someone posted to import a csv using a package called rpy2. I spent 2 hours getting rpy2 installed but I think I finally did after so much searching and then manually installing it because apparently it doesn't work normally in Windows and monkeying about with Paths and writing some custom pip thing that I don't even really know why it was necessary.

At this point at least I would run "import rpy2" and no error message popped up. Maybe I had broken it so much that it didn't even know to give me an error message because none of the rpy2 functions would actually work.

My questions now are: 1. How the heck does anyone ever learn how to use this? I'm the first to admit that I'm no programming guru, but I have some experience with several different languages and actually getting started with Python makes absolutely no sense. Everything feels like the biggest clusterfuck of dependencies and "install X, Y, and Z to get W to work but to get X to work install A and B which require C to run on Windows." 2. Is there a simple, idiot-proof (because that's what I am apparently) guide to moving from R to Python for data analysis and hopefully eventually machine learning?

Rosalind posted:

My questions now are: 1. How the heck does anyone ever learn how to use this? I'm the first to admit that I'm no programming guru, but I have some experience with several different languages and actually getting started with Python makes absolutely no sense. Everything feels like the biggest clusterfuck of dependencies and "install X, Y, and Z to get W to work but to get X to work install A and B which require C to run on Windows." 2. Is there a simple, idiot-proof (because that's what I am apparently) guide to moving from R to Python for data analysis and hopefully eventually machine learning?

Viking_Helmet posted:

Could you post some sample data and what you would want the outcome to look like?

Rosalind posted:

vikingstrike posted:

import pandas as pd
frame = pd.read_csv(my.csv)

GrAviTy84 posted:

I just started at this company and it's utilities data so I don't know how sensitive it is. here's some fake data showing sort of what's going on.

code:

	Event ID	Event value	location	event cause
1	1		1		LA		weather
2	1		1		LA		weather
3	1		1		LA		weather
4	2		1		Santa Ana	weather
5	3		1		LA		3rd party
6	4		1		Anaheim	        maintenance
7	4		1		Anaheim	        maintenance
8	4		1		Anaheim	        maintenance
9	4		1		Anaheim	        maintenance
10	4		1		Anaheim	        maintenance
11	5		1		Santa Ana	3rd party
12	5		1		Santa Ana	3rd party

so I want to make a function that takes this dataframe and 'collapses' repeats in certain columns and spits out a df with the same column structure. so for instance function(df, Event_ID, Event_value) would return a dataframe that looked like:

code:

	Event ID	Event value	location        event cause
1	1			3	LA              weather
2	2			1	Santa Ana	weather
3	3			1	LA              3rd party
4	4			5	Anaheim	        maintenance
5	5			2	Santa Ana	3rd party

damnfan posted:

Above is a binary search problem problem I was working on where there were a max of 3 errors to correct. My corrections were:


This would fail for the input shown below. Does anyone know why the binary search failed there? The problem was timed and submitted with a result of 66% correct.

Viking_Helmet posted:

Walk through what your function is doing, step by step:

code:

A = [1, 2, 5, 9, 9]
X = 2
#first step
m = (0 + 4) // 2 # 2
A[2] #5, greater than X so:
r = 1
#second step
m = 1 // 2 = 0
A[0] # 1, less than X so:
l = 1
#third step
l == r ##so we fail out here

Dominoes posted:

Thanks, homes. From what I gather so far, sounds like I need to roll my own, STS!!! Django.

Fusion Restaurant posted:

1. Do people use spaces or tabs to indent? I've noticed when copy pasting that sublime text is giving me tabs by default while Spyder is giving me spaces.

2. Also, a pandas Q which was a little too abstract for me to easily answer on stackexchange:
I have a bunch of csvs which I've downloaded from a site. Each is data for one week, all have the same columns, and I'd eventually like to concatenate them all into a big pandas dataframe.

Is the best way to do this to first clean each csv (which will involve dropping some columns, so making it smaller), and then merge?

Or should I just combine all the csvs into one giant csv, then turn it into a dataframe, and then clean that dataframe?

My concern is that the second method might leave me with a dataframe which is too big to really work with in memory (ie RAM) while I'm cleaning it and paring down the # of columns. Would the first approach actually be more memory efficient?

The last thing I guess I could do is to do the data cleaning on the csv's directly by editing the strings, or by reading them into a base python dictionary/list -- but I wasn't sure if that would actually save any memory? It would definitely be much more annoying.

e: Actually maybe what I really should get is a recommendation of a good guide to memory management in pandas. I'm pretty familiar with it in R, and how to efficiently do things w/in that language, but am totally lost in Python/pandas.

creatine posted:

I've been using tkinter to make small GUI programs but I've found it's incredibly frustrating to get to layout the way I want. Does anyone have resources/guides for wxPython or PyQt5? Those are the two I've seen most talked about so I figured I'd start there.

Munkeymon posted:

There aren't some already kicking around out there?

Whybird posted:

I've just started with wxPython's Phoenix fork (I started with tkinter too, and then I stopped when I realised it had no support for screen-readers). I'm currently using this, which is pretty terse but it seems OK so far.

baka kaba posted:

That causes issues - the original version moves the left and right extremes towards each other in a way that they should eventually arrive on the same index, at which point the loop immediately ends. That's where the desired element should be, so then it checks if it's there or not

Your version allows another loop when the indices match, so if they're at the end of the array (if X is the largest element) then you move L forward and it's out of bounds. You could get around this by doing your equality check first, but what if X isn't in the array?

Your solution sort of changes how the algorithm works with that early return. "3 changes" is kind of arbitrary but you said "3 errors" first, which implies small corrections - otherwise what counts as a single change, y'know? I think if you look at the original code with the edge cases I mentioned (specifically 2 elements where 1st<X and 2nd=X, eg [2,5] with X=5) you'll see a problem, and there's an easy fix for it

damnfan posted:

Thanks, I was stuck with the binary search implementation that I was familiar with rather than thinking of other implementations. The edit that satisfied the requirements ended up just being:

code:

m = (l + r + 1) // 2

Azuth0667 posted:

Tkinter is a pain in the rear end.

vikingstrike posted:

Why would it be that much different in python? Of course working on smaller chunks of a larger data set and then putting them together will lower the amount of RAM you need at any one time.

Viking_Helmet posted:

Ah, cool - I think you just want the basic groupby functionality. Assuming you have your example loaded into a dataframe, this should do it:

code:

grp = df.groupby(['Event ID','location','event cause']).sum()
grp = grp.reset_index()
print(grp)

code:

 Event ID   location  event cause  Event value
0        1         LA      weather            3
1        2  Santa Ana      weather            1
2        3         LA    3rd party            1
3        4    Anaheim  maintenance            5
4        5  Santa Ana    3rd party            2

accipter posted:

GUI programming is so totally foreign. It really takes a while to learn how to go about doing things. (At least it did for me when I learned Qt)

Azuth0667 posted:

Yeah it is it literally reminds me of learning Spanish. The most frustrating part of tkinter is that the tutorials I find are all 2-3 years out of date or in some other version of python so their example code doesn't even work.

HardDiskD posted:

Just spitballing here, but try to give the NamedTemporaryFile an unique prefix and/or suffix? Also, you might want to give it delete=False.

fletcher posted:

I started logging the filenames and they are definitely unique (/tmp/tmp04m80o4b & /tmp/tmp8sljj5jq) but the logs are still getting jumbled.

I'm almost certain it comes down to adding the new handler to the root logger:

code:

logging.getLogger('').addHandler(deploy_log_handler)

This seems to impact the root logger of other threads, I'm not really clear on the how and why though.