Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Locked thread
the
Jul 18, 2004

by Cowcaster
How would I handle reading times that are like:

23:32.6
1:02:55.7

Where it's usually in MM:SS but occasionally HH:MM:SS

I tried using datetime, but that has issue with the tenths of a second. And I'm also wondering how to handle checking whether or not there's an hour present. Should I search to see if there are two colons present in the string?

I need to plot the data, so I have just been trying to read all of the times and then convert it all to seconds. Unless there's a way to plot datetime data?

Adbot
ADBOT LOVES YOU

Thermopyle
Jul 1, 2003

...the stupid are cocksure while the intelligent are full of doubt. —Bertrand Russell

the posted:

How would I handle reading times that are like:

23:32.6
1:02:55.7

Where it's usually in MM:SS but occasionally HH:MM:SS

I tried using datetime, but that has issue with the tenths of a second. And I'm also wondering how to handle checking whether or not there's an hour present. Should I search to see if there are two colons present in the string?

I need to plot the data, so I have just been trying to read all of the times and then convert it all to seconds. Unless there's a way to plot datetime data?

I'm not sure if it will handle it for you, but look at python-dateutil's parser module.

onionradish
Jul 6, 2006

That's spicy.

the posted:

How would I handle reading times that are like:

23:32.6
1:02:55.7

Where it's usually in MM:SS but occasionally HH:MM:SS

I tried using datetime, but that has issue with the tenths of a second. And I'm also wondering how to handle checking whether or not there's an hour present. Should I search to see if there are two colons present in the string?

I need to plot the data, so I have just been trying to read all of the times and then convert it all to seconds. Unless there's a way to plot datetime data?
If you're on Python 2.6+, datetime has added a '%f' format for milliseconds microseconds, which will take care of the tenths of seconds. Checking for the number of colons using str.count() and using either '%M:%S.%f' or '%H:%M:%S.%f' seems easy enough.

%f treats the microsecond value as a fraction, so '.6' gets correctly parsed into '600000' microseconds.
\/ \/ \/

onionradish fucked around with this message at 02:53 on Jul 17, 2014

the
Jul 18, 2004

by Cowcaster

onionradish posted:

If you're on Python 2.6+, datetime has added a '%f' format for milliseconds microseconds, which will take care of the tenths of seconds. Checking for the number of colons using str.count() and using either '%M:%S.%f' or '%H:%M:%S.%f' seems easy enough.

Not sure if this will work, looks like microsecond only takes stuff like "000000, 000001, ..., 999999" according to the official docs.

edit: NM, I got it to work, thanks for the help

the fucked around with this message at 02:39 on Jul 17, 2014

Gangsta Lean
Dec 3, 2001

Calm, relaxed...what could be more fulfilling?
I'm late and you said you got it already, but I'd do something like this, looping over the formats in order of which ever is expected to occur more often. I threw in a bad value to show that it raises the most recent exception if it can't find a match.

code:
#!/usr/local/bin/python2.7
    
import datetime


def parse_time(time_str):
    formats = ['%M:%S.%f', '%H:%M:%S.%f']
    
    for fmt in formats:
        try:
            return datetime.datetime.strptime(time_str, fmt).time()
        except ValueError:
            continue
    raise


times = ['23:32.6', '1:02:55.7', '1']
for t in times:
    print 'parsed is {}'.format(parse_time(t))

the
Jul 18, 2004

by Cowcaster
I did it a more idiotic way, but whatever works.

I'd like to get your guys opinions on this. I decided I wanted to look at race data from a 5k, and see if there's a correlation between age and time. It's obvious there is one, right? Older people tend to finish slower. Younger people tend to finish faster. Much younger people may end up finishing slower. There's a sweet spot right around 20-35, I'd think.

I found a bunch of race sites that had their results in CSV files, and I downloaded a bunch of them (10 or so race results). I ended up with 4500+ participants.

code:
import csv
import datetime
import matplotlib.pyplot as plt

data = []

for i in range(4,13,1):
	with open('Results ('+str(i)+').csv', 'rb') as r:
		reader = csv.reader(r, delimiter=',')
		l = list(reader)
	for row in l[1:]:
		time = row[5]
		if i == 5 or i == 10: #To account for two files that had odd table ordering
			age = row[9]
		else: 
			age = row[8]
		if time.count('.') == 1: #If there's a microsecond
			if time.count(':') == 1:
				ttime = datetime.datetime.strptime(time,'%M:%S.%f')
				minute = float(ttime.minute)
				second = float(ttime.second)
				total = second/60 + minute
				data.append([total,float(age)])
			if time.count(':') == 2:
				ttime = datetime.datetime.strptime(time,'%I:%M:%S.%f')
				hour = float(ttime.hour)
				minute = float(ttime.minute)
				second = float(ttime.second)
				total = second/60 + minute + hour*60
				data.append([total,float(age)])
		if time.count('.') == 0: #If there's no microsecond
			if time.count(':') == 1:
				ttime = datetime.datetime.strptime(time,'%M:%S')
				minute = float(ttime.minute)
				second = float(ttime.second)
				total = second/60 + minute
				data.append([total,float(age)])
			if time.count(':') == 2:
				ttime = datetime.datetime.strptime(time,'%I:%M:%S')
				hour = float(ttime.hour)
				minute = float(ttime.minute)
				second = float(ttime.second)
				total = second/60 + minute + hour*60
				data.append([total,float(age)])

x = []
y = []

for row in data:
	x.append(row[0])
	y.append(row[1])

plt.figure(1)
plt.plot(x,y,'r.')
plt.xlabel('Time in Minutes')
plt.ylabel('Age in Years')
plt.xlim(15,65)
plt.ylim(5,90)
plt.show()
My plot looks... not correlated at all. In fact, it looks all over the place. So, statistical error? Grabbing the data wrong? Or is there just no correlation? I'm at a loss.

SurgicalOntologist
Jun 17, 2004

Well you should at least check what kind of correlation you do have, that's not completely uncorrelated. Best way, IMO, is to pip install seaborn then
Python code:
import seaborn as sns

grid = sns.jointplot(y, x)  # put age on the x
# also why didn't you call it age

grid.ax_joint.set_xlabel('age')
grid.ax_joint.set_ylabel('time')
It could be that your data is correct but in regard to this particular race there's only a small correlation between age and time. I mean, where was the race? Who put it on? Etc. The relative lack of people around age 20 strikes me as unusual. The jointplot includes histograms so it can help you get a sense of the distributions as well as the correlation.

SurgicalOntologist fucked around with this message at 04:43 on Jul 17, 2014

the
Jul 18, 2004

by Cowcaster

SurgicalOntologist posted:

It could be that your data is correct but in regard to this particular race there's only a small correlation between age and time. I mean, where was the race? Who put it on? Etc. The relative lack of people around age 20 strikes me as unusual. The jointplot includes histograms so it can help you get a sense of the distributions as well as the correlation.

It's around 10-11 different races. I'm thinking I should try to find even more to see if it evens out, but I thought 4500 total runners would be a sufficient data sample.

Another theory is that maybe older runners tend to be more experienced, so they'll still put down roughly consistent good times, but younger runners are all over the map. A 25 year old could just as likely be a first timer as well as a serious athlete, but a 45 year old most likely won't be doing it for the first time.

However, you can kinda see the shape I was looking for if you look at the limits of times around 20-30 minutes with respect to the elderly. No 80 year old puts down faster than 30. No 70 year old puts down faster than 25. No 60 year old puts down faster than 23, and so on. Maybe looking at median times would be a better option?

BeefofAges
Jun 5, 2004

Cry 'Havoc!', and let slip the cows of war.

Don't overthink it, just plot a bunch of different stuff and see what happens. IPython and pandas are great for this sort of rapid experimentation with data.

SurgicalOntologist
Jun 17, 2004

the posted:

It's around 10-11 different races. I'm thinking I should try to find even more to see if it evens out, but I thought 4500 total runners would be a sufficient data sample.

Another theory is that maybe older runners tend to be more experienced, so they'll still put down roughly consistent good times, but younger runners are all over the map. A 25 year old could just as likely be a first timer as well as a serious athlete, but a 45 year old most likely won't be doing it for the first time.

However, you can kinda see the shape I was looking for if you look at the limits of times around 20-30 minutes with respect to the elderly. No 80 year old puts down faster than 30. No 70 year old puts down faster than 25. No 60 year old puts down faster than 23, and so on. Maybe looking at median times would be a better option?

Like I said, instead of talking about what you can kinda see, actually run a correlation and actually look at the distributions of age and time. Then you can start thinking about theories.

Lyon
Apr 17, 2003

the posted:

It's around 10-11 different races. I'm thinking I should try to find even more to see if it evens out, but I thought 4500 total runners would be a sufficient data sample.

Another theory is that maybe older runners tend to be more experienced, so they'll still put down roughly consistent good times, but younger runners are all over the map. A 25 year old could just as likely be a first timer as well as a serious athlete, but a 45 year old most likely won't be doing it for the first time.

However, you can kinda see the shape I was looking for if you look at the limits of times around 20-30 minutes with respect to the elderly. No 80 year old puts down faster than 30. No 70 year old puts down faster than 25. No 60 year old puts down faster than 23, and so on. Maybe looking at median times would be a better option?

Race data is tricky for a couple of reasons. First while I've never done the research, I've heard that runners don't come into their prime until their mid thirties a number of times. Second races tend to self select people of a certain fitness level who are confident they can finish the race. Realistically age counts for less than your conditioning/fitness level/amount of training time. If you picked 4,500 random people and forced them to run a 5k out of the blue I bet you would see the visual correlation you were expecting.

The data is probably right and it is just your expectations that are wrong. I would follow SurgicalOntologist's advice and run an actual correlation.

Lyon fucked around with this message at 07:36 on Jul 17, 2014

the
Jul 18, 2004

by Cowcaster
edit: Fixed my problem

the fucked around with this message at 17:26 on Jul 17, 2014

thegasman2000
Feb 12, 2005
Update my TFLC log? BOLLOCKS!
/
:backtowork:
So I completed the codeacademy course, well kinda and skipped the repetition. So I now need a small manageable project to get goign with. Also whats my best option for messing around in python. Running OSX with xcode installed and working.

good jovi
Dec 11, 2000

'm pro-dickgirl, and I VOTE!

thegasman2000 posted:

So I completed the codeacademy course, well kinda and skipped the repetition. So I now need a small manageable project to get goign with. Also whats my best option for messing around in python. Running OSX with xcode installed and working.

Find a problem that you have and solve it. Even if it is solved adequately by some existing project. It's a lot more motivating when you're making something for yourself, not just to have done it.

the
Jul 18, 2004

by Cowcaster

thegasman2000 posted:

So I completed the codeacademy course, well kinda and skipped the repetition. So I now need a small manageable project to get goign with. Also whats my best option for messing around in python. Running OSX with xcode installed and working.

Find every prime number between 0 and 100.

edit: in 20 lines or less

SurgicalOntologist
Jun 17, 2004

the posted:

Find every prime number between 0 and 100.

edit: in 20 lines or less

Show us a jointplot of your data already. I'm legitimately curious.

the
Jul 18, 2004

by Cowcaster

SurgicalOntologist posted:

Show us a jointplot of your data already. I'm legitimately curious.

Haha, sorry. I was doing ~work stuff~ all day (beatbox, salesforce, etc). So now I have some downtime and can finally learn Pandas.

SurgicalOntologist
Jun 17, 2004

Haha fair enough. You don't need pandas for that though (well you need it installed as a dependency), you can make the plots straight from your existing lists in one line, with Seaborn. But take your time, I was just being impatient.

thegasman2000
Feb 12, 2005
Update my TFLC log? BOLLOCKS!
/
:backtowork:
Sorry I am sure your all fed up of answering this but whats the text editor of choice for a python newbie? Looking for something that will prompt and highlight errors such as indentation before compiling if possible. Trying out textmate at the moment but its not highlighting an error I am getting.

onionradish
Jul 6, 2006

That's spicy.

thegasman2000 posted:

Sorry I am sure your all fed up of answering this but whats the text editor of choice for a python newbie? Looking for something that will prompt and highlight errors such as indentation before compiling if possible. Trying out textmate at the moment but its not highlighting an error I am getting.
PyCharm is more than a text editor, but I personally found it very helpful when I started learning.

It catches syntax errors before the script is run, warns when code isn't following PEP8 "best practices" in formatting so I learned good habits early, can pop-up documentation on functions and arguments, and includes breakpoint debugging and other tools to inspect variables, step through loops, etc.

I've grown into using some of its advanced features like integration with version control and test automation, but you can ignore all of those kinds of features when starting and just use it to edit scripts.

The Community Edition is free.

thegasman2000
Feb 12, 2005
Update my TFLC log? BOLLOCKS!
/
:backtowork:
perfect thanks!

Colonel J
Jan 3, 2008
This might be dumb but I can't see it right now. I'm trying to make images using misc.imsave and it seems the function works unlike I thought it would? The doc is very unspecific as to how it owrks so I'd just like to make sure I'm getting this right.

I have this code:
code:
from scipy import misc
import numpy

width = 128;
height = 64;
img = numpy.zeros((width, height));
for i in range(width):
	for j in range(height):
		img[i,j] = i*j;

misc.imsave("testimage.png",img);
which gives this image vvv . The problem is the width and height are inverted compared with how I wanted it to turn out... Am I doing something wrong? Is the image just rotated? I'm doing some coordinate sensitive stuff so I want to make sure I'm doing this right. Thanks, I've never programmed in Python before and some things tend to throw me off.


edit: I guess it's fine this way, I was just surprised because I thought it would work the other way, but switching around the indices makes it work the way I want, kinda.

Colonel J fucked around with this message at 16:58 on Jul 18, 2014

the
Jul 18, 2004

by Cowcaster
How do I do a conditional IF statement that checks to see if a datetime object is newer than a certain date?

The dates are in the format: datetime.datetime(2013, 8, 26, 17, 21, 35)

And I need to do something like:

code:
if datetime >= 2012, 5, 1, 00, 00, 00
edit: ended up doing this:

code:
checkdate = datetime.datetime(2012, 5, 01, 00, 00, 00)
if diff(mydate.date() - checkdate.date()).date > 0:

the fucked around with this message at 18:44 on Jul 18, 2014

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

the posted:

How do I do a conditional IF statement that checks to see if a datetime object is newer than a certain date?

The dates are in the format: datetime.datetime(2013, 8, 26, 17, 21, 35)

And I need to do something like:

code:
if datetime >= 2012, 5, 1, 00, 00, 00
edit: ended up doing this:

code:
checkdate = datetime.datetime(2012, 5, 01, 00, 00, 00)
if diff(mydate.date() - checkdate.date()).date > 0:

Compare it to another datetime object. Check out the "Supported operations" for datetime.

edit:

code:
if mydate >= checkdate:
    # do stuff

Space Kablooey
May 6, 2009


code:
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime
>>> date = datetime(2013, 8, 26, 17, 21, 53)
>>> date
datetime.datetime(2013, 8, 26, 17, 21, 53)
>>> today = datetime.now()
>>> today
datetime.datetime(2014, 7, 18, 14, 45, 55, 374920)
>>> date >= today
False
>>> date == today
False
>>> date <= today
True
>>> date != today
True
>>> 
fake edit: beaten

bobua
Mar 23, 2003
I'd trade it all for just a little more.

First time touching python. Trying to send some serial commands to an lcd display. The docs give a Dec and Hex code for each command. Dec 17(hex 11) enables the backlight for example.

So to enable the backlight, I use:
port.write(chr(17).encode())

To actually write text I use:
s = "My Text"
port.write(s.encode())

All is well.


Moving on, you can also play sounds. Dec 221(hex DD) should play A#. So I tried:
port.write(chr(221).encode())

And got nothing. I looped through all the notes and got nothing. So I thought maybe I should try sending the hex instead of the decimal. I tried:
cmd = "\xDD"
port.write(str(cmd).encode())
and got nothing.

Is this the right way to send hex in python 3? Any other suggestions?

suffix
Jul 27, 2013

Wheeee!

bobua posted:

First time touching python. Trying to send some serial commands to an lcd display. The docs give a Dec and Hex code for each command. Dec 17(hex 11) enables the backlight for example.

So to enable the backlight, I use:
port.write(chr(17).encode())

To actually write text I use:
s = "My Text"
port.write(s.encode())

All is well.


Moving on, you can also play sounds. Dec 221(hex DD) should play A#. So I tried:
port.write(chr(221).encode())

And got nothing. I looped through all the notes and got nothing. So I thought maybe I should try sending the hex instead of the decimal. I tried:
cmd = "\xDD"
port.write(str(cmd).encode())
and got nothing.

Is this the right way to send hex in python 3? Any other suggestions?

Beware of encoding! By default str.encode will encode strings as utf-8, so you'll get:
code:
>>> chr(221).encode()
b'\xc3\x9d'
which probably isn't what you want.

Now your way should work if you encode as latin1, which is sometimes handy if you get a string from other sources:
code:
>>> chr(221).encode('latin1')
b'\xdd'
but you shouldn't need to use unicode strings at all here, so I think it would be easier to make a byte string directly:
code:
>>> bytes([221])
b'\xdd'
>>> b'\xDD'
b'\xdd'

DickParasite
Dec 2, 2004


Slippery Tilde
Hey all,

Anyone got any good tutorials or explanations for using an oracle wallet to connect to an oracle database in Python? I've googled the crap out of it and I can't find anything.

Haystack
Jan 23, 2005





Have you looked into what cx_oracle can do? That's the oracle driver SQLalchemy uses.

DickParasite
Dec 2, 2004


Slippery Tilde

Haystack posted:

Have you looked into what cx_oracle can do? That's the oracle driver SQLalchemy uses.

I've been using cx-oracle. I need to use the oracle wallet with cx-oracle, which seems like it should be possible. Thanks for the link, I'll take a closer look at it.

ShadowHawk
Jun 25, 2000

CERTIFIED PRE OWNED TESLA OWNER
Do the various Python web frameworks (Django, Flask, Bottle, etc.) all spawn a separate process or separate thread for each incoming user hitting a given route? Does this vary if you switch from the internal test webserver to something serious?

Plorkyeran
Mar 22, 2007

To Escape The Shackles Of The Old Forums, We Must Reject The Tribal Negativity He Endorsed
No. Simple dev servers generally just only handle one request at a time and hope that they can process requests fast enough that any additional ones that come in don't time out. Real production deployments involve a fixed (or bounded to a small range) number of worker processes with a load balancer.

Carrier
May 12, 2009


420...69...9001...
I'm trying to write a program that would model movement to a changing target every t seconds. i.e. Say we deal only in 1 dimension, and I start at position x = 0. I receive some data every (say) 5 seconds that says the target position is a new position P. I then want to model the position of my point x so it moves say + 1 towards the target every second until the next update of target position, when it adjusts and starts moving towards the new target position from its current position.

Is there a simple way to do this? I've been getting bogged down in trying to understand the Threading module since that is what a quick browse of StackOverflow seemed to suggest I would need to use, but I've never used it before and I'm getting myself more and more confused each time...

Space Kablooey
May 6, 2009


I need to parse ISO 8601 dates and I found the dateutil module (https://pypi.python.org/pypi/python-dateutil/1.5). However, the latest version for python 2.X is 1.5, which is 4 years old by now. Is there any recommendations against it?

Lysidas
Jul 26, 2002

John Diefenbaker is a madman who thinks he's John Diefenbaker.
Pillbug
dateutil 2.2 is listed as supporting Python 2.6, 2.7, 3.2, and 3.3 according to https://pypi.python.org/pypi/python-dateutil/2.2. It's fine on 3.4 too, but the package classifiers haven't been updated yet.

Space Kablooey
May 6, 2009


I thought the author just picked all those versions as a catch-all. Thanks. :)

suffix
Jul 27, 2013

Wheeee!

ShadowHawk posted:

Do the various Python web frameworks (Django, Flask, Bottle, etc.) all spawn a separate process or separate thread for each incoming user hitting a given route? Does this vary if you switch from the internal test webserver to something serious?

It depend on the framework and the server. IIRC CherryPy has a decent threaded server for development mode, while Flask and Bottle just uses a single-threaded server.

When you're using a separate WSGI server, it will usually spawn a number of worker processes or threads and delegate incoming requests to them, but I think most of them can be configured to thread/process-per-request.

The WSGI PEP specifies how a framework indicates support for multithreading or multiprocessing.

quote:

wsgi.multithread This value should evaluate true if the application object may be simultaneously invoked by another thread in the same process, and should evaluate false otherwise.
wsgi.multiprocess This value should evaluate true if an equivalent application object may be simultaneously invoked by another process, and should evaluate false otherwise.

SurgicalOntologist
Jun 17, 2004

Carrier posted:

I'm trying to write a program that would model movement to a changing target every t seconds. i.e. Say we deal only in 1 dimension, and I start at position x = 0. I receive some data every (say) 5 seconds that says the target position is a new position P. I then want to model the position of my point x so it moves say + 1 towards the target every second until the next update of target position, when it adjusts and starts moving towards the new target position from its current position.

Is there a simple way to do this? I've been getting bogged down in trying to understand the Threading module since that is what a quick browse of StackOverflow seemed to suggest I would need to use, but I've never used it before and I'm getting myself more and more confused each time...

Explain a little more what the requirements are for your program. Are you recording the data, plus interpolating, and recording all the points? Or do you want to provide some sort of interface with which an another thread can ask for the current estimated position whenever it wants?

When you say "my point" and "target" are these two different physical things? In other words, is "my point" an estimate of the current position of the target or is it some modeled point that's chasing a physically separated target?

Carrier
May 12, 2009


420...69...9001...

SurgicalOntologist posted:

Explain a little more what the requirements are for your program. Are you recording the data, plus interpolating, and recording all the points? Or do you want to provide some sort of interface with which an another thread can ask for the current estimated position whenever it wants?

When you say "my point" and "target" are these two different physical things? In other words, is "my point" an estimate of the current position of the target or is it some modeled point that's chasing a physically separated target?

I'm basically trying to simulate an object responding to commands to move to a certain position and I want to simulate some sort of delay in this response so that sometimes, it might not make it to the position its supposed to reach before it gets given another command. Sorta simulating like a car being told to go to point A, then to point B, and so on but it can only travel at a certain speed to get there. In the long term I was hoping to plot this movement on a graph as I read a set of preset target positions from a file, but for now I'd settle for just numerically being able to do this kind of simulation. So visually, I'd have something like (using O for object, T for target position):

code:
---------O------------T--------- t = 0
------------O---------T--------- t = 1
---------------O------T--------- t = 2
------------------O---T--------- t = 3
---------------------O---------T t = 4
With a new target position being defined at t = 4 etc. and the potential for the new target position to be in the opposite direction.

Adbot
ADBOT LOVES YOU

fritz
Jul 26, 2003

Carrier posted:

I'm basically trying to simulate an object responding to commands to move to a certain position and I want to simulate some sort of delay in this response so that sometimes, it might not make it to the position its supposed to reach before it gets given another command. Sorta simulating like a car being told to go to point A, then to point B, and so on but it can only travel at a certain speed to get there. In the long term I was hoping to plot this movement on a graph as I read a set of preset target positions from a file, but for now I'd settle for just numerically being able to do this kind of simulation. So visually, I'd have something like (using O for object, T for target position):

code:
---------O------------T--------- t = 0
------------O---------T--------- t = 1
---------------O------T--------- t = 2
------------------O---T--------- t = 3
---------------------O---------T t = 4
With a new target position being defined at t = 4 etc. and the potential for the new target position to be in the opposite direction.

Maybe try a low pass filter, like an exponential smoother or something? Also try asking here: http://forums.somethingawful.com/showthread.php?threadid=3359430

  • Locked thread