|
Your indentation is still hosed up.
|
# ? Sep 18, 2011 18:56 |
|
|
# ? May 9, 2024 19:00 |
|
In case it's not clear to you how important indentation is in Python, this works: code:
code:
|
# ? Sep 18, 2011 19:07 |
|
tripwire posted:Your indentation is still hosed up. Janin posted:In case it's not clear to you how important indentation is in Python, Thank you! Fixed indentation. ugh. http://pastebin.com/f8ikFEzt Updated. New error: code:
|
# ? Sep 18, 2011 19:08 |
|
You don't seem to know anything about Python. That's not a criticism, just an observation. Because of this, you do not know how to debug problems. Therefore, you need to change how to approach your task, to minimize how many problems you encounter. Download a fresh copy of ols.py. Make one change at a time, then run it. If it stops working, then undo your last change and try again. This will be slow and tedious, but will allow you to work with a language you do not know how to use. (previous draft posted for possible information) --------------- You have to paste the whole error, not just random bits of it. code:
code:
if I uncomment the first line and remove the x[..]= stuff, I get this error code:
|
# ? Sep 18, 2011 19:21 |
|
IMlemon posted:Hey, Sounds like a PYTHONPATH issue. http://docs.python.org/tutorial/modules.html#the-module-search-path By default, the current working directory is included in the module search path, so it will find libtcodpy. When you move the file to the tests directory, the current working directory no longer includes libtcodpy.
|
# ? Sep 18, 2011 19:22 |
|
Alright, new information and new problems! Code: http://pastebin.com/CSG72kgF the formula may be wrong, but i'm not positive, because it's not yielding the correct results from x1, x2, and x3 when i run it. it's numpy, scipy, and OLS if you are familiar. The program is yielding results like this: ============================================================================== Dependent Variable: y Method: Least Squares Date: Sun, 18 Sep 2011 Time: 14:53:32 # obs: 36 # variables: 4 ============================================================================== variable coefficient std. Error t-statistic prob. ============================================================================== const 2.632107 6.527869 0.403211 0.689475 x1 -0.000025 0.005023 -0.005031 0.996017 x2 -0.000020 0.000058 -0.343861 0.733200 x3 0.002403 0.000505 4.760467 0.000040 ============================================================================== Models stats Residual stats ============================================================================== R-squared 0.745477 Durbin-Watson stat 2.422127 Adjusted R-squared 0.721616 Omnibus stat 66.159622 F-statistic 31.241855 Prob(Omnibus stat) 0.000000 Prob (F-statistic) 0.000000 JB stat 710.782536 Log likelihood -104.954015 Prob(JB) 0.000000 AIC criterion 6.053001 Skew 4.150146 BIC criterion 6.228947 Kurtosis 23.123622 ============================================================================== and these numbers don't fit in correctly when I do the math out by hand. For example, rsquare is .9857, which is wrong on my chart. Any ideas? Thanks.
|
# ? Sep 18, 2011 20:30 |
|
Use PDB to step through your program and look for where the values are different from what you are calculating.
|
# ? Sep 19, 2011 00:24 |
|
My testing thinger has reached v1.1.4. Barring outside suggestions and/or bugfixes, this is likely to be the last version for a good long while, as my personal design goals/needs are now met. The best thing to happen since I first wrote about it here is the tutorial. The second-best thing is the ability to test "loosely" against regexes anywhere, including arbitrary leafnodes of structures which are being compared deeply. I'd appreciate feedback on the tutorial, as I finished writing it late last night. I know I need to get some editorial distance, then come back and clean it up.
|
# ? Sep 21, 2011 00:50 |
|
I like the >>> ABORT >>> ABORT >>> ABORT for test run failure You should find a way to mike sirens go off and lights start flashing
|
# ? Sep 21, 2011 02:12 |
|
tripwire posted:I like the Heh There have been a couple of times in the past when I have made a change to code, rerun my test suite, and been so sure that it would be fine that my eye skimmed right over test failure messages. There is no certain guard against complacency, but I tried to style failure messages to stand out in comparison to the "smallness" of a clean run, and tried to ensure that an entire script blowing up had a different enough look to be downright visually jarring.
|
# ? Sep 21, 2011 03:42 |
|
So why wouldn't something like this work?code:
code:
I can get it work by doing something like this... code:
|
# ? Sep 21, 2011 04:28 |
|
Either put it in parenthesis, or use line continuations.code:
code:
|
# ? Sep 21, 2011 04:42 |
|
Janin posted:Either put it in parenthesis, or use line continuations. karma_coma's second example works just fine. the first one is missing a token in the inner comprehension.
|
# ? Sep 21, 2011 04:44 |
|
Bear with me here people, dont raise the pitch forks too easily. I am still a beginner to this all, going through and actually STUDYING and learning this time as opposed to finding the print statement and making 100 line programs comprised of nothing but print("my housemate is a chav"). As I study each chapter I have a book and I write down what I feel I have learned from the chapter. So uses of functions, string methods, operators and their usages. Basic stuff. It might sound overkill to some of you but I can be very unfocused and every time I have tried to learn I fail cause I tend to not commit things to memory, suddenly I get lost in it and I give up but not this time! Anyhow two actual questions, I am going about this the right way as I have a few programmers as friends and I kind of feel they are suggesting my methods are unrequired as they like to state "they just picked it up and ran with it". If there is a better method I would love to hear it. Secondly I am playing around with better editors, not that I hate idle and I notice quite a few of them auto complete some of what I write. IE auto inserting and completing "" and (). Should I disable this? I feel if I get into the habit of letting it do it for me I may forget its significance. Also for learning I am following along with: Python Programming for the Absolute Beginner, 3rd Edition - Michael Dawson I feel it is quite good and one of the first that has really "reached" me. I just seem to get it.
|
# ? Sep 21, 2011 04:46 |
|
tripwire posted:karma_coma's second example works just fine. the first one is missing a token in the inner comprehension. gently caress, you're right. I've got problems elsewhere.
|
# ? Sep 21, 2011 04:58 |
|
tripwire posted:No, the square bracket of a list is an implied parenthesis.
|
# ? Sep 21, 2011 05:18 |
|
insidius posted:Anyhow two actual questions, I am going about this the right way as I have a few programmers as friends and I kind of feel they are suggesting my methods are unrequired as they like to state "they just picked it up and ran with it". If there is a better method I would love to hear it. That does sound a little extreme but I wouldn't listen to your friends. A lot of experienced programmers have forgotten what it's like to be a beginner. Does the book you're using have exercises? If not, I'd say either get one that does or keep doing the reviewing thing you're doing, because actively writing something is so much better than just passively reading. quote:Secondly I am playing around with better editors, not that I hate idle and I notice quite a few of them auto complete some of what I write. IE auto inserting and completing "" and (). Should I disable this? I feel if I get into the habit of letting it do it for me I may forget its significance. I say disable the auto-complete. The more you have to type yourself, the more you will retain and understand. You will know when you're at the point where typing all that stuff yourself is just extra work.
|
# ? Sep 21, 2011 05:25 |
|
insidius posted:learning techniques I always say the only way to learn is to write lots of programs. Taking notes is good, but it won't give you the experience that typing an actual program and seeing how it works. You say you don't want to just write a program with print statements, but that's going to be your first program in a new language regardless of how much you know about programming. When you learn something else, see if you can incorporate it into a more complicated program.
|
# ? Sep 21, 2011 07:49 |
|
Eggnogium posted:That does sound a little extreme but I wouldn't listen to your friends. A lot of experienced programmers have forgotten what it's like to be a beginner. When I was learning to program a few years ago, my most valuable advice came from people who were just past the learning stage I was at. A great majority of people who had been programming for years didn't seem to remember what it was like.
|
# ? Sep 21, 2011 16:23 |
|
The Gnome posted:the horror Can you post the stack trace. Also code properly formatted php:<? from __future__ import division from scipy import c_, ones, dot, stats, diff from scipy.linalg import inv, solve, det from numpy import log, pi, sqrt, square, diagonal from numpy.random import randn, seed import time class ols: """ Author: Vincent Nijs (+ ?) Email: v-nijs at kellogg.northwestern.edu Last Modified: Mon Jan 15 17:56:17 CST 2007 Dependencies: See import statement at the top of this file Doc: Class for multi-variate regression using OLS For usage examples of other class methods see the class tests at the bottom of this file. To see the class in action simply run this file using 'python ols.py'. This will generate some simulated data and run various analyses. If you have rpy installed the same model will also be estimated by R for confirmation. Input: y = dependent variable y_varnm = string with the variable label for y x = independent variables, note that a constant is added by default x_varnm = string or list of variable labels for the independent variables Output: There are no values returned by the class. Summary provides printed output. All other measures can be accessed as follows: Step 1: Create an OLS instance by passing data to the class m = ols(y,x,y_varnm = 'y',x_varnm = ['x1','x2','x3','x4']) Step 2: Get specific metrics To print the coefficients: >>> print m.b To print the coefficients p-values: >>> print m.p """ def __init__(self,y,x,y_varnm = 'y',x_varnm = ''): """ Initializing the ols class. """ self.y = y self.x = c_[ones(x.shape[0]),x] self.y_varnm = y_varnm if not isinstance(x_varnm,list): self.x_varnm = ['const'] + list(x_varnm) else: self.x_varnm = ['const'] + x_varnm # Estimate model using OLS self.estimate() def estimate(self): # estimating coefficients, and basic stats self.inv_xx = inv(dot(self.x.T,self.x)) xy = dot(self.x.T,self.y) self.b = dot(self.inv_xx,xy) # estimate coefficients self.nobs = self.y.shape[0] # number of observations self.ncoef = self.x.shape[1] # number of coef. self.df_e = self.nobs - self.ncoef # degrees of freedom, error self.df_r = self.ncoef - 1 # degrees of freedom, regression self.e = self.y - dot(self.x,self.b) # residuals self.sse = dot(self.e,self.e)/self.df_e # SSE self.se = sqrt(diagonal(self.sse*self.inv_xx)) # coef. standard errors self.t = self.b / self.se # coef. t-statistics self.p = (1-stats.t.cdf(abs(self.t), self.df_e)) * 2 # coef. p-values self.R2 = 1 - self.e.var()/self.y.var() # model R-squared self.R2adj = 1-(1-self.R2)*((self.nobs-1)/(self.nobs-self.ncoef)) # adjusted R-square self.F = (self.R2/self.df_r) / ((1-self.R2)/self.df_e) # model F-statistic self.Fpv = 1-stats.f.cdf(self.F, self.df_r, self.df_e) # F-statistic p-value def dw(self): """ Calculates the Durbin-Waston statistic """ de = diff(self.e,1) dw = dot(de,de) / dot(self.e,self.e); return dw def omni(self): """ Omnibus test for normality """ return stats.normaltest(self.e) def JB(self): """ Calculate residual skewness, kurtosis, and do the JB test for normality """ # Calculate residual skewness and kurtosis skew = stats.skew(self.e) kurtosis = 3 + stats.kurtosis(self.e) # Calculate the Jarque-Bera test for normality JB = (self.nobs/6) * (square(skew) + (1/4)*square(kurtosis-3)) JBpv = 1-stats.chi2.cdf(JB,2); return JB, JBpv, skew, kurtosis def ll(self): """ Calculate model log-likelihood and two information criteria """ # Model log-likelihood, AIC, and BIC criterion values ll = -(self.nobs*1/2)*(1+log(2*pi)) - (self.nobs/2)*log(dot(self.e,self.e)/self.nobs) aic = -2*ll/self.nobs + (2*self.ncoef/self.nobs) bic = -2*ll/self.nobs + (self.ncoef*log(self.nobs))/self.nobs return ll, aic, bic def summary(self): """ Printing model output to screen """ # local time & date t = time.localtime() # extra stats ll, aic, bic = self.ll() JB, JBpv, skew, kurtosis = self.JB() omni, omnipv = self.omni() # printing output to screen print '\n==============================================================================' print "Dependent Variable: " + self.y_varnm print "Method: Least Squares" print "Date: ", time.strftime("%a, %d %b %Y",t) print "Time: ", time.strftime("%H:%M:%S",t) print '# obs: %5.0f' % self.nobs print '# variables: %5.0f' % self.ncoef print '==============================================================================' print 'variable coefficient std. Error t-statistic prob.' print '==============================================================================' for i in range(len(self.x_varnm)): print '''% -5s % -5.6f % -5.6f % -5.6f % -5.6f''' % tuple([self.x_varnm[i],self.b[i],self.se[i],self.t[i],self.p[i]]) print '==============================================================================' print 'Models stats Residual stats' print '==============================================================================' print 'R-squared % -5.6f Durbin-Watson stat % -5.6f' % tuple([self.R2, self.dw()]) print 'Adjusted R-squared % -5.6f Omnibus stat % -5.6f' % tuple([self.R2adj, omni]) print 'F-statistic % -5.6f Prob(Omnibus stat) % -5.6f' % tuple([self.F, omnipv]) print 'Prob (F-statistic) % -5.6f JB stat % -5.6f' % tuple([self.Fpv, JB]) print 'Log likelihood % -5.6f Prob(JB) % -5.6f' % tuple([ll, JBpv]) print 'AIC criterion % -5.6f Skew % -5.6f' % tuple([aic, skew]) print 'BIC criterion % -5.6f Kurtosis % -5.6f' % tuple([bic, kurtosis]) print '==============================================================================' if __name__ == '__main__': ########################## ### testing the ols class ########################## # creating simulated data and variable labels seed(1) data = randn(100,5) # the data array # intercept is added, by default m = ols(data[:,0],data[:,1:],y_varnm = 'y',x_varnm = ['x1','x2','x3','x4']) m.summary() # if you have rpy installed, use it to test the results have_rpy = False try: print "\n" print "="*30 print "Validating OLS results in R" print "="*30 import rpy have_rpy = True except ImportError: print "\n" print "="*30 print "Validating OLS-class results in R" print "="*30 print "rpy is not installed" print "="*30 if have_rpy: y = data[:,0] x1 = data[:,1] x2 = data[:,2] x3 = data[:,3] x4 = data[:,4] rpy.set_default_mode(rpy.NO_CONVERSION) linear_model = rpy.r.lm(rpy.r("y ~ x1 + x2 + x3 + x4"), data = rpy.r.data_frame(x1=x1,x2=x2,x3=x3,x4=x4,y=y)) rpy.set_default_mode(rpy.BASIC_CONVERSION) print linear_model.as_py()['coefficients'] summary = rpy.r.summary(linear_model) print summary ?> Well except for the <? tags it seems to insert o_O
|
# ? Sep 21, 2011 17:00 |
|
Man I tried to work out what the thing is doing but everytime I try, my eyes glaze over at the dense soup of inscrutable acronym variables. It's kind of funny that theres a variable called nobs though, so props for that.
|
# ? Sep 21, 2011 23:50 |
|
tripwire posted:Man I tried to work out what the thing is doing but everytime I try, my eyes glaze over at the dense soup of inscrutable acronym variables. Everytime I see this, I immediately say "gently caress you" out loud and close the editor/web page. Like you, I can't make it past more than a few lines without my eyes glazing over.
|
# ? Sep 22, 2011 03:41 |
|
Thermopyle posted:Everytime I see this, I immediately say "gently caress you" out loud and close the editor/web page. Like you, I can't make it past more than a few lines without my eyes glazing over. It's great when someone posts dense numeric code like that and goes "help, the results I'm getting aren't what I'm expecting".
|
# ? Sep 22, 2011 03:45 |
|
I know what the code is trying to do from school, but the only thing I can imagine he is doing is not creating his dependent/independent variable matrices correctly for input. I noticed in earlier code he posted that he might not have been using the X matrix correctly. Aside from that, I think he should check the answers in a stat program that has OLS built in to make sure he isn't making errors in his calculations when checking. Btw, those variable names do make more sense if you've done the stats/math in school.
|
# ? Sep 22, 2011 15:02 |
|
vikingstrike posted:I know what the code is trying to do from school, but the only thing I can imagine he is doing is not creating his dependent/independent variable matrices correctly for input. I noticed in earlier code he posted that he might not have been using the X matrix correctly. That's not an excuse for anyone to code that way. Programs should be written for people first. Who gains anything from calling a function for calculating model log-likelihood "ll"? Look at the PBRT book as an example of how to do math/sciencey code correctly. The variables are named descriptively and the code is astoundingly readable for being so numerically oriented.
|
# ? Sep 22, 2011 15:32 |
|
BeefofAges posted:For some reason all of the scientific code I've ever seen is like this. It's awful. having worked at a research dept at a university it appears that it's because alot of the guys who write this are scientists who are learning programming on the fly.
|
# ? Sep 22, 2011 15:53 |
|
I'm not defending the choice of variable names, but I'm just saying that if you understand the context of the programming it's not too bad. Representations like 'e' for residuals is how the math is written in textbooks/teachers everywhere. Also, what foobat says holds a lot of truth. When you get people that are learning to code on the fly and then they start teaching other people it doesn't lead to happy places. Just lots of confusion. vikingstrike fucked around with this message at 17:08 on Sep 22, 2011 |
# ? Sep 22, 2011 17:05 |
|
foobat posted:having worked at a research dept at a university it appears that it's because alot of the guys who write this are scientists who are learning programming on the fly. And for some reason a lot of novice programmers like to use abbreviated names wherever possible. I would rather type out/use autocomplete for the extra characters in log_likelihood instead of trying to remember all my shorthands.
|
# ? Sep 22, 2011 20:13 |
|
FoiledAgain posted:And for some reason a lot of novice programmers like to use abbreviated names wherever possible. I would rather type out/use autocomplete for the extra characters in log_likelihood instead of trying to remember all my shorthands. Not using an intellisense type editor? Years ago I was all about the short-hand to save the time it took to convert my thoughts into code, now it's all about first few letters tab, keep going...
|
# ? Sep 22, 2011 22:36 |
|
Does anybody know a good book for teaching Python to an experienced programmer? He's gotten pretty far on his own, but more modern/advanced language features (such as 'yield' and 'property') are causing some trouble. Apparently he's been reading Dive Into Python, which never updated past version 2.2. Someone in this thread recommended Learning Python and Learn Python the Hard Way as alternatives to DIP, but these are aimed at someone who is learning programming for the first time. I'd like something that assumes proficiency in a C-family (C/C++/Java) language and then goes from there.
|
# ? Sep 23, 2011 00:01 |
|
Janin posted:Does anybody know a good book for teaching Python to an experienced programmer? Why not just read the official documentation on these features?
|
# ? Sep 23, 2011 01:09 |
|
BeefofAges posted:Why not just read the official documentation on these features? Seconding this -- This, (plus reading this thread) is how I learned Python coming from a C/C++ and matlab... background.
|
# ? Sep 23, 2011 01:20 |
|
BeefofAges posted:Why not just read the official documentation on these features?
|
# ? Sep 23, 2011 02:47 |
|
Janin posted:Because that assumes you know what they're called, how to find the docs, and know what section of the docs they're like to be in. I wonder if "The Python Library by Example" would be sufficient to get him more familiar.
|
# ? Sep 23, 2011 03:15 |
|
Captain Capacitor posted:I wonder if "The Python Library by Example" would be sufficient to get him more familiar.
|
# ? Sep 23, 2011 03:28 |
|
Yeah that looks great! I hadn't seen that book before.
|
# ? Sep 23, 2011 03:37 |
|
MEAT TREAT posted:Yeah that looks great! I hadn't seen that book before. That's because Doug only published it recently AFAIK it's culled from a few years' worth of his related blog series. Addison-Wesley Developer's Library authors reprazent!
|
# ? Sep 23, 2011 04:11 |
|
bitprophet posted:That's because Doug only published it recently AFAIK it's culled from a few years' worth of his related blog series. I'm on my way to ordering myself a copy. I find I always have to go back to his blog to remind myself how to do certain things (I'm looking at you, urllib2!). m0nk3yz Once you remember this thread exists, would you mind adding TPLBE to the OP?
|
# ? Sep 23, 2011 12:46 |
|
It happens the otherway round as well, a programmer does some scientific work and end up being told they've totally missed the point and don't understand the science. I once suggested a form of pair programming with one researcher, one programmer. No one really got it and said no.
|
# ? Sep 23, 2011 13:48 |
|
|
# ? May 9, 2024 19:00 |
|
We should all be happy that it was nobs and not just n.
|
# ? Sep 23, 2011 15:12 |