Python

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

SurgicalOntologist: Jun 17, 2004

Baby Babbeh posted:

That... makes sense. I was overcomplicating this. It returns another df rather than changing it in place, right?

As Cingulate mentioned, that would be an in-place function call. But I thought I would chime in to show how to do it as a function returning a new df:

Python code:

new_df = df.where(df == 0, 1)

The semantics are kind of reversed, think of it like "keep the original where the condition is true, otherwise change the value", so the above is the equivalent of Cingulate's example df[df != 0] = 1.

Anyways, DataFrame.where (or more commonly for me Series.where) is one of the most common operations when I'm using method chaining.

# ? Jun 9, 2017 17:39

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 11:50

Cingulate: Oct 23, 2012; by Fluffdaddy

Interesting, I literally never use .where. Have to look into that.

Also :same:

ing the py3redirect

# ? Jun 9, 2017 18:59

accipter: Sep 12, 2003

While we are on the subject of documentation, I also use http://devdocs.io from time to time. You can specify what documentation and version you want to search, and it even downloads the documentation for offline use.

# ? Jun 9, 2017 22:46

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

accipter posted:

You can with py3redirect!

# ? Jun 9, 2017 23:59

LochNessMonster: Feb 3, 2005; I need about three fitty

I'm trying to do something with a counter where you can poll something for x time with a y interval.

Seems pretty straightforward, I had a bug in it so the while condition never got met (variable wasn't an integer but a string ). So I tested if my fix would work and I ran into this. For this project it doesn't really matter but I'm pretty curious where the 0.00000000000000004 comes from in the 3rd iteration and why it only increased 4.6 to 4.699999999999999.

code:

>>> time = 10
>>> freq = 1
>>> count_incr = int(freq) / int(time)
>>> while counter < time:
...   counter += count_incr
...   print(counter)
...
0.1
0.2
0.30000000000000004
0.4
0.5
0.6
0.7
0.7999999999999999
0.8999999999999999
0.9999999999999999
1.0999999999999999
1.2
1.3
1.4000000000000001
1.5000000000000002
1.6000000000000003
1.7000000000000004
1.8000000000000005
1.9000000000000006
2.0000000000000004
2.1000000000000005
2.2000000000000006
2.3000000000000007
2.400000000000001
2.500000000000001
2.600000000000001
2.700000000000001
2.800000000000001
2.9000000000000012
3.0000000000000013
3.1000000000000014
3.2000000000000015
3.3000000000000016
3.4000000000000017
3.5000000000000018
3.600000000000002
3.700000000000002
3.800000000000002
3.900000000000002
4.000000000000002
4.100000000000001
4.200000000000001
4.300000000000001
4.4
4.5
4.6
4.699999999999999
4.799999999999999
4.899999999999999
4.999999999999998
5.099999999999998
5.1999999999999975
5.299999999999997
5.399999999999997
5.4999999999999964
5.599999999999996
5.699999999999996
5.799999999999995
5.899999999999995
5.999999999999995
6.099999999999994
6.199999999999994
6.299999999999994
6.399999999999993
6.499999999999993
6.5999999999999925
6.699999999999992
6.799999999999992
6.8999999999999915
6.999999999999991
7.099999999999991
7.19999999999999
7.29999999999999
7.39999999999999
7.499999999999989
7.599999999999989
7.699999999999989
7.799999999999988
7.899999999999988
7.999999999999988
8.099999999999987
8.199999999999987
8.299999999999986
8.399999999999986
8.499999999999986
8.599999999999985
8.699999999999985
8.799999999999985
8.899999999999984
8.999999999999984
9.099999999999984
9.199999999999983
9.299999999999983
9.399999999999983
9.499999999999982
9.599999999999982
9.699999999999982
9.799999999999981
9.89999999999998
9.99999999999998
10.09999999999998

# ? Jun 10, 2017 10:43

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

Obviously there's something wrong with your megahurts, so you'll need a new processing unit.

It's because of floating point numbers:

http://floating-point-gui.de

e: You'll want to change the logic of that loop so that it counts by integers instead of floats. When you do the division part your integers are turning into floats.

e2: The overly simplified gist of it is that your computer does not know how to represent a value like 0.3 exactly, so it needs to approximate. That's what that extra 0.0...4 is coming from, and that's why some of your other values are actually less than what you are expecting. This is why the general wisdom is to never compare floating point numbers if the result is critical, e.g.:

code:


# this is bad
if a == b:
    foo()

# this is better
eps = 1e-8
if abs(a - b) < eps:
    foo()

Boris Galerkin fucked around with this message at 11:44 on Jun 10, 2017

# ? Jun 10, 2017 11:24

LochNessMonster: Feb 3, 2005; I need about three fitty

Boris Galerkin posted:

Obviously there's something wrong with your megahurts, so you'll need a new processing unit.

It's because of floating point numbers:

http://floating-point-gui.de

e: You'll want to change the logic of that loop so that it counts by integers instead of floats. When you do the division part your integers are turning into floats.

Never knew floating points behave like this, learning something knew everyday.

# ? Jun 10, 2017 11:36

Boris Galerkin: Dec 17, 2011; I don't understand why I can't harass people online. Seriously, somebody please explain why I shouldn't be allowed to stalk others on social media!

LochNessMonster posted:

Never knew floating points behave like this, learning something knew everyday.

It's more of a hardware limitation than an actual thing. In real life 0.3 is exactly 0.3, but we lose some accuracy when we represent it in a computer.

I imagine for most people none of this has any direct relevance, but if you're doing any sort of numerical/computational work then this type of stuff comes up all the time. Doing simulations for example we already accept that there is an inherent error from the actual math equations and simplifications to them, but we also accept that there are errors from discretization (dividing the problem up into multiple smaller parts), and we also must be aware of problems in roundoff/precision due to the computer. Lots of fields use 64 bit floats for example because it gives us more precision (more decimals).

I remember one of my earliest courses in this field our professor intentionally misled us to use MATLAB to write our first finite difference solver to solve a problem that produced nonsense results because the default floating point precision in MATLAB (at that time? not sure if it's the case still) was 32bit. Due to error propagation these errors eventually ruined the solution. Telling MATLAB (or using C/Fortran double precision numbers) to use 64 bit reals on the other hand fixed the problem because these errors never had a chance to propagate.

Boris Galerkin fucked around with this message at 12:03 on Jun 10, 2017

# ? Jun 10, 2017 11:59

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

LochNessMonster posted:

Never knew floating points behave like this, learning something knew everyday.

Did u think that 64 bits was enough to represent all of the reals?

# ? Jun 10, 2017 12:26

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

Boris Galerkin posted:

It's more of a hardware limitation than an actual thing. In real life 0.3 is exactly 0.3, but we lose some accuracy when we represent it in a computer.

I imagine for most people none of this has any direct relevance, but if you're doing any sort of numerical/computational work then this type of stuff comes up all the time. Doing simulations for example we already accept that there is an inherent error from the actual math equations and simplifications to them, but we also accept that there are errors from discretization (dividing the problem up into multiple smaller parts), and we also must be aware of problems in roundoff/precision due to the computer. Lots of fields use 64 bit floats for example because it gives us more precision (more decimals).

I remember one of my earliest courses in this field our professor intentionally misled us to use MATLAB to write our first finite difference solver to solve a problem that produced nonsense results because the default floating point precision in MATLAB (at that time? not sure if it's the case still) was 32bit. Due to error propagation these errors eventually ruined the solution. Telling MATLAB (or using C/Fortran double precision numbers) to use 64 bit reals on the other hand fixed the problem because these errors never had a chance to propagate.

Look at this scrub who doesn't compensate for floating pt errors and is agnostic to precision.

Mixed precision iterative refinement :hellyeah:

# ? Jun 10, 2017 12:28

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

Next y'all will be like "use doubles" or maybe even "use quads" and then come back like "my prog is mega slow"

# ? Jun 10, 2017 12:31

shrike82: Jun 11, 2005

Just use the Decimal class

# ? Jun 10, 2017 14:39

LochNessMonster: Feb 3, 2005; I need about three fitty

Malcolm XML posted:

Did u think that 64 bits was enough to represent all of the reals?

I didn't think about this at all, nor do I know what anything in your sentence means. Ignorance is bliss in some cases.

# ? Jun 10, 2017 18:04

QuarkJets: Sep 8, 2008

Malcolm XML posted:

Did u think that 64 bits was enough to represent all of the reals?

That has nothing to do with floating point errors though; exact Decimal types exist without having to represent all of the reals

# ? Jun 11, 2017 01:38

FAT32 SHAMER: Aug 16, 2012

Lol if u have to think about the 64th decimal place in your code just lol

# ? Jun 11, 2017 01:45

Malcolm XML: Aug 8, 2009; I always knew it would end like ｔｈｉｓ．

QuarkJets posted:

That has nothing to do with floating point errors though; exact Decimal types exist without having to represent all of the reals

but the point is that no finite representation can represent real numbers w/o some loss and imprecision, fixed point decimal does it differently than IEEE-754

arbitrary precision is a different game but still limited by practicality

# ? Jun 11, 2017 01:46

QuarkJets: Sep 8, 2008

Boris Galerkin posted:

It's more of a hardware limitation than an actual thing. In real life 0.3 is exactly 0.3, but we lose some accuracy when we represent it in a computer.

I imagine for most people none of this has any direct relevance, but if you're doing any sort of numerical/computational work then this type of stuff comes up all the time. Doing simulations for example we already accept that there is an inherent error from the actual math equations and simplifications to them, but we also accept that there are errors from discretization (dividing the problem up into multiple smaller parts), and we also must be aware of problems in roundoff/precision due to the computer. Lots of fields use 64 bit floats for example because it gives us more precision (more decimals).

I remember one of my earliest courses in this field our professor intentionally misled us to use MATLAB to write our first finite difference solver to solve a problem that produced nonsense results because the default floating point precision in MATLAB (at that time? not sure if it's the case still) was 32bit. Due to error propagation these errors eventually ruined the solution. Telling MATLAB (or using C/Fortran double precision numbers) to use 64 bit reals on the other hand fixed the problem because these errors never had a chance to propagate.

I don't think that it's accurate to call that a hardware limitation, otherwise Decimal couldn't exist

# ? Jun 11, 2017 01:48

Symbolic Butt: Mar 22, 2009; (_!_); Buglord

Malcolm XML posted:

Did u think that 64 bits was enough to represent all of the reals?

Malcolm XML is a smart guy but this post is dumb as gently caress.

I mean, people start programming and with some experience assume that "float" is just a fancy name for numbers that intuitively behave like fixed-point numbers. This reasoning deflects the fact that real numbers are infinite.

I don't think many introductory materials go into teaching the details of floating points, it's a somewhat low level thing. But it's too bad, because imo every introductory material should tackle a bit into the classic 0.1 + 0.2 = 0.30000000000000004 to clear out people's minds from intuitively assuming they're fixed-point.

# ? Jun 11, 2017 01:50

QuarkJets: Sep 8, 2008

Malcolm XML posted:

but the point is that no finite representation can represent real numbers w/o some loss and imprecision, fixed point decimal does it differently than IEEE-754

arbitrary precision is a different game but still limited by practicality

And I agree with that, just not the vaguer post that you made earlier (because you can crank up the precision and use clever software to circumvent many of the common problems in floating point arithmetic without having to represent all real numbers; for instance financial software can use the Decimal type instead of float, but Decimal itself does not represent all reals with arbitrary precision)

# ? Jun 11, 2017 02:11

FAT32 SHAMER: Aug 16, 2012

I was lost on the floating point topic until I found this on quora

quote:

Most answers here address this question in very dry, technical terms. I'd like to address this in terms that normal human beings can understand.

Imagine that you are trying to slice up pizzas. You have a robotic pizza cutter that can cut pizza slices exactly in half. It can halve a whole pizza, or it can halve an existing slice, but in any case, the halving is always exact.

That pizza cutter has very fine movements, and if you start with a whole pizza, then halve that, and continue halving the smallest slice each time, you can do the halving 53 times before the slice is too small for even its high-precision abilities. At that point, you can no longer halve that very thin slice, but must either include or exclude it as is.

Now, how would you piece all the slices in such a way that would add up to one-tenth (0.1) or one-fifth (0.2) of a pizza? Really think about it, and try working it out. You can even try to use a real pizza, if you have a mythical precision pizza cutter at hand. :-)

# ? Jun 11, 2017 02:29

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

It's like with decimal, you can do however many tenths and hundredths and so on you like, but there are some numbers you just can't represent perfectly, like 1/3. You can get close, and the more decimal places you have the less it matters, but eventually you run out and have to settle for 0.3333333, which is as accurate as you're gonna get in base 10

Base 2 has similar issues, just with different numbers - your fractions have to have a denominator that's a power of 2 instead of a power of 10, and that's a no-go way more often

# ? Jun 11, 2017 03:14

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

funny Star Wars parody posted:

Lol if u have to think about the 64th decimal place in your code just lol

lmao if you don't understand every abstraction layer right down to how the rock we shoot lightning into to do numbers does stuff

# ? Jun 11, 2017 04:12

baka kaba: Jul 19, 2003; PLEASE ASK ME, THE SELF-PROFESSED NO #1 PAUL CATTERMOLE FAN IN THE SOMETHING AWFUL S-CLUB 7 MEGATHREAD, TO NAME A SINGLE SONG BY HIS EXCELLENT NU-METAL SIDE PROJECT, SKUA, AND IF I CAN'T PLEASE TELL ME TO
EAT SHIT

When you get down to the low levels and it's all discrete values like energy quanta... can some numbers even exist? #whoa

I say we ban these false numbers

# ? Jun 11, 2017 04:45

FAT32 SHAMER: Aug 16, 2012

what if quantum computer creates a black hole right in the middle of silicon valley?

it'd be a net positive

# ? Jun 11, 2017 05:26

Cingulate: Oct 23, 2012; by Fluffdaddy

Malcolm XML posted:

Did u think that 64 bits was enough to represent all of the reals?

Python is a language that's being used by people with very little CS knowledge (e.g., me). That's, from what I can tell, by design: Python is easy to learn, easy to read, welcoming and forgiving. I think the thread has a good track record of being exactly like that, too.

# ? Jun 11, 2017 14:13

Dominoes: Sep 20, 2007

Your avatar is ridiculous.

# ? Jun 11, 2017 15:00

KernelSlanders: May 27, 2013; Rogue operating systems on occasion spread lies and rumors about me.

funny Star Wars parody posted:

Lol if u have to think about the 64th decimal place in your code just lol

Dex posted:

lmao if you don't understand every abstraction layer right down to how the rock we shoot lightning into to do numbers does stuff

Not everyone in this thread builds web apps.

# ? Jun 11, 2017 16:30

Dex: May 26, 2006; Quintuple x!!!

Would not escrow again.

VERY MISLEADING!

nobody mentioned web apps

# ? Jun 11, 2017 16:58

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

KernelSlanders posted:

Not everyone in this thread builds web apps.

This thread has always been open to beginners. The poster who has been asking questions is obviously a beginner. Floating point stuff obviously is weird to beginners.

Thermopyle fucked around with this message at 17:44 on Jun 11, 2017

# ? Jun 11, 2017 17:21

FAT32 SHAMER: Aug 16, 2012

KernelSlanders posted:

Not everyone in this thread builds web apps.

# ? Jun 11, 2017 18:15

Eela6: May 25, 2007; Shredded Hen

Thermopyle posted:

This thread has always been open to beginners. The poster who has been asking questions is obviously a beginner. Floating point stuff obviously is weird to beginners.

Numerical Analysis is not an easy subject and it frustrates me when people pretend it's simple & intuitive.

# ? Jun 11, 2017 19:21

Nippashish: Nov 2, 2005; Let me see you dance!

Eela6 posted:

Numerical Analysis is not an easy subject and it frustrates me when people pretend it's simple & intuitive.

In the vast (vast) majority of cases they are in fact very simple and intuitive. It's extremely rare to need a mental model of floating point that is more sophisticated than "they're like real numbers except that they get funny when they're really big or really small" even if you work with them every day.

# ? Jun 11, 2017 19:41

Eela6: May 25, 2007; Shredded Hen

Nippashish posted:

In the vast (vast) majority of cases they are in fact very simple and intuitive. It's extremely rare to need a mental model of floating point that is more sophisticated than "they're like real numbers except that they get funny when they're really big or really small" even if you work with them every day.

Python code:

In [8]: .3/3 == .1
Out[8]: False

# ? Jun 11, 2017 19:45

Nippashish: Nov 2, 2005; Let me see you dance!

Python code:

>>> abs(.3 / 3 - 0.1) < 1e-12
True

They're weird when they're small. One of the implications is you shouldn't expect exact equality.

# ? Jun 11, 2017 19:52

Thermopyle: Jul 1, 2003; ...the stupid are cocksure while the intelligent are full of doubt. �Bertrand Russell

Nippashish posted:

Python code:
>>> abs(.3 / 3 - 0.1) < 1e-12
True
They're weird when they're small. One of the implications is you shouldn't expect exact equality.

That's why its confusing to beginners.

# ? Jun 11, 2017 19:53

Nippashish: Nov 2, 2005; Let me see you dance!

Thermopyle posted:

That's why its confusing to beginners.

That's why I'm suggesting a very simple mental model that is intuitive and also sufficient for even non-beginners. Teaching people that floating point numbers are dark and spooky and complicated isn't very productive, because very few people need to care about that level of detail.

I'm responding to "Numerical Analysis is not an easy subject and it frustrates me when people pretend it's simple & intuitive" by pointing out that for most practical purposes it can be made to be exactly that.

# ? Jun 11, 2017 19:58

Eela6: May 25, 2007; Shredded Hen

Nippashish posted:

That's why I'm suggesting a very simple mental model that is intuitive and also sufficient for even non-beginners. Teaching people that floating point numbers are dark and spooky and complicated isn't very productive, because very few people need to care about that level of detail.

I'm responding to "Numerical Analysis is not an easy subject and it frustrates me when people pretend it's simple & intuitive" by pointing out that for most practical purposes it can be made to be exactly that.

I'm not against teaching people rules of thumb, but pretending that they're more than just rules of thumb is dumb. An important part of being a professional is knowing the limits of your knowledge.

# ? Jun 11, 2017 20:05

LochNessMonster: Feb 3, 2005; I need about three fitty

Nippashish posted:

That's why I'm suggesting a very simple mental model that is intuitive and also sufficient for even non-beginners. Teaching people that floating point numbers are dark and spooky and complicated isn't very productive, because very few people need to care about that level of detail.

As the root cause of this discussion as well as a beginner in terms of programming I can tell you that you're over simplifying things.

Before this discussion I had no clue that floats would give irregular results. I was trying to do some really basic math functions and they were off by 10-20%.

If I'm running into that in my 2nd project with basic funtionality I'd say it's defintely something almost everyone will care about rather sooner than later.

I'd rather have people tell me I should watch out with where I'm using floats (like they did, thank you) than tell me not to worry because I don't need to care about that level of detail.

# ? Jun 11, 2017 20:10

Eela6: May 25, 2007; Shredded Hen

As a note, the fact that approximate equality is often useful when comparing floats comes up often enough that isclose was added to the math and cmath modules in Python 3.5.

Python code:

In [13]: from math import isclose
    ...: isclose(.3/3, .1)
    ...:
Out[13]: True

# ? Jun 11, 2017 20:16

Adbot: ADBOT LOVES YOU

# ? May 15, 2024 11:50

Nippashish: Nov 2, 2005; Let me see you dance!

LochNessMonster posted:

If I'm running into that in my 2nd project with basic funtionality I'd say it's defintely something almost everyone will care about rather sooner than later.

I'd rather have people tell me I should watch out with where I'm using floats (like they did, thank you) than tell me not to worry because I don't need to care about that level of detail.

Floating point numbers are not real numbers, and you need to be aware of this when using them. The particular way in which they are not real numbers is quite complicated, and is rarely relevant. I offered some additional guidance on how to think about the relationship between floats and reals.

Floating point weirdness is also one of those topics that programmers love to expand on at length when it comes up (I'm slightly surprised no one has warned you about storing currency in floats yet) and my suggestion to "not worry about it" should be taken in the context of your post triggering a page of responses. The fact that you went away from the discussion with the impression that you should "watch out" when using floats is exactly what I was trying to mitigate.

# ? Jun 11, 2017 20:53

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > Python

«‹›230 »