|
oystertoadfish posted:gently caress p-values Fortran rules
|
#
?
Mar 10, 2016 15:56
|
|
|
|
| # ? Jun 9, 2024 06:21 |
|
|
It strikes me as very strange that anybody is using landline polling at this point -- even people who are post-cellphone age have been trained by decades of phone spam to screen calls. My 85-year-old parents screen calls. If you poll only people who will pick up the phone when their landphone rings, and stay on the line when they're told it's a survey -- also a technique used by spammers -- that's getting to be a rarer and rarer subset of Americans.
|
|
#
?
Mar 10, 2016 16:49
|
|
|
Arsenic Lupin posted:It strikes me as very strange that anybody is using landline polling at this point -- even people who are post-cellphone age have been trained by decades of phone spam to screen calls. My 85-year-old parents screen calls. If you poll only people who will pick up the phone when their landphone rings, and stay on the line when they're told it's a survey -- also a technique used by spammers -- that's getting to be a rarer and rarer subset of Americans. Yeah, landline-only polling made sense 8 years ago but it is TYOOL 2016 and I know all of three people with a landline phone, at this point they're generally something that only specific people in specific demographic groups have. It really doesn't make any sense to limit your polling to landlines.
|
|
#
?
Mar 10, 2016 18:26
|
|
|
Silver should have stopped using endorsements the moment Jeb! started polling under 5%, but he's still using them. When the polls plus more than doubles the predicted outcome for Marco loving Rubio (who's "winning" the "endorsement primary") to almost 30% likely win for Illinois, you really gotta just scrap that poo poo
|
|
#
?
Mar 10, 2016 18:37
|
|
|
smokyprogg posted:Silver should have stopped using endorsements the moment Jeb! started polling under 5%, but he's still using them. When the polls plus more than doubles the predicted outcome for Marco loving Rubio (who's "winning" the "endorsement primary") to almost 30% likely win for Illinois, you really gotta just scrap that poo poo Silver's problem is that he is too rigid with his analysis to write off endorsements as being irrelevant in this cycle, because endorsements mattered a lot (to his perception) in every previous election that polling was done in, so clearly it has to matter here as well. Ironically, despite the internet being one of the main reasons he is famous, he has completely failed to account for the impact that the internet has on campaigning and on the election cycle In an age where people can get all the information they need to make up their own minds, they no longer need a preferred politician or union to tell them how to vote.
|
|
#
?
Mar 10, 2016 18:40
|
|
|
Mirthless posted:Yeah, landline-only polling made sense 8 years ago but it is TYOOL 2016 and I know all of three people with a landline phone, at this point they're generally something that only specific people in specific demographic groups have. It really doesn't make any sense to limit your polling to landlines. Make that four.  We use it as a spam trap for businesses. Anything personal that is *not* from our parents goes to the appropriate cell.
|
|
#
?
Mar 10, 2016 18:49
|
|
|
I hope Silver completely blows the election and the Unskewed polls guy just loving drills it and me and him will high-five and talk about how Silver is a bitch-rear end pussy while Silver cries into his spreadsheets as an discredited stupid dumb fraud who sucks at life and is bad.
|
|
#
?
Mar 10, 2016 18:54
|
|
|
Goetta posted:Fortran rules some of my work is in a model that's basically a quarter century's worth of one guy's stream of consciousness in Fortran. there are random exceptions and data from studies from the eighties up to a few years ago scattered throughout the source code and 90 code is shoved in the middle of blocks of 77. it's like some kind of historical forensics or archaeology just trying to understand what's going on i like the computer history aspect of Fortran but i did my phd in r btw this might be the thread to whine about it, but unskewing was such a silly thing for that one guy we've been making fun of for four years to call his fat fingered adjustments. did he ever actually correct for a skew so the distribution was symmetric about the mean, or was he just using it in the colloquial sense the whole time? sometimes distributions have non zero skew, actually practically all of them do, it's OK! like the distribution of states or congressional districts by partisanship for example
|
|
#
?
Mar 10, 2016 18:54
|
|
|
then I'll kick his rear end e - goddammit
|
|
#
?
Mar 10, 2016 18:56
|
|
|
oystertoadfish posted:some of my work is in a model that's basically a quarter century's worth of one guy's stream of consciousness in Fortran. there are random exceptions and data from studies from the eighties up to a few years ago scattered throughout the source code and 90 code is shoved in the middle of blocks of 77. it's like some kind of historical forensics or archaeology just trying to understand what's going on The ~methodology~ behind UnSkewed Polls was: "I think there are too many Democrats in this sample. I'm going to reweigh the polls so that the electorate looks like 2010."
|
|
#
?
Mar 10, 2016 19:01
|
|
|
luv dat methodology
|
|
#
?
Mar 10, 2016 19:07
|
|
|
Pinterest Mom posted:The ~methodology~ behind UnSkewed Polls was: "I think there are too many Democrats in this sample. I'm going to reweigh the polls so that the electorate looks like 2010." Also boosted's unskewing spreadsheets just took an additional flat 5 or 10% off every state contest to account for media bias or some stupid bullshit
|
|
#
?
Mar 10, 2016 19:24
|
|
|
Romney and the GOP leadership all apparently believed that guy is the funniest thing to me. Some real wizard behind the curtain poo poo that summed up that campaign nicely.oystertoadfish posted:some of my work is in a model that's basically a quarter century's worth of one guy's stream of consciousness in Fortran. there are random exceptions and data from studies from the eighties up to a few years ago scattered throughout the source code and 90 code is shoved in the middle of blocks of 77. it's like some kind of historical forensics or archaeology just trying to understand what's going on That's cool, it sounds like we do sort of similar things.
|
|
#
?
Mar 10, 2016 21:26
|
|
|
I've done it
|
|
#
?
Mar 10, 2016 23:25
|
|
|
Calling cellphones has become prohibitively expensive to the point where reaching 18-30 year olds, Bernie's strong demographic, is very difficult. Unless the law changes, doing public polls over the phone are going to get less and less representative. And even if it changes, people are answering their phone less and less. Accurate polling is a huge challenge now and I think it will only get worse until the unlikely event a much larger share of the population is on online panels or people younger than 30 stop screening their calls.
|
|
#
?
Mar 11, 2016 00:37
|
|
|
Has internet polling improved at all and can it realistically replace traditional phone polling? That seems like the only option at this point.
|
|
#
?
Mar 11, 2016 02:04
|
|
|
MaxxBot posted:Has internet polling improved at all and can it realistically replace traditional phone polling? That seems like the only option at this point. I think if Facebook ever decided to move beyond its product-survery stuff and into political polling, it could provide some very accurate data. Unfortunately, political polling isn't nearly as profitable.
|
|
#
?
Mar 11, 2016 02:52
|
|
|
speaking from a social science rather than polling background i'm pretty sure internet surveys and polls had great response rates when they were new (because people thought using a computer made something more "official" or scientific), became horrible with the onslaught of spam and bullshit most people put up with online including needing to get around increasingly sophisticated filters, and have never really recovered; big social media sites like facebook and twitter would be amazing places to conduct research but they're very resistant to people outside their companies using their platforms that way
|
|
#
?
Mar 11, 2016 03:01
|
|
|
MaxxBot posted:Has internet polling improved at all and can it realistically replace traditional phone polling? That seems like the only option at this point. There are internet polling firms that get weighed into the averages on RCP, 538, etc. Zogby and Google Consumer Surveys for instance. These have not been very active in calling the state-by-state races though. They only do national polls. IBD uses cellphones as well as landlines and has been more accurate than the landline-only polls thus far, though still off quite a bit at times.
|
|
#
?
Mar 11, 2016 06:01
|
|
|
NNick posted:Calling cellphones has become prohibitively expensive to the point where reaching 18-30 year olds, Bernie's strong demographic, is very difficult. Unless the law changes, doing public polls over the phone are going to get less and less representative. Seriously, I'm not even that young anymore at 28, and I haven't had a landline for over 2 years now.
|
|
#
?
Mar 11, 2016 10:14
|
|
|
the fact that nate silver built an excel file that took 10 minutes to load as a statistical analysis tool makes him look like a giant fool to me saying "i'm not a programmer" has it's limits. when he got to the point he did with his excel files, it's time to become a programmer and actually use appropriate tools
|
|
#
?
Mar 11, 2016 10:55
|
|
|
Arsenic Lupin posted:It strikes me as very strange that anybody is using landline polling at this point -- even people who are post-cellphone age have been trained by decades of phone spam to screen calls. My 85-year-old parents screen calls. If you poll only people who will pick up the phone when their landphone rings, and stay on the line when they're told it's a survey -- also a technique used by spammers -- that's getting to be a rarer and rarer subset of Americans. In the last UK election the polls got it wrong, and the general explanation that's been pushed is that voters that were slightly more difficult to reach got written off too quickly, and those tended to be Conservative voters. So the pollsters tried to contact 2000 people, actually contacted 1600 and called it good, whereas trying to contact 800 and not stopping until they'd actually contacted all of them would have given a more accurate picture. But part of why that explanation has gotten so much coverage is because it pushes the message "left-wing voters were all at home on welfare, while right-wing voters were out working  ". The Tories new mass data, highly targeted campaigning having a disproportionate but subtle effect was probably a much bigger factor.
|
|
#
?
Mar 11, 2016 13:33
|
|
|
Condiv posted:the fact that nate silver built an excel file that took 10 minutes to load as a statistical analysis tool makes him look like a giant fool to me Spreadsheets that take 10 minutes to load aren't born that way, but are a running series of things getting bolted on. It would have been quite literally more than double the work to recreate it as a program, especially if the person has no idea how to code.
|
|
#
?
Mar 11, 2016 13:42
|
|
|
Condiv posted:the fact that nate silver built an excel file that took 10 minutes to load as a statistical analysis tool makes him look like a giant fool to me Its kinda adorable that you think this is rare and Nate Silver is somehow extra dumb to end up with a stupidly bloated excel file. ^^ yep. By the time anyone realises what's happened it's way too late.
|
|
#
?
Mar 11, 2016 14:56
|
|
|
jojoinnit posted:Its kinda adorable that you think this is rare and Nate Silver is somehow extra dumb to end up with a stupidly bloated excel file. it not being rare doesn't make it an idiot move. excel is obviously a bad fit for this kind of problem and he blew it off with "i'm not a programmer". he should've learned instead of creating yet another shitheap. Oil! posted:Spreadsheets that take 10 minutes to load aren't born that way, but are a running series of things getting bolted on. It would have been quite literally more than double the work to recreate it as a program, especially if the person has no idea how to code. of course it was a "simple" thing that had poo poo bolted on until it was unmanageable. but part of making any kind of algorithm is realizing when you need to redesign because your problem space has outgrown your current design also, the "double the work to recreate it" point is kind of silly. yes it takes double the work to start doing things the right way instead of hacking on what you already have until it barely does what you want, but the longer you wait the harder it becomes and the harder it is to actually improve your work Condiv has issued a correction as of 15:55 on Mar 11, 2016 |
|
#
?
Mar 11, 2016 15:53
|
|
|
From the PYF awful/funny graphs and charts thread:Judge Schnoopy posted:Here have this abomination, where the scale is so bad it doesn't even encompass all (perceived and non-data based) points
|
|
#
?
Mar 11, 2016 17:51
|
|
|
A good buddy of mine used to be an accountant. I remember years ago when he was asking about getting to borrow to a faster computer, because he was trying to do with with an excel spreadsheet. This was back in 2007, and his requirements were that it have Excel 07 on it, since that was the version where Excel boosted the row limit past 65,000. See, this spreadsheet had several hundred thousand rows in it. He also said the spreadsheet had "over a million large formulas" in it as well (this is why he needed the larger computer, his laptop couldn't actually open the thing anymore). After he asked that, we proceeded to mock him mercilessly for not switching over to a database. I believe one exact reply was "When a spreadsheet gets that complicated or huge, it's time to recognize that it's become something else entirely, bite the bullet, and pay someone to build the database application it has become."
|
|
#
?
Mar 11, 2016 18:09
|
|
|
technical debt is a pretty common issue in software projects big or small to point to something close to hand, the forums are a good example of codebases tending to accrete rather than getting re-written from scratch periodically
|
|
#
?
Mar 11, 2016 22:22
|
|
|
i watched this thread be closed, moved to SAS, and then somehow come back. what's happening
|
|
#
?
Mar 11, 2016 22:37
|
|
|
Condiv posted:the fact that nate silver built an excel file that took 10 minutes to load as a statistical analysis tool makes him look like a giant fool to me or use some of that ESPN cash to hire someone to program instead of sending Clare Malone to Oklahoma for little to no reason.
|
|
#
?
Mar 11, 2016 22:38
|
|
|
shrike82 posted:technical debt is a pretty common issue in software projects big or small i'm in the middle of rewriting a codebase from scratch because it's poo poo cobbled together with bash scripts rewriting stuff is not a cardinal sin when stuff doesn't work or works very poorly. in this case, excel is a poor replacement for R or other languages with statistics libraries because it's much slower and can only handle a fraction of the computational load. when it started maxxing his computer out for hours he should've moved to a language that would finish calculations in minutes Condiv has issued a correction as of 22:59 on Mar 11, 2016 |
|
#
?
Mar 11, 2016 22:57
|
|
|
nah, it's pretty common at trading desks for example to use Excel spreadsheets backed by C libraries to build pricing models probably more complex than PECATO does
|
|
#
?
Mar 11, 2016 23:05
|
|
|
shrike82 posted:nah, it's pretty common at trading desks for example to use Excel spreadsheets backed by C libraries to build pricing models
|
|
#
?
Mar 11, 2016 23:07
|
|
|
Pinely posted:or use some of that ESPN cash to hire someone to program instead of sending Clare Malone to Oklahoma for little to no reason. listen Nate Statboy Silver can either pay someone to code a database for his stupid wrong formulas or he can try to find out the BEST BURRITO IN AMERICA and keep his spergsheets I mean priorities man
|
|
#
?
Mar 11, 2016 23:35
|
|
|
5-turdy-8
|
|
#
?
Mar 11, 2016 23:35
|
|
|
e_angst posted:A good buddy of mine used to be an accountant. I remember years ago when he was asking about getting to borrow to a faster computer, because he was trying to do with with an excel spreadsheet. This was back in 2007, and his requirements were that it have Excel 07 on it, since that was the version where Excel boosted the row limit past 65,000. See, this spreadsheet had several hundred thousand rows in it. He also said the spreadsheet had "over a million large formulas" in it as well (this is why he needed the larger computer, his laptop couldn't actually open the thing anymore). I actually build excel macros for banks and funds and such. Yes, Excel is absolutely not ideal for a lot high end database work, and you have to hack the poo poo out it sometime to do basic things But the one feature it has, which will keep me employed even when robots take the other white collar jobs, is that it is the final word in ubiquity. Both in the sense that everyone has excel and also that everyone knows how to use it a bit. The power ofubiquity can't be stressed enough. There are projects I know will be green lighted because I can build the reports and database , painfully, in Excel that wouldn't fly if even loving MS Access was needed.
|
|
#
?
Mar 12, 2016 00:26
|
|
|
Condiv posted:i'm in the middle of rewriting a codebase from scratch because it's poo poo cobbled together with bash scripts Ok, but imagine this is the scenario - millions of people are reading your projections on a daily basis, and you can't afford to take that time to learn a new language and refactor things - especially when you've never formally programmed anything like that before. You simply don't have the time, it's not there. What do you do? You continue along with your half-baked thing that works and besides which it's too late to do much about. Just look at how many websites are built on crap. SA makes heavy use of php, an abomination. So does wikipedia. I'd think Silver probably eventually did what you're suggesting. But real world constraints are A Thing.
|
|
#
?
Mar 12, 2016 00:58
|
|
|
Broken Machine posted:Ok, but imagine this is the scenario - millions of people are reading your projections on a daily basis, and you can't afford to take that time to learn a new language and refactor things - especially when you've never formally programmed anything like that before. You simply don't have the time, it's not there. What do you do? You continue along with your half-baked thing that works and besides which it's too late to do much about. Just look at how many websites are built on crap. SA makes heavy use of php, an abomination. So does wikipedia. oddly enough, taking the time to rewrite it would free a poo poo ton of time to do projections. you know, since the excel file made his computer unusable for hours (and had to be restarted sometimes due to crashing). and like someone mentioned, he didn't exactly have to do it himself. in any case, it's pretty sad for a "statistician" to not know R these days. my biologist coworkers who need me to fix their computers daily all know R
|
|
#
?
Mar 12, 2016 01:15
|
|
|
Condiv posted:oddly enough, taking the time to rewrite it would free a poo poo ton of time to do projections. you know, since the excel file made his computer unusable for hours (and had to be restarted sometimes due to crashing). and like someone mentioned, he didn't exactly have to do it himself. in any case, it's pretty sad for a "statistician" to not know R these days. my biologist coworkers who need me to fix their computers daily all know R I'm just saying, if you look at all of the popular technology things, most any of them have architectural cruft that happens due to growing pains coupled with bad decisions. It's why linux is rewriting init and displaying graphics is still problematic.
|
|
#
?
Mar 12, 2016 01:50
|
|
|
|
| # ? Jun 9, 2024 06:21 |
|
|
The thing is, Nate Silver was the brand. Annoying Nate Silver in any way increased the risk that he would become unavailable. If the 400-pound gorilla wants to write in Word*Star, everybody else puts up with the file formats.
|
|
#
?
Mar 12, 2016 04:41
|
|