Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
The Something Awful Forums > The Finer Arts > TV IV > Silicon Valley Season 4: Hards-On for everybody!
 
  • Locked thread
Kin
Nov 4, 2003

Sometimes, in a city this dirty, you need a real hero.
Could a compression algorithm like this theoretically be used to help process larger files quicker?

I currently deal with stupidly big data sets clocking in at over a gig for a csv and I can almost hear the excel reports that process this screaming at me as they slowly chug through the calculations.

* # ¿ May 4, 2016 19:38
  • Quote
Adbot
ADBOT LOVES YOU

# ¿ Apr 27, 2024 21:47
  • Quote
Kin
Nov 4, 2003

Sometimes, in a city this dirty, you need a real hero.

enki42 posted:

Depends on the bottleneck. If it's transferring the files over a network or something, maybe (this wouldn't be the case with CSV and excel reports). But you'd need to decompress it, at least in memory, to do anything useful with it.

In your case your problem is that you're using the wrong tool. 1 GB isn't anywhere close to the realm of "big data", but the strategy for anything involving CSV should be to first get it into something that can process it more effectively (excel is not that thing)

Oh yeah, don't get me wrong, i fully understand excel isn't the best tool for this (we typically use it as a test case before running it through our server), i'm just not familiar in the slightest with how processors handle/work through the data at a memory level and wasn't sure if there would be some nifty compression based mcguffin that would mean PiedPiper's application had the potential for much grander uses beyond data transfer and storage.

* # ¿ May 4, 2016 23:33
  • Quote
  • Locked thread
The Something Awful Forums > The Finer Arts > TV IV > Silicon Valley Season 4: Hards-On for everybody!