Brain Candy posted:

if only java had immutable data structures with minimal performance costs, like, java.util.Collections.unmodifiableSet

Bloody posted:

as of this post i would up the punishment to 3 day probations for lawchat

Shaggar posted:

if the databases are on the same server, union the tables across all the databases.

Soricidus posted:

that's not an immutable data structure

Sweeper posted:

so i have a data set where it is a string and a count for some number of sets. i basically want to find the top 10 things by count in a union of all these sets. the problem is this data is very large and exists in multiple databases.... what should i be looking for? do i actually need to look through the entirety of each set to guarantee that nothing is missed?

Brain Candy posted:

totally is, if nobody gets to alter the backing set. maybe you think persistent is the same as immutable?

Kevin Mitnick P.E. posted:

java gets it right. except i guess in the case that you give command line arguments that can't be decoded by the platform default codec in which case you've earned whatever hell you find yourself in

Soricidus posted:

no, i think "if nobody gets to alter the backing set" is a very big if.

it solves one problem, i.e. passing someone a collection without copying it and knowing they won't modify it. but it doesn't solve other problems, such as being passed a collection without copying it and knowing the caller won't modify it.

Suspicious Dish posted:

As long as humans continue to exist, data will continue to be messy, and attempts to say "this is text and this is bytes" are naive and unrealistic, no matter how firmly you say it.

Python 3's Unicode support is truly embarrassing.

Notorious b.s.d. posted:

seriously though, counting things is the "hello world" for hadoop/cascading.

write an etl package to dump everything to hdfs or s3, spin up a few nodes, whomp.

Brain Candy posted:

you know its immutable because because that's what it said in the documentation/annotations. as long as non-language designers get to make new data structures, that's about where you are going to end up, +/- pretend guarantees like const

everything you point out about the data structure also applies to the things it contains. if you want to get a stronger guarantee than this, you'll need to stop people from making mutable things period

Sweeper posted:

too slow it is for a UI, no? It is also filterable so it's in a relational database atm. whoever did the setup decided that when they needed more space they would just stand up another pair of databases (master/slave) and now we have 7 pairs of databases and I want to be able to do joins on the data across them

Sweeper posted:

like I could have just dumped the data into dynamo and done scans on it, but it ends up being super loving expensive and a poo poo ton of full table scans to count the counts for ever column in the database.

Notorious b.s.d. posted:

there exist sharding proxies for mysql (tungsten) and postgres (pgpool) that let you use a subset of sql in parallel... but they're a pain in the butt

if your data is small enough, the cheapest/best answer is probably gonna be a DB server that is 10x as big.

off the shelf x86 will do several TB of RAM now


by definition a precise count requires a table scan. that's why map/reduce frameworks are good at counting: guaranteed ceiling on worst-case time and it is a simple aggregate function

you can use indices to get estimated counts on postgres/oracle but they're only estimates

Sweeper posted:

i'm thinking i could stop query after some count < x where x is probably the lowest in the top 10 counts. i was hoping for something simple dome smart math person came up with where i could be like 85% sure or w/e that there won't be any lower summed counts than that

no one is going to green light my one giant db plan sadly

Sweeper posted:

i'm thinking i could stop query after some count < x where x is probably the lowest in the top 10 counts. i was hoping for something simple dome smart math person came up with where i could be like 85% sure or w/e that there won't be any lower summed counts than that

Notorious b.s.d. posted:

if estimates are good enough, take a random sample and work with that data instead.

i'm no ace at statistics, but a sample of 1,000 adults is good enough to make high-confidence estimates about the behavior of 100 million voters, so a little thought could go a long way

Booblord Zagats posted:

I figure I should share a story my dad told me about idiot contractors. Not sure If I've told it on here before, but since he just told it again the other night when we were talking to him, I'll go ahead and tell it again because it's a good story.

When my dad first made E-8 they moved him out of the Army Space Program Office and to a DISA project clearing the handing over of training assets like flight simulators and MILES gear to US allies. In all it was a pretty sweet gig because he'd get to fly all over the world and chill out for a few weeks in places like Belgium, Italy, and Australia while making sure what had been handed over by US contractors was satisfactory from both an operational and security standpoint. It's probably the main reason he was hired by a 3-letter within 48 hours of retiring from the Army.

He ended up making friends with a bunch of Australian helicopter pilots due to his frequent trips to their base when they were getting upgraded Kiowa simulators. The contractor for the sims was one my dad hated working with though, because they half-assed every thing they touched, meaning he'd spend months sending them notes CC'd to DISA's chief which always led to a poo poo storm that was worse for him than the people who weren't doing their jobs. The Kiowa sims started off really lovely though, because the company making them just directly imported the software for USMC Mid 80's AH-1 Cobra simulators and just made it work in a faux Kiowa cockpit. Which really didn't work for a lot of reasons. 1) Australia didn't really need to train pilots in 1994 on the proper way to close the Fulda Gap and 2) the Kiowa and the Cobra don't handle anywhere near the same

So they (contractors)go in and after months of arguing with both DISA and the Australian pilots that a simulator that doesn't simulate the right thing is worthless finally make some of the needed updates. They fixed the handling characteristics (by hiring two engineers from Bell who designed the sim the US Army uses) and changing Central Germany to Australia by making the ground brown instead of green and throwing Ayers Rock where some of the German Alps had been. And just to be cute, they replaced random Soviet soldiers that would flee when they spotted the helicopter to Kangaroos just as an added "gently caress you, here's some added realism" poo poo.

The Australian pilots are pretty happy with the changes and spend the first few minutes flying from one side of their base to the other, attempting loops and all other manner of loving around. Then the Aussie lead pilot decides to fly out to Ayers Rock since one of the contractors had mentioned it. As he's flying out to it, he see's some movement on the ground and changes course to check it out. His copilot confirms its a pack of kangaroos so dude figures "gently caress it, lets buzz some kangaroos"

He drops down to 10 feet above the ground and flies over them as fast as he can while the Kangaroos scatter. Then as he pulls up, he realizes he has lost control of his tail rotor and his panels are lighting up like a Christmas tree. Dude tries to put it down gentle, but then gets informed he's been hit by heavy machine gun fire and the sim freezes. Everyone is dumbfounded by this.

Then one of the sim techs has a revelation, they had been shot down by the Kangaroos. After they buzzed them, one of the kangaroos fled to cover, then fired a shoulder launched SAM at them, hitting the tail rotor, while he tried to recover, some other kangaroos set up a 13.7mm gun and shot him up. Because the coders had only changed the graphics of the Soviet soldiers to Kangaroos and nothing else, meaning they still had a full platoon's worth of Soviet anti-helicopter weapons.


So if anything, it proved that everything in the Australian outback is designed to gently caress poo poo up.

Brain Candy posted:

everything you point out about the data structure also applies to the things it contains. if you want to get a stronger guarantee than this, you'll need to stop people from making mutable things period

Damiya posted:

js owns and node remains a great platform for scripts and tasks..

Just don't do server in it that's all

Soricidus posted:

yes? i agree. java already did this for strings and it worked out really well. taking it further would be great.

but that doesn't alter the only fact i'm arguing, which is that java does not provide immutable collections. all it provides is a way to create an immutable view of a mutable collection, which is plang level poo poo. it's not even visible in the type system, let alone statically checked.

Brain Candy posted:

totally is, if nobody gets to alter the backing set. maybe you think persistent is the same as immutable?

hackbunny posted:

I can't believe a clanger like me has to point this out, but that's totally not an immutable data structure. an immutable data structure has mutator methods, except instead of mutating the object they return a new, immutable object with the result of the mutation, which you can then operate on, or compare-and-swap with the original to make the result globally visible (the GC will then handle the clean-up of the original object, and as a clanger I'm terribly envious of this). usually, the structure of a true immutable data structure is specifically optimized for this kind of copy-on-write mutation, so that multiple objects resulting from the mutation of a common object can share most or all of the data that wasn't affected by the mutation. this kind of structure tends to look an awful lot like the call tree of a recursive algorithm, frozen in memory, and that's why immutable data structures are a gateway drug to functional programming

(flangers please don't take offense at my very clangerish view of immutable data structures)

ComradeCosmobot posted:

two words: surrogate characters

Malcolm XML posted:

yeah this is pretty much it

CoW is so amazingly powerful, systems peeps have been using it forever w.r.t virtual memory/fork etc

that + TCO basically making recursion costless is great

Cybernetic Vermin posted:

i'd have to dig up the post i made like a hundred pages ago about doing reference counting by doing lifetime management on the collections level to make this post complete, but one of the niceties of that type of approach is that you can have the collections mutate if they only have a single reference, while maintaining the appearance of immutability in all other cases. makes for a solid optimization while keeping things nice and simple

BONGHITZ posted:

Soricidus posted:

yes? i agree. java already did this for strings and it worked out really well. taking it further would be great.

but that doesn't alter the only fact i'm arguing, which is that java does not provide immutable collections. all it provides is a way to create an immutable view of a mutable collection, which is plang level poo poo. it's not even visible in the type system, let alone statically checked.

hackbunny posted:

I can't believe a clanger like me has to point this out, but that's totally not an immutable data structure. an immutable data structure has mutator methods, except instead of mutating the object they return a new, immutable object with the result of the mutation, which you can then operate on, or compare-and-swap with the original to make the result globally visible (the GC will then handle the clean-up of the original object, and as a clanger I'm terribly envious of this). usually, the structure of a true immutable data structure is specifically optimized for this kind of copy-on-write mutation, so that multiple objects resulting from the mutation of a common object can share most or all of the data that wasn't affected by the mutation. this kind of structure tends to look an awful lot like the call tree of a recursive algorithm, frozen in memory, and that's why immutable data structures are a gateway drug to functional programming

(flangers please don't take offense at my very clangerish view of immutable data structures)

Otto Skorzeny posted:

how about environments that have flaky TCE and detonate your stack when you expected you'd be fine

quote:

To configure the dealer, all we have to do is

JavaScript code:

app.config(function (CarProviderProvider) {
    CarProviderProvider.setDealerName('Good');
});

Hey, it's just config - no need to change any of the real code!!

I'd write a plain old JS equivalent but trying to wrap my head around all of the indirection in the above example is making me want to crawl under a desk and bang my head on the floor until the brainmeats come out so I don't have to subject myself to this madness any further.

quote:

“But, why CarProviderProvider instead of CarProvider”

Here's a tip. If you find yourself asking a question like this. If you find yourself asking a question which requires this sort of answer and then this sort of question to be asked YOU'VE DONE IT WRONG.

Max Facetime posted:

oh god I hope this hasn't been posted yet http://codeofrob.com/entries/you-have-ruined-javascript.html