|
Can you just copy/paste your thing and put it in code tags?
|
#
?
May 15, 2013 07:33
|
|
|
|
| # ? May 29, 2024 16:10 |
|
|
tarepanda posted:Can you just copy/paste your thing and put it in code tags? code:
|
|
#
?
May 15, 2013 07:37
|
|
|
That would do it.
|
|
#
?
May 15, 2013 07:45
|
|
|
I need some pointers for where to go from here. Assume I'm a moron - anything I say that sounds knowledgeable is probably the only thing I know, and I may not really know it. I constructed an Adalight recently, with some customization on my end to use slightly different hardware. I've run into a snag with my intended use: it doesn't work for video games. OpenGL, D3D, etc. programs just send either pure black or pure white. Currently, the image is captured via Java.awt.Robot.createScreenCapture(), in the course of a Processing IDE program. This can be done in small chunks for each LED, or by a single frame capture; the latter is significantly faster on my system. This captured data is processed to find appropriate RGB values for each RGB LED (downscale -> find average color -> gamma correction), then pushed over serial to an Arduino. The Arduino pushes the RGB values to the LED chain. More abstractly, the Java program I'm working with is:
It seems to me that if I want this setup to work for video games, I can change step 1 to work with a program like FRAPS. Seeing as I don't know how to do that directly, I was thinking something like:
This seems a little convoluted to me, but functional. Anyone smarter than me have a better idea or some tweaks that might make this faster/easier/more convenient? Factory Factory fucked around with this message at 17:08 on May 15, 2013 |
|
#
?
May 15, 2013 16:48
|
|
|
 raminasi fucked around with this message at 22:53 on May 15, 2013 |
|
#
?
May 15, 2013 22:48
|
|
|
Are there any particularly excellent online resources where I can learn external sorting techniques? We've got a program that, among other things, needs to take 1TB+ of binary data records and sort them by their embedded times (epoch seconds, usually). Right now our current program (C++ for what that matters) does a lot of work with an Oracle database on the local machine, so we figured we'd use that: make a two-column table of TIME and DATA, insert records one by one, and let Oracle handle the sorting via ORDER BY once the whole raw data file's been read. It works but I honestly have no idea if that's particularly efficient or not. (For what it's worth, the binary in the DATA column is different from what's initially read out of the raw file, so we can't really just store file seek offsets.) Ciaphas fucked around with this message at 03:22 on May 16, 2013 |
|
#
?
May 16, 2013 03:19
|
|
|
The most common way is to break it up into chunks small enough to fit into memory, sort those chunks, and then do a multiway merge to combine them back together.
|
|
#
?
May 16, 2013 04:52
|
|
|
Ciaphas posted:(For what it's worth, the binary in the DATA column is different from what's initially read out of the raw file, so we can't really just store file seek offsets.) I don't grok this. I mean, it makes sense that you're doing some processing on the record from the raw file in order to produce the value for the DATA column, but why can't you wait to do that processing until after you've sorted? You might still need to do an external sort, of course, but depending on how approximately sorted your raw file is and exactly how large the raw records are, it might be a significant win. Also, if you're on a UNIXy system, you might find the <db.h> library convenient for handling large files with fixed record sizes; try man recno.
|
|
#
?
May 16, 2013 06:13
|
|
|
I can certainly ask tomorrow about presorting the raw data before it even gets to our main program, but there's stuff in there that's dependent on FILE order in addition to time order, so reordering it before we start extracting metadata might invalidate some of it. I'll have to check. (This data really sucks and is terrible, yes.) That being said, the records aren't fixed in size, unfortunately (I'll still look at db.h and recno though, thanks!). The raws range anywhere from 100 bytes to 8 kbytes, and the initially-processed stuff that we're presently throwing to Oracle for sorting is about half a kbyte bigger than that.
|
|
#
?
May 16, 2013 06:51
|
|
|
I have a small programming question about c if anyone has a minute to spare. (Note this task is not work but uni related...just so you know why it makes no sense) I have to write a program in c witch gets as input an array of ints. Based on this you build a tree of processes using fork. 2 0 3 this would mean that the root process forks two processes, the first one has 0 children and the second one has three. So far so good. Was able to do this without much problem. Once finished with the tree the root process has to run another process witch will run the program pstree http://www.linfo.org/pstree.html Here is where I get stuck. I could just use sleep in children processes so that they would all wait for one second before the whole tree would end itself. During this time pstree could do its work. Somehow I doubt this is the correct solution as I have no idea how large an array I will receive for input and how much time my program has to finish its execution. So, I need a way to run a process from root witch will run pstree (with the root process id) and once it is finished, start killing all the processes until the entire tree has been terminated. so for input 1 5 0 3 this would be the example output (note the trailing zeros can be omitted) pre:mytask---mytask---mytask
| |-mytask---mytask
| | |-mytask
| | |-mytask
| |-mytask
| |-mytask
| |-mytask
|-pstree
Grogsy fucked around with this message at 08:25 on May 16, 2013 |
|
#
?
May 16, 2013 08:23
|
|
|
Grogsy posted:So, I need a way to run a process from root witch will run pstree (with the root process id) and once it is finished, start killing all the processes until the entire tree has been terminated. This is one option. You could have each process keep an array of the process IDs of its children and then, when they receive a signal from the parent, signal their children to quit before quitting itself. Have you learned about mutexes or semaphores? If so, it's likely you're expected to use those to signal the child processes. That would probably be easier.
|
|
#
?
May 16, 2013 15:03
|
|
|
Ciaphas posted:That being said, the records aren't fixed in size, unfortunately (I'll still look at db.h and recno though, thanks!). The raws range anywhere from 100 bytes to 8 kbytes, and the initially-processed stuff that we're presently throwing to Oracle for sorting is about half a kbyte bigger than that. How many records are there, total? How big are the timestamps? If timestamps are 4 bytes, and indices into the data file are 8 bytes, you should be able to sort a pretty large number of row index, timestamp tuples in memory before using merge sort.
|
|
#
?
May 16, 2013 19:53
|
|
|
Scaevolus posted:How many records are there, total? How big are the timestamps? If timestamps are 4 bytes, and indices into the data file are 8 bytes, you should be able to sort a pretty large number of row index, timestamp tuples in memory before using merge sort. The timestamps aren't always 4-byte epoch times, sadly (  ); they can be either 4, 6, 8, or 10 bytes and it varies within a single file (though the type is identified before the timestamp). The number of records varies from dataset to dataset, but last one I clapped eyes on was just shy of two billion records.I didn't pay anything like enough attention in college to sorting techniques and concerns, more is the pity for me now  . The steps I've got in code (pseudo or otherwise) right now are- Read in raw record - Do file-order-dependent processing - Determine timestamp, add timestamp+processed record to vector - When at some memory usage limit (1GB for now), sort the vector in place and dump it to merge file # 1 - Repeat above until all raw data is sorted and output to files At this point I'm a little lost. I THINK the following steps are as follows... - Read back in records from each file in order, proceeding to the next file when I've read about (memory_limit / number_of_merge_files) bytes worth of records - Sort these results - Do time-order processing and result output - Repeat until all merge files are EOF Does this seem about right? I feel like I'm missing a step somewhere
|
|
#
?
May 16, 2013 22:20
|
|
|
You've got me really curious now as to where these datasets come from.
|
|
#
?
May 16, 2013 23:00
|
|
|
Ciaphas posted:At this point I'm a little lost. I THINK the following steps are as follows... Think about having several books in order: code:code:code:
|
|
#
?
May 16, 2013 23:03
|
|
|
A multiway merge is pretty straightforward to implement - essentially you want to maintain a heap consisting of the lowest element from each merge file. The merge proceeds by peeking at the lowest element in the heap, writing it to the output, then replacing it with the next element from the same input file. Add some buffering to cut down on disk reads and that's essentially the second part of an external sort (and you already seem to understand the first part well enough).
|
|
#
?
May 17, 2013 00:55
|
|
|
 Never even occurred to me that a full merge sort was no longer needed after everything was externalized. I am the Worst Programmer.Thanks both of you for the insight, I think I've got a handle on the process now. Now it's just a matter of actually, y'know, doing it 
|
|
#
?
May 17, 2013 01:11
|
|
|
Why isn't Lua discussed much here? Like, I see literally no mention of it. Is it because its the same as another language? Is it referred to as something different? Is it frowned upon? Sorry for the absolutely trivial question, I'm trying to learn, in general, about programming an my first and strongest language is Lua.
|
|
|
#
?
May 17, 2013 19:28
|
|
|
Lua is a pretty simple language (although you can do very complex things with the simple tools it offers), but it's also one mostly intended for embedding into other things, and it quickly becomes very domain-specific. Two different systems that use Lua for extension/customisation/configuration/something else can be very different to program against, so it's hard to look at Lua as a single language. But no, Lua is its own language and has no other names. There are, however, other languages based on it and around it. (And as I mentioned, practically every usage of Lua could count as a separate language too.) If you want to get better at programming in general, I would suggest trying language different from Lua. I think C++ (or maybe C) would be a decent choice, as it will force you to learn new paradigms. You can also later (maybe not the best to start out with) try your hands with embedding Lua into your C++ program, which will be a very good learning experience.
|
|
#
?
May 17, 2013 20:22
|
|
|
Yeah once you've built a foundry and then a lathe with it, a fun lil' project is to put together a mill!
|
|
#
?
May 17, 2013 21:29
|
|
|
I'm going to go ahead and ask: Is there anything that one can do in OpenCobol that one couldn't just do in C and with a lot more portability to boot? Other than having fun of course.
|
|
#
?
May 17, 2013 22:30
|
|
|
Without having looked OpenCobol up in any way, ever, presumably it will let you build and run those existing millions of lines of code in your old ERP system that has been continually extended since the late 1970's, and maybe integrate it with other languages.
|
|
#
?
May 18, 2013 00:02
|
|
|
nielsm posted:Without having looked OpenCobol up in any way, ever, presumably it will let you build and run those existing millions of lines of code in your old ERP system that has been continually extended since the late 1970's, and maybe integrate it with other languages. Great now I have a desire to work on an llvm front end to Cobol, thanks guys!
|
|
#
?
May 18, 2013 14:12
|
|
|
nielsm posted:Without having looked OpenCobol up in any way, ever, presumably it will let you build and run those existing millions of lines of code in your old ERP system that has been continually extended since the late 1970's, and maybe integrate it with other languages. OpenCobol literally translates Cobol to C and then compiles the C code, so I don't think that would work.
|
|
#
?
May 18, 2013 16:56
|
|
|
Those in front end and back end web development, what's your take on stuff like this. http://blog.mongodb.org/post/492628...ent=buffera01df I guess I'm a little hesitant to start drinking the one language top to bottom kool-aid, I think every good language and its' library set has it's place, but I don't think Javascript has it's place literally everywhere. Personally I feel like I'm heading towards using Scala / Python for my server stuff, depending on the project at hand. Am I just being a stick in the mud here, what are the downsides to an all Javascript stack that these glowing articles don't really mention?
|
|
#
?
May 19, 2013 04:15
|
|
|
Maluco Marinero posted:Am I just being a stick in the mud here, what are the downsides to an all Javascript stack that these glowing articles don't really mention? JavaScript and MongoDB is the downside.
|
|
#
?
May 19, 2013 04:24
|
|
|
MongoDB is pretty awful, but there's no reason you need to use it for an all-Javascript stack. At the moment I think the all-Javascript stack mostly only makes sense if your backend is basically just a data access layer with authentication and stuff (which is basically all that's left for a purely-CRUD app with a Javascript MVC framework on the frontend). There's an absolute explosion of libraries for node.js at the moment, but there's still a lot fewer rock-solid ones than for other languages, and of course the language itself isn't great.
|
|
#
?
May 19, 2013 05:03
|
|
|
one language everywhere might be ok if it was a good language.
|
|
#
?
May 19, 2013 06:56
|
|
|
Javascript is.. quirky, but actually is a pretty good language. It's no surprise it's getting more use on serverside and elsewhere.
|
|
#
?
May 19, 2013 08:44
|
|
|
pigdog posted:Javascript is.. quirky, but actually is a pretty good language. It's no surprise it's getting more use on serverside and elsewhere. JavaScript is a deeply flawed language. Deeply flawed languages are bad even when they are good IMO. Google I/O turned me on to Dart, though. It seems like a proper language, not to mention a bit more traditional, conceptually, which makes it feel instantly more familiar, whereas "web development" is an entirely different category of programming because of how dramatically different javascript forces it to be. I feel that this is not the natural order of things, even if the destiny of web apps is to go client-side - web app dev should not be quite so significantly removed from application dev.
|
|
#
?
May 19, 2013 09:26
|
|
|
Gul Banana posted:one language everywhere might be ok if it was a good language. Quick! Name one good language.
|
|
#
?
May 19, 2013 15:37
|
|
|
qntm posted:Quick! Name one good language. haskell
|
|
#
?
May 19, 2013 16:01
|
|
|
qntm posted:Quick! Name one good language. Python
|
|
#
?
May 19, 2013 16:22
|
|
|
qntm posted:Quick! Name one good language. I get your point, but I think there's a lot of languages where you wouldn't have as large a portion of developers saying the same sort of things they say about JS. I mean, if you pick a widely used language like say Python, there's going to be a lot of things that a lot of developers don't like about it, but, at the same time, it seems to me that there's not going to be a lot of developers saying it's a bad language. I dunno, it just seems as if there's an actual quality difference between JS and other widely used languages that goes beyond the typical "I don't like semantic whitespace" or "too verbose" complaints you hear about other languages.
|
|
#
?
May 19, 2013 17:03
|
|
|
The developers who have serious beef with Python are either monsters or are trying to use Python for a job that's better performed with a compiled language like C++. No language is perfect, but Python is definitely a 'good language' The most common complaint I've seen, and I've gone looking for them, is that Python uses whitespace. And then they obstinately try very hard to create examples showing how hard it is to understand whitespace in Python. If the worst problem can be overcome by anyone with a 90 IQ, then Python must be pretty great. QuarkJets fucked around with this message at 18:42 on May 19, 2013 |
|
#
?
May 19, 2013 18:38
|
|
|
Well, it depends on what you mean by "complaint". Any Python developer will tell you that there's warts, like from foo import *, and the ever-infamous mutable tuple:code:
|
|
#
?
May 19, 2013 18:51
|
|
|
With apologies to Churchill, Python is the worst language except for all the others.
|
|
#
?
May 19, 2013 19:11
|
|
|
qntm posted:Quick! Name one good language. Well, this is sort of where I'm coming from I guess. Deciding that you'll use one language for your entire stack is essentially saying that language is the correct tool for every job, and I very much doubt Javascript and it's ecosystem is that language. That sort of zealotry is bad wherever it comes from, just like my goto example for this, a library for compiling Haskell to Javascript.
|
|
#
?
May 19, 2013 22:54
|
|
|
Maluco Marinero posted:Well, this is sort of where I'm coming from I guess. Deciding that you'll use one language for your entire stack is essentially saying that language is the correct tool for every job
|
|
#
?
May 20, 2013 00:43
|
|
|
|
| # ? May 29, 2024 16:10 |
|
|
Plorkyeran posted:Not really. All that it's saying is that the advantages of a single-language codebase outweigh the advantages of using a better language for one piece of it. Even if you assume that Python is strictly better than Javascript, how much better does it need to be to make up for that you'll inevitably end up with a bunch of duplicated code written in both Python and Javascript? That's true, however I think the more pertinent question with JavaScript is how bad does a language have to be before you won't consider doing everything in that language. JavaScript certainly heads into that territory with the way it requires significant work to be usable in projects of high complexity, and pretty much any element of safety comes not from the language but from what the project developers directly introduce. JavaScript is an ultra flexible language but with that comes plenty of rope for developers to hang themselves. I guess my feeling is that any server purporting to be the source of truth and stability in data should probably be held to a higher standard than JavaScript typically provides.
|
|
#
?
May 20, 2013 01:39
|
|