Register a SA Forums Account here!
JOINING THE SA FORUMS WILL REMOVE THIS BIG AD, THE ANNOYING UNDERLINED ADS, AND STUPID INTERSTITIAL ADS!!!

You can: log in, read the tech support FAQ, or request your lost password. This dumb message (and those ads) will appear on every screen until you register! Get rid of this crap by registering your own SA Forums Account and joining roughly 150,000 Goons, for the one-time price of $9.95! We charge money because it costs us money per month for bills, and since we don't believe in showing ads to our users, we try to make the money back through forum registrations.
 
  • Post
  • Reply
carry on then
Jul 10, 2010

by VideoGames

(and can't post for 10 years!)

I was illustrating a point.

Adbot
ADBOT LOVES YOU

Hughlander
May 11, 2005

carry on then posted:

I was illustrating a point.

Which you did. Just not the one you meant to.

carry on then
Jul 10, 2010

by VideoGames

(and can't post for 10 years!)

Sorry then. Checked exceptions are good and any language without them is utter garbage, I was wrong.

Janitor Prime
Jan 22, 2004

PC LOAD LETTER

What da fuck does that mean

Fun Shoe

carry on then posted:

Sorry then. Checked exceptions are good and any language without them is utter garbage, I was wrong.

God bless Java

Dominoes
Sep 20, 2007

New to Java. Trying to get the Interactive Brokers API working. How do I install packages?

I'm using code like this to import:
import com.ib.client.ComboLeg;


I ran an installer from Interactive Brokers, which created a directory on my hard drive that includes a directory tree. There are also files called . classpath, .project and build.xml. Based on searching, it looks like these are Ant files, but I've only found instructions that involve installing IDEs. Is there a console command or something?

In Python, you'd run 'pip install packagename'. , or 'python setup.py install' .

It looks like dropping the package folder in the same directory as my code works, but I'd like to install it as an official module.

Dominoes fucked around with this message at 19:22 on Jan 22, 2015

Jock Seppuku
Apr 16, 2008

Dominoes posted:

New to Java. Trying to get the Interactive Brokers API working. How do I install/import packages?

I'm using code like this to import:
Java code:
import com.ib.client.ComboLeg;
import com.ib.client.Contract;
I'm getting the error ib.java:10: error: package com.ib.client does not exist, probably because the package isn't installed.

I ran an installer from Interactive Brokers, which created a directory on my hard drive that includes a directory tree. There are also files called . classpath, .project and build.xml. Based on searching, it looks like these are Ant files, but I've only found instructions that involve installing third-party IDEs. Is there a console command or something?

Ie in Python, you'd run 'pip install packagename'. , or 'python setup.py install' .

It looks like dropping the package folder in the same directory as my code works, but I'd like to install it as an official module.

If you're not using an IDE, you can specify the classpath at execution time. Basically, compile all the Interactive Brokers classes into one directory, then when you run your program, point the -cp flag to that directory. For more information, checkout this documentation from Oracle.

Really though, if you're doing Java development for any extended period of time, you should use an IDE and a build system (Eclipse and Maven are free, and reasonable choices).

Sedro
Dec 31, 2008
Just run 'ant' in that directory and it should pop you out a jar

HFX
Nov 29, 2004

Brain Candy posted:

I love this example, because even if you check before you do anything there's a race if something else to deletes the file or changes permissions between your check and the read.

This can happen more often then people expect especially if there is a network mount. I've also seen it occur in a log monitor which went to open the log right as the log was rolling.

As to check exception vs unchecked exceptions, put me in the camp that it would be nice to not always have to deal with the boilerplate. However, the problem I have personally dealt with both in C++ and C# is that it can be hard to figure out all possible exceptions that might be thrown. Of course, a lot of bad lazy Java devs will simply make a try catch(Exception) block and then throw a RuntimeException out of that if I'm lucky. The worst part is, they often don't save the exception, or print out a message that will be System.err.println("Exception occured" + e).

Maybe the solution is to have an IDE tell me all possible exceptions and create warnings if I don't deal with them? Actually, that is bad too since most programmers tend to treat a warning as it is all good.

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...

Dominoes posted:

New to Java. Trying to get the Interactive Brokers API working. How do I install packages?

I'm using code like this to import:
import com.ib.client.ComboLeg;


I ran an installer from Interactive Brokers, which created a directory on my hard drive that includes a directory tree. There are also files called . classpath, .project and build.xml. Based on searching, it looks like these are Ant files, but I've only found instructions that involve installing IDEs. Is there a console command or something?

In Python, you'd run 'pip install packagename'. , or 'python setup.py install' .

It looks like dropping the package folder in the same directory as my code works, but I'd like to install it as an official module.

.classpath and .project are eclipse metadata project files. build.xml is almost certainly for Apache Ant.

Use ant to build that raw source into a JAR, then use the jar as your module.

FateFree
Nov 14, 2003

I'm looking for the absolute fastest way to process a large volume of data and turn it into a file based hash table. I have about 25,000 files, each around 256 megs worth of data thats divided into 52 byte chunks. What I need to do is take each chunk, find its hash value, and write it to a file with the name of the hash value. The hash table has 2048 files total (named /0.hash, /1.hash ... /2047.hash)

As you can imagine there is a lot of IO, and performance here is in terms of days saved so I'm really looking for the fastest approach.

My first approach was to create a map of hash value > bytearray, load a file one a time, break it apart into chunks and place each chunk into the appropriate bytearray from the map. When the file is finished, I iterate through the map and flush each bytearray into its own file. This works but I think there is a big slowdown writing to 2048 files after each file is read. Maybe an improvement is to batch several files before flushing them? Or using another data structure instead of the bytearray (technically a ByteArrayOutputStream).

Also this is using standard java IO, I'm not very familiar with NIO so I don't know if it could apply here. I'd appreciate any ideas for speed that I can look into and start profiling. Thanks

MrMoo
Sep 14, 2000

What's the background of this app? Consider on Linux mounting the source files in XFS/ext4 read-only then using an overlayfs with a tmpfs based read/write volume.

FateFree
Nov 14, 2003

MrMoo posted:

What's the background of this app? Consider on Linux mounting the source files in XFS/ext4 read-only then using an overlayfs with a tmpfs based read/write volume.

Its runnin on a 6TB NTFS harddrive so I don't think thats an option. Another thing is the hd is filled with about 4.5 TB of data, so the unprocessed files will have to be deleted in order to make space for the hash files.

MrMoo
Sep 14, 2000

Even reading at 100MB/s that is 18 hours, you are going to need some form of accelerated storage, either SSD or a multi-spindle RAID array.

Where is the data sourced from, can you acquire it in parallel to feed a farm of servers?

MrMoo fucked around with this message at 03:46 on Jan 27, 2015

FateFree
Nov 14, 2003

Honestly 18 hours is great, my previous projection was 11 days. The data was in fact sourced in parallel (over several days), but it was dumped onto this drive so thats where I'm at now. Its just a matter of hashing it as fast as possible.

MrMoo
Sep 14, 2000

With a single disk you could get ~100MB/s read, so two seconds to slurp an entire file into memory, then process, then dump out the files. However a single disk will have an IOPS of ~100 so that can be 40+ seconds (1 IO for file, 1 IO for metadata) to write out which is your 11 days for the entire data set.

You need a compromise somewhere, either better hardware or a break in the output requirements.

MrMoo fucked around with this message at 04:12 on Jan 27, 2015

FateFree
Nov 14, 2003

Thats a good point.. 2048 was the size chosen for the hash so the hash files can later be sorted in memory (at ~2 gig per file). A potential compromise is to cut the number of files down to 1024 or 512 even, and then I'd need to find a sorting solution that can sort a file externally in pieces rather than in memory all at once..

My Rhythmic Crotch
Jan 13, 2011

Curious if anyone here has experience with Jetty hot deployment. It is a feature which supposedly reloads your .war packaged web application when you replace the existing .war file. I'm encountering a bug where Jetty will have difficulty finding a class in my application (even though the class is obviously there) and will throw a "class not found" exception until Jetty has been fully stopped and restarted. So currently I just do a full restart each time I deploy a new version of the application. This would normally not be a big deal but even worse, sometimes Jetty takes a while (1, 2 minutes tops) to restart since I have it configured to acquire a static chunk of memory on startup.

Anyway, if there is any advice people have for getting bulletproof performance out of Jetty, I would love to hear it.

ExcessBLarg!
Sep 1, 2001

FateFree posted:

I have about 25,000 files, each around 256 megs worth of data thats divided into 52 byte chunks.
Is that 512 byte chunks? 52 bytes doesn't really make sense. What's your hash size?

FateFree posted:

As you can imagine there is a lot of IO, and performance here is in terms of days saved so I'm really looking for the fastest approach.
The biggest inefficiency with spinny disks is the seek latency. Fortunately it can be amortized by writing large chunks, but that means every I/O needs to deal with units 1 MB or larger.

If you're using a 128 bit hash, you're only writing 4 kB to each of the 2048 output files after processing one 256 MB input file, so yes, you're being destroyed in seeks. Furthermore, since you're only writing 4 kB to each file at a time, the file system itself is probably not a great job on block allocation, such that the contents of each one of those files is widely spread, making read-back be a slow operation later.

Honestly I'd probably write the hashes to one giant file and figure out the right way of sorting it later. Merge sort using temporary external files is a common approach, and I think is what GNU sort uses (although it operates on line data).

elite_garbage_man
Apr 3, 2010
I THINK THAT "PRIMA DONNA" IS "PRE-MADONNA". I MAY BE ILLITERATE.
I'm looking for a little help using Xstream

Currently I have my output looking like this:

code:
<venues>
  <venue>
    <venueCode>org59</venueCode>
    <venueName>Georgia Dome</venueName>
    <street>1 Georgia Dome Drive Northwest</street>
    <city>Atlanta</city>
    <state>GA</state>
    <zip>30313</zip>
    <country>USA</country>
    <capacity>71228</capacity>
  </venue>
</venues>
How can I get the street, city, state, zip and address blocks so they fall under an <address> field? so it looks like this:

code:
<venues>
<venue>
  <venueCode>org59</venueCode>
  <venueName>Georgia Dome</venueName>
  <address>
    <street>1 Georgia Dome Drive Northwest</street>
    <city>Atlanta</city>
    <state>GA</state>
    <zip>30313</zip>
    <country>USA</country>
  </address>
  <capacity>71228</capacity>
</venue>
</venues>

leftist heap
Feb 28, 2013

Fun Shoe
XStream just does a Bean to XML mapping right? So presumably right now you have a Venue class with street, city, state, etc. fields right? Instead you need an Address class with those fields and make that a field of Venue instead. Am I way off?

elite_garbage_man
Apr 3, 2010
I THINK THAT "PRIMA DONNA" IS "PRE-MADONNA". I MAY BE ILLITERATE.
I'm sure you're right, but I'm having trouble getting it implemented. I made an Address class, added the alias, and had the Venue class extend Address. Now I just get all the Address information first, and it still lacks the field blocks of <address>

Volguus
Mar 3, 2009

elite_garbage_man posted:

had the Venue class extend Address

Pretty sure this is the problem (never worked with xstream, but i did work with jaxb and friends). drat curriculum and their "inheritance" focus. Why would Venue extend Address? Polymorphism is ... sometimes needed. Mostly it is not. Do not extend objects for the sake of it. Professors should be getting rid of this "polymorphism everywhere" bug. This is not 1995 anymore.

Think about it like this:

"A is a special case of B. Having either A or B most of the time won't make a difference." If that's true, then sure, A extends B. If that's false, then hell no, do not ever ever extend anything.

And in your case:
Venue has Address. What does that tell you in term of their relationship?

Malloc Voidstar
May 7, 2007

Fuck the cowboys. Unf. Fuck em hard.
Regarding Apache HttpClient 4.3:
Does anyone know how to change the user-agent after I've constructed an HttpClient? During building I can change it using HttpClientBuilder's setUserAgent(String) but I can't figure out how to change it afterwards.
It used to be fairly easy to change (via HttpProtocolParams.setUserAgent) but they've now deprecated that class. It says to use "use configuration classes provided 'org.apache.http.config' and 'org.apache.http.client.config'" in the docs but I can't find anything in those packages that allows changing the user-agent.

I can change it manually on every request by doing setHeader but I'd rather use a safer method where I won't accidentally forget to setup a request like that.

Gravity Pike
Feb 8, 2009

I find this discussion incredibly bland and disinteresting.
I'm not finding anything that'd allow you to change the default UserAgent within an HttpClient. I expect that this has to do with thread-safety concerns, and trying to make the HttpClient immutable/stateless. You can mess with configuration when it's still a Builder, because there's no risk of some other thread changing state in the middle of a request. You can specify the header during a request, because there's no threat of that impacting another concurrent HttpRequest. If you are able to change the Client's default UserAgent, however, you can get into situations where you've started creating a request, get half of a config change, submit the request, and then get the other half of the config-change. :(

Java code:
public static void main(String[] args) {
  final ExecutorService exec = Executors.newFixedThreadPool(2);
  final HttpClient httpClient = HttpClients.createDefault();
  final Callable<Response> task1 = new Callable<>() {
    public Response call() {
      httpClient.setUserAgent("Chrome, I guess?");
      return httpClient.execute(new HttpGet("http://task1.com"));
    }
  };
  final Callable<Response> task2 = new Callable<>() {
    public Response call() {
      httpClient.setUserAgent("FireFlam!");
      return httpClient.execute(new HttpGet("http://task2.com"));
    }
  };

  exec.submit(task1);
  exec.submit(task2);

  exec.awaitTimeout(30, TimeUnit.SECONDS);
}
The developer would expect task1.com to be gotten with the user agent: chrome, and task2.com to be gotten with FireFlam, but (were .setUserAgent a method on httpClient), the user agent for each request would be non-deterministic.

covener
Jan 10, 2004

You know, for kids!

Gravity Pike posted:

I'm not finding anything that'd allow you to change the default UserAgent within an HttpClient. I expect that this has to do with thread-safety concerns, and trying to make the HttpClient immutable/stateless. You can mess with configuration when it's still a Builder, because there's no risk of some other thread changing state in the middle of a request. You can specify the header during a request, because there's no threat of that impacting another concurrent HttpRequest. If you are able to change the Client's default UserAgent, however,

An option: HTTPRequestInterceptor hooked into the client that tweaked each request as it went out?

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE
Maven seems to waste a stunning amount of time doing nothing in particular. Kicking off each step in the chain incurs massive wait time as the plugin launches - for example it takes like 1:30 when it it launches maven-surefire-plugin with skipped tests. All in all I'm looking at a 9-minute build for a 7MB WAR file, almost all of which is precompiled libraries, on an i7-4770k. This happens under both Netbeans and Eclipse. Is there anything I can do to fix my build times here?

Sedro
Dec 31, 2008
Maven is slow but not that slow. How long does it take for these?
pre:
mvn test -DskipTests
mvn test -Dmaven.test.skip=true

fletcher
Jun 27, 2003

ken park is my favorite movie

Cybernetic Crumb

Paul MaudDib posted:

Maven seems to waste a stunning amount of time doing nothing in particular. Kicking off each step in the chain incurs massive wait time as the plugin launches - for example it takes like 1:30 when it it launches maven-surefire-plugin with skipped tests. All in all I'm looking at a 9-minute build for a 7MB WAR file, almost all of which is precompiled libraries, on an i7-4770k. This happens under both Netbeans and Eclipse. Is there anything I can do to fix my build times here?

Are you running anti-virus? I've never had a problem with it but some coworkers were complaining of slow build times and we fixed it by whitelisting the directories that have source code and the maven repo. Build times substantially improved after that.

Paul MaudDib
May 3, 2006

TEAM NVIDIA:
FORUM POLICE

fletcher posted:

Are you running anti-virus? I've never had a problem with it but some coworkers were complaining of slow build times and we fixed it by whitelisting the directories that have source code and the maven repo. Build times substantially improved after that.

Yes, it's antivirus. I spun up a Lubuntu VM on the same machine, it did a "clean package" in 7 seconds flat. My Tomcat deploy times went from 61 seconds to 6 seconds. Come to think of it, Selenium performed a lot better on a server VM that didn't include the standard AV. At the time I blamed that one on not being able to get it to load a clean browser profile properly, with Tomcat I just didn't have a frame of reference. But in hindsight I think all three of those apps are probably tripping the read/write scanning agent. I never put the pieces together, but Sophos just doesn't play nice with Java.

We run Sophos+Bit9 and some other anti-malware. This combo gave me hell installing Netbeans, it would just silently kill the installer. Today I tried Gradle, and I realized it was doing the same silent kill thing to the gradle daemon, and the pieces fell into place for me. :negative:

Paul MaudDib fucked around with this message at 01:10 on Feb 6, 2015

Woodsy Owl
Oct 27, 2004
A clarifying question; static methods in a class increase the memory footprint of instances of that class, right? Isn't this is one of the arguments for independent factory classes?

edit: A second question on memory footprints; if methods, static and non-static, increase the memory footprint of an object, what would be the argument (aside from looser scope) against creating a 'reader' class to get and set the fields of an object?

Woodsy Owl fucked around with this message at 11:56 on Feb 8, 2015

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...

Woodsy Owl posted:

A clarifying question; static methods in a class increase the memory footprint of instances of that class, right? Isn't this is one of the arguments for independent factory classes?

edit: A second question on memory footprints; if methods, static and non-static, increase the memory footprint of an object, what would be the argument (aside from looser scope) against creating a 'reader' class to get and set the fields of an object?

http://stackoverflow.com/a/1826807

I don't know why you would assume that methods, static or otherwise, would be a part of the instance. The instance contains a reference to the Class, and any member variables. It would make sense for instances to have references to functions in languages like JavaScript, although iirc in JavaScript you just overload methods with the default falling through to your prototype so even there it's not abysmal.

Frank Zappa
Feb 6, 2004

Electric Aunt Jemima - Goddess of Love
I've got a Java question which I'm pretty sure is pretty easy, but I'm quite new to this whole Java programming thing. (I'm taking a class in it right now.)

I've got a homework question that requires me to process a string through a loop and reverses the order of the string. So, I'm trying to use the charAt() method in the string class to concatenate that letter to the end of a string. So...

code:
System.out.print(inputText.charAt(textLength));     //This line works just fine.
outputText = outputText.concat(inputText.charAt(textLength));    //...and this line doesn't.
I think the reason it doesn't work is that inputText.charAt(textLength) has a character output, and concat expects a string input. Am I right? And if so, how do I fix it?

Thanks for your help!

Frank Zappa fucked around with this message at 18:58 on Feb 8, 2015

carry on then
Jul 10, 2010

by VideoGames

(and can't post for 10 years!)

Frank Zappa posted:

I've got a Java question which I'm pretty sure is pretty easy, but I'm quite new to this whole Java programming thing. (I'm taking a class in it right now.)

I've got a homework question that requires me to process a string through a loop and reverses the order of the string. So, I'm trying to use the charAt() method in the string class to concatenate that letter to the end of a string. So...

code:
System.out.print(inputText.charAt(textLength));     //This line works just fine.
outputText = outputText.concat(inputText.charAt(textLength));    //...and this line doesn't.
I think the reason it doesn't work is that inputText.charAt(textLength) has a character output, and concat expects a character input. Am I right? And if so, how do I fix it?

Thanks for your help!

Let's look at the JavaDoc: http://docs.oracle.com/javase/7/docs/api/java/lang/String.html#concat(java.lang.String)

quote:

public String concat(String str)

It takes a String, not a character, so that's why yours doesn't work. There are a several ways to take a character and put it on the end of the string, but the common one is to use the + operator, which is overridden to concatenate a value and a String if one of the arguments is a String already.

code:
outputText += inputText.charAt(textLength);
If you're curious, what's happening is that the compiler converts (e: actually looking around, it doesn't always do this, but the behavior is the same either way) this to use something called a StringBuilder: http://docs.oracle.com/javase/7/docs/api/java/lang/StringBuilder.html

Which has an append() method, which is overloaded to have variants that take all manner of primitive types as well as any Object. So whenever the conditions are met to make + a concatenation operation, the line I gave you would be replaced at compile time with something like

code:
StringBuilder sb = new StringBuilder();
sb.append(outputText); // calls append(String)
sb.append(inputText.charAt(textLength)); //calls append(char)
outputText = sb.toString();
(It's probably written much more compactly, but I wanted to write it out to demonstrate how that particular class works.)

carry on then fucked around with this message at 19:09 on Feb 8, 2015

Frank Zappa
Feb 6, 2004

Electric Aunt Jemima - Goddess of Love

carry on then posted:

very helpful stuff

Cool... thanks a bunch!!

Volmarias
Dec 31, 2002

EMAIL... THE INTERNET... SEARCH ENGINES...
And to follow up, if you're allowed to for homework, you should just use StringBuilder. A typical novice mistake for java is to do what you're doing, which is to build a string by appending to an accumulator string over and over, since that involves creating a new builder and a new immutable string in every single round of your loop.

Woodsy Owl
Oct 27, 2004

Volmarias posted:

http://stackoverflow.com/a/1826807

I don't know why you would assume that methods, static or otherwise, would be a part of the instance. The instance contains a reference to the Class, and any member variables. It would make sense for instances to have references to functions in languages like JavaScript, although iirc in JavaScript you just overload methods with the default falling through to your prototype so even there it's not abysmal.

Thanks for the link, that cleared it up.

Woodsy Owl fucked around with this message at 15:51 on Feb 10, 2015

DholmbladRU
May 4, 2006
Trying to wrap my head around the concept of a Map. I understand they can be compared in some respects to an associative array in PHP.

What I am wondering is, is it possible to create a Map constructor which assigns the index names. But have the ability to assign the values later by index.

Volguus
Mar 3, 2009

DholmbladRU posted:

Trying to wrap my head around the concept of a Map. I understand they can be compared in some respects to an associative array in PHP.

What I am wondering is, is it possible to create a Map constructor which assigns the index names. But have the ability to assign the values later by index.

There is no concept of index in a map. You have key-value. The keys and values can be anything (objects), with some requirements on the keys behaviour depending on the map implementation that you're using (a common one is HashMap).

DholmbladRU
May 4, 2006

Volguus posted:

There is no concept of index in a map. You have key-value. The keys and values can be anything (objects), with some requirements on the keys behaviour depending on the map implementation that you're using (a common one is HashMap).

So when creation of the map is performed the values have to be assigned? I guess instead of a map I could use a object to accomplish something similar.

Adbot
ADBOT LOVES YOU

Master_Odin
Apr 15, 2010

My spear never misses its mark...

ladies

DholmbladRU posted:

Trying to wrap my head around the concept of a Map. I understand they can be compared in some respects to an associative array in PHP.

What I am wondering is, is it possible to create a Map constructor which assigns the index names. But have the ability to assign the values later by index.

To add onto Volguus, the best you could do is something like:
code:
import java.util.HashMap;
import java.util.Map;

public class HelloWorld{

     public static void main(String []args){
        Map<String, String> hashMap = new HashMap<String, String>();
        for (String key : new String[] {"test", "test2"}) {
            hashMap.put(key,null);
        }
        
        for(String key: hashMap.keySet())
            System.out.println(key + " - " + hashMap.get(key));
        System.out.println();
        
        hashMap.put("test","has a value now");
        for(String key: hashMap.keySet())
            System.out.println(key + " - " + hashMap.get(key));
        System.out.println();
     }
}
Where you could put a key with a null value (assuming HashMap and not something like HashTable), but it's really not quite the same as associative arrays in PHP.

  • 1
  • 2
  • 3
  • 4
  • 5
  • Post
  • Reply