The Perl Short Questions Megathread: executable line noise

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > The Perl Short Questions Megathread: executable line noise

«‹›72 »

Erasmus Darwin: Mar 6, 2001

Captain Frigate posted:

one of the things i need to unpack is an unsigned long long int, and i'm just not sure how to go about that because it doesn't look like unpack does that.

It looks like some versions of Perl support 'Q' for packing/unpacking an unsigned 64-bit integer. You can test it by doing:
perl -e 'pack("Q",0);'

If you get an error about Q being an invalid type for pack, it won't work. Otherwise, you're good to go.

If it doesn't work, you're on the right track with reading the value in two chunks. However, if by padding it with zeros, you mean multiplying it by a power of 10, that won't exactly work since the boundary between the two parts is a power of 2. So you want to do a bitshift (which will pad your value with zero bits).

HOWEVER, it's more complicated than that. If your version of Perl wasn't built with 64-bit integers (which if it was, you should be able to just use 'Q' as an unpack argument), you're going to run into overflow issues. So you'll probably want to use Math::BigInt instead.

# ? Jul 6, 2009 15:08

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 14:08

Triple Tech: Jul 28, 2006; So what, are you quitting to join Homo Explosion?

I never really know how to read prof output... But that sorting subroutine looks a little chunky, and it's hard to understand what it's doing. If it turns out it has a high computation cost, you can Schwartzian transform it and reduce the number of calculations from N^2 to N.

# ? Jul 6, 2009 15:12

wigga please: Nov 1, 2006; by mons all madden

I was thinking along the same lines, implement a quicksort there and be done with it, but even if it's 10 times faster, the script as a whole is still 98% too slow. Ideally I need 20000 simulations against up to 8 opponents with random card and a random turn + river card in under a second. Doesn't look like there's much tweaking possible to get Games::Cards::Poker to run that fast, so I'm hoping I'll be able to call functions from the PSim DLL (in C++) I included.

# ? Jul 6, 2009 15:19

Erasmus Darwin: Mar 6, 2001

Erasmus Darwin posted:

So you'll probably want to use Math::BigInt instead.

Ok, I think I've got a decent Math::BigInt-based solution. Since you didn't specify, I'm just going to stick with reading a number in little-endian order. Assuming that $bin contains the binary representation of the 64-bit int, this should get it turned into a nice Math::BigInt object:

code:

$num = Math::BigInt->new('0x' . reverse(unpack('h*', $bin)));

Edit: If you want to read in a number in network/big-endian order, this should work:

code:

$num = Math::BigInt->new('0x' . unpack('H*', $bin));

However, byte order issues give me a headache, so I could have gotten something jumbled up. Caveat emptor, and make sure you test this code when you toss it into your script.

Erasmus Darwin fucked around with this message at 15:33 on Jul 6, 2009

# ? Jul 6, 2009 15:29

Captain Frigate: Apr 30, 2007; you cant have it, you dont have nuff teef to chew it

Erasmus Darwin posted:

Ok, I think I've got a decent Math::BigInt-based solution. Since you didn't specify, I'm just going to stick with reading a number in little-endian order. Assuming that $bin contains the binary representation of the 64-bit int, this should get it turned into a nice Math::BigInt object:
code:
$num = Math::BigInt->new('0x' . reverse(unpack('h*', $bin)));
Edit: If you want to read in a number in network/big-endian order, this should work:
code:
$num = Math::BigInt->new('0x' . unpack('H*', $bin));
However, byte order issues give me a headache, so I could have gotten something jumbled up. Caveat emptor, and make sure you test this code when you toss it into your script.

Thanks a lot man. I appreciate it. I had an interesting issue though. I can do something like:

code:

$num = 2**45;
print $num;

with no issues but if I try:

code:

print unpack("Q", pack("Q", 2**38)),"\n";

perl crashes. Is that normal?

EDIT: Rather, does this still mean that perl isn't configured for 64 bit ints, or is that ok?

Captain Frigate fucked around with this message at 19:25 on Jul 7, 2009

# ? Jul 7, 2009 19:18

S133460: Mar 17, 2008; hello

To find out if your perl is a 64bit binary:

code:

$ file `which perl`

A 32bit perl can still emulate support of 64bit integers. This flag enables 64-bit scalar values (even if your perl is 32bits):

code:

$ perl -V:use64bitint

If the use64bitall flag is set, your perl was built with 64bit types (including 64-bit pointers):

code:

$ perl -V:use64bitall

# ? Jul 7, 2009 21:01

Erasmus Darwin: Mar 6, 2001

Captain Frigate posted:

perl crashes. Is that normal?

I can only get two results out of the various perl installations I've got lying around: The correct answer (on the one 64-bit install), and the invalid pack option error (on all the rest). However, the only one with any of the 64-bit -V options enabled is the full 64-bit one (64-bit Centos 5 box).

# ? Jul 7, 2009 23:04

Captain Frigate: Apr 30, 2007; you cant have it, you dont have nuff teef to chew it

Erasmus Darwin posted:

I can only get two results out of the various perl installations I've got lying around: The correct answer (on the one 64-bit install), and the invalid pack option error (on all the rest). However, the only one with any of the 64-bit -V options enabled is the full 64-bit one (64-bit Centos 5 box).

Hm, so you don't get the right answer on non-64-bit installs? Well either way, its looks like your solution works like a charm! Thanks a lot!

# ? Jul 8, 2009 14:18

S133460: Mar 17, 2008; hello

I'm pretty sure you only get the Q/q specifiers for pack on 64bit system with a perl built with 64bit support. You can still get 64-bit-wide integer scalar support without that exact setup though.

# ? Jul 8, 2009 16:05

Erasmus Darwin: Mar 6, 2001

Captain Frigate posted:

Hm, so you don't get the right answer on non-64-bit installs?

On the 32-bit installs that I tried it on, trying to invoke pack or unpack with the Q option produced a fatal error that ended the script. So I didn't get the right answer, but I didn't get a wrong answer, either -- the script just dies.

If you're going to run your script on a variety of machines, you can always use an eval block to test whether or not it works. Something like the following quick-and-ugly, untested code:

code:

eval {
  # Note: Use of string comparisons is deliberate.
  $a = 2**63 + 1;
  die '2**63 + 1 is wrong or inaccurate' if $a ne '9223372036854775809';
  $b = unpack('Q', pack('Q', $a));
  die 'mismatch from pack/unpack' if $a ne $b;
}
$64bit_ops_ok = ! $@;

It'd probably be more ideal to test it with 2**64 - 1, but I was worried about the 2**64 part of the calculation producing an overflow / float promotion even on a 64-bit platform, and I'm too lazy to see if that's the case.

And ideally, you could just check the config options satest3 mentioned that perl was compiled with, but since you're getting incorrect results from your own setup, an actual test might be more worthwhile.

Also, depending on how you're using the 64-bit value in the rest of your code, you might want to look into more efficient options. There's Math::BigInt::Lite which uses built-in scalars for the numbers until your reach the overflow point where it transparently flips to Math::BigInt. There are also more efficient libraries for doing Math::BigInt's calculations such as Math::BigInt::GMP and Math::BigInt::Pari.

quote:

Well either way, its looks like your solution works like a charm! Thanks a lot!

Glad I could help. It was a fun problem, and I was actually a little surprised when it turned out to be a one-line solution (excluding the 'use' line for the library, of course). I was expecting to have to feed two 32-bit integers into a Math::BigInt object with a bitshift in-between. I was just using the unpack('h*', $bin) part to double-check my assumptions about how the data actually looked when I noticed how easy it would be to manipulating that into a string that could initialize a Math::BigInt.

# ? Jul 8, 2009 18:50

S133460: Mar 17, 2008; hello

quote:

It'd probably be more ideal to test it with 2**64 - 1, but I was worried about the 2**64 part of the calculation producing an overflow / float promotion even on a 64-bit platform, and I'm too lazy to see if that's the case.

You could also try evaluating 1 << 32, since the shift operators are implemented via the C shift operators and always perform integer arithmetic (as do the other numeric bitwise operators).

# ? Jul 9, 2009 02:55

Captain Frigate: Apr 30, 2007; you cant have it, you dont have nuff teef to chew it

Ok, the 64-bit solution worked like a charm, but now I'm even more confused than before. Some background: right now I am trying to parse a binary file, and I have this:

code:

        $pos_1=tell(VID);
	$number_read_1=read (VID,$buf,12); #read in type, header size, and data size
	if ($number_read_1 != 12){
		print "end of file\n";
		last;
	}
	$filesize += $number_read_1;
	curpos;
	
	$number_read_2=0;
	$number_read_2=read(VID,$timebuf,8); 
	if ($number_read_2 != 8){
		print "end of file\n";
		last;
	}
	curpos;
	$filesize += $number_read_2;
	
	$header_read = $number_read_1+$number_read_2;
	$pos_2=tell(VID);
	
	if($pos_1 + ($header_read) != $pos_2) {
		error("read in too many");
	}

where curpos prints out the file index, and error is defined as:

code:

sub error {
	curpos;
		$filesizek=$filesize/1000;
		print "total filesize read in (in kilobytes)= $filesizek";
		$arg=$_[0];
		die "$arg: $!";
	}

Now, when I run the code without the second read term (and the stuff that goes along with it), it runs fine, and I end up stepping through the file just fine. But when I read in more than 12 bytes, either by running with the second read, or by keeping the second read commented out and just reading in more bytes in the first read, the error subroutine kills the program, and I am not sure how this is possible. It must be reading in more than I tell it to, or skipping or something, but I have no idea how. The weird thing is that it only happens a ways into the binary file, after running properly for a good number of times.

Another thing: I have personally looked at the file in a Hex editor and have done the math myself and I made sure that the file itself was giving me the correct data. The header is as long as it says it is and so is the data.

Does anyone have any experience with anything like this?

# ? Jul 14, 2009 17:07

Erasmus Darwin: Mar 6, 2001

This is a bit of a wild guess, but you aren't doing this on a Windows machine, are you? If so, you'll need to do "binmode(VID);" after you open it. From reading the manpages, I'm not sure that would be an issue, but it seems like it potentially could be.

# ? Jul 14, 2009 19:03

Captain Frigate: Apr 30, 2007; you cant have it, you dont have nuff teef to chew it

Erasmus Darwin posted:

This is a bit of a wild guess, but you aren't doing this on a Windows machine, are you? If so, you'll need to do "binmode(VID);" after you open it. From reading the manpages, I'm not sure that would be an issue, but it seems like it potentially could be.

Wow, worked like a charm! What does that actually do? The differing end-of-line conventions wouldn't really matter in this situation, would they?

Captain Frigate fucked around with this message at 19:43 on Jul 14, 2009

# ? Jul 14, 2009 19:35

tef: May 30, 2004; -> some l-system crap ->

Captain Frigate posted:

Wow, worked like a charm! What does that actually do? The differing end-of-line conventions wouldn't really matter in this situation, would they?

http://perldoc.perl.org/functions/binmode.html

quote:

if you don't use binmode() on these systems, \cM\cJ sequences on disk will be converted to \n on input, and any \n in your program will be converted back to \cM\cJ on output. This is what you want for text files, but it can be disastrous for binary files.

# ? Jul 14, 2009 19:51

Erasmus Darwin: Mar 6, 2001

The short answer is that I'm not entirely sure. The manpage more or less says that if it's a binary file and there's any doubt, you should just go ahead and binmode it to be safe.

The longer answer is that some of the file I/O functions use a notion of "characters" where character is defined based on the file mode, the file contents, and so on. To complicate matters, other file I/O functions (such as seek and tell) always use bytes instead of characters for efficiency reasons. The manpage for read implies that it uses characters but that the default method is to treat them as bytes -- apparently it's not quite that simple since binmode fixed it.

Presumably, you were running across something where the character count and the byte count differed. Maybe it was a \r\n combo in a string. Maybe it was doing some utf8 weirdness. I'm not really sure since I haven't delved into how it handles that sort of stuff.

# ? Jul 14, 2009 19:53

Captain Frigate: Apr 30, 2007; you cant have it, you dont have nuff teef to chew it

tef posted:

http://perldoc.perl.org/functions/binmode.html

Well, I guess that would explain what I've got going on over here and why it only happens intermittently. Sorry to be such a nuisance, I'd never used perl before about two weeks ago when they told me to start on this project and told me I'd be using it. Thanks!

# ? Jul 14, 2009 20:10

Mithaldu: Sep 25, 2007; Let's cuddle.

I'm currently writing tests for a web application.

One thing I'd like to test is a function that stores a message but also sends out an email to the receiver. Right now I'm thinking of doing it this way:

code:

#!/usr/bin/perl -w

use strict;
use warnings;

use Test::More 'no_plan';

BEGIN { use_ok('WNUser'); }

{	
	my @emails;
	local *WNUser::send_email = sub { push @emails, $_[1] };
	my $app = new WNUser;
	my $output = $app->send_message;

	# checks for $output and @emails here
}

Is this a sane way to do that or am i overlooking better ways?

Also, how can i prevent it from complaining about WNUser::send_email being used only once?

# ? Jul 15, 2009 10:33

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

Mithaldu posted:

Is this a sane way to do that or am i overlooking better ways?

Seems fine. If you use something like Email::Sender then you can do much better. You can swap out the actual email logic with an environment variable. It can add emails to a SQLite database so that you can test email you send in forked processes.

Mithaldu posted:

Also, how can i prevent it from complaining about WNUser::send_email being used only once?

code:

no warnings 'redefine';

# ? Jul 15, 2009 10:42

Mithaldu: Sep 25, 2007; Let's cuddle.

Sartak posted:

Seems fine. If you use something like Email::Sender then you can do much better. You can swap out the actual email logic with an environment variable. It can add emails to a SQLite database so that you can test email you send in forked processes.

Yes, in fact i do, Email::Sender::Simple to be exact.

code:

sub sendmail_text {
    my ( $cyc, $option_hash ) = @_;

    $option_hash->{from} ||= $wn_email;

    my $email = Email::Simple->create(
        header => [
            To      => $option_hash->{to},
            From    => $option_hash->{from},
            Subject => $option_hash->{subject},
        ],
        body => $option_hash->{body},
    );

    sendmail($email);
}

I originally wanted to just replace this function, as i don't need to test whether sendmail works, only whether it's trying to send the right mails. How would i go about replacing the logic?

Sartak posted:

code:

no warnings 'redefine';

Ah, thanks.

# ? Jul 15, 2009 10:52

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

Mithaldu posted:

I originally wanted to just replace this function, as i don't need to test whether sendmail works, only whether it's trying to send the right mails. How would i go about replacing the logic?

See http://search.cpan.org/~rjbs/Email-Sender-0.091940/lib/Email/Sender/Manual/QuickStart.pm#Picking_a_Transport

# ? Jul 15, 2009 11:00

Mithaldu: Sep 25, 2007; Let's cuddle.

Oh wow, complete with example for testing, nice. Thanks.

# ? Jul 15, 2009 11:05

Mario Incandenza: Aug 24, 2000; Tell me, small fry, have you ever heard of the golden Triumph Forks?

Sartak posted:

code:

no warnings 'redefine';

I think redefine warnings only get triggered when you've got a sub foo declaration that overwrites an existing one, I'm not sure if assigning directly to a glob will raise it. In this case no warnings 'once' is probably the one Mithaldu's looking for.

# ? Jul 15, 2009 11:39

Mithaldu: Sep 25, 2007; Let's cuddle.

Didn't try it out until now, but you're right. 'once' is indeed the correct tag for that.

# ? Jul 15, 2009 11:50

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

atomicstack posted:

In this case no warnings 'once' is probably the one Mithaldu's looking for.

Oh you're right. I answered the question in my head, not the one Mithaldu asked. Sorry about that.

# ? Jul 15, 2009 11:52

Dijkstracula: Mar 18, 2003; You can't spell 'vector field' without me, Professor!

Quick question: I've got a large file that I've mmapped using PerlIO. Should I be accessing it through the buffered io functions (read(), write(), etc) or the unbuffered ones (sysread(), syswrite()), or does it matter (edit: so long as I'm consistent)?

Dijkstracula fucked around with this message at 21:16 on Jul 15, 2009

# ? Jul 15, 2009 20:56

freedrull: May 2, 2009; happy burger wednesdays @ fortchan.org

I'm trying to make an irc bot with Bot::BasicBot...but I cannot figure out how to produce color text!

This SHOULD produce bold text:

code:

	$self->say( 
		channel => "#blahblah",
		body => "\0x02 poop \0x02",
	);

But instead produces nothing! However plain text works just fine.

code:

	$self->say( 
		channel => "#blahblah",
		body => "poop",
	);

Is it possible colors are not supported by this module??? Hmm...

# ? Jul 18, 2009 23:53

tef: May 30, 2004; -> some l-system crap ->

I would assume this line is mangling them:

my ($who, $body) = $self->charset_encode($who, $body);

# ? Jul 19, 2009 00:15

freedrull: May 2, 2009; happy burger wednesdays @ fortchan.org

tef posted:

I would assume this line is mangling them:

my ($who, $body) = $self->charset_encode($who, $body);

Hmmm I think so too....perhaps by specifying the charset I can fix it?

charset=>"utf-8",

code:

MyBot->new(
    server => 'paradox.esper.net',
    port => '6666',
    channels => [ '#vlank_channel' ],
    nick => 'vblank',
    charset=> "utf-8",
)->run();

Doesn't seem to work either though....

# ? Jul 19, 2009 00:31

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

Use String::IRC. It just works with Bot::BasicBot.

# ? Jul 19, 2009 11:01

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

Oh dear I'm giving another talk. This time at YAPC::Asia, September 10-11 in Tokyo.

API Design posted:

Too few projects demand good API design as a critical goal. A clean and extensible API will pay for itself many times over in fostering a community of plugins. We certainly cannot anticipate the ways in which our users will bend our modules, but designing an extensible system alleviates the pain. There are many lessons to be learned from Moose, HTTP::Engine and IM::Engine, Dist::Zilla, KiokuDB, Fey, and TAEB.

The most important lesson is to decouple the core functionality from the "fluff" such as sugar and middleware. This forces you to have a solid API that ordinary users can extend. This also lets users write their own sugar and middleware. In a tightly-coupled system, there is little hope for extensibility.

In this talk, you will learn how to make very productive use of Moose's roles to form the foundation of a pluggable system. Roles provide excellent means of code reuse and safe composition. I will also demonstrate how to use Sub::Exporter to construct a more useful and flexible sugar layer.

# ? Jul 20, 2009 14:20

CanSpice: Jan 12, 2002; GO CANUCKS GO

Sartak posted:

Oh dear I'm giving another talk. This time at YAPC::Asia, September 10-11 in Tokyo.

YAPC more like YAMC.

# ? Jul 20, 2009 19:55

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

Moose owns :shobon:

# ? Jul 20, 2009 20:08

Mario Incandenza: Aug 24, 2000; Tell me, small fry, have you ever heard of the golden Triumph Forks?

Moose roles, types + coercion, and MX::AH have greatly improved my quality of life at work, no exaggeration. The time I used to spend writing fiddly low-level bullshit I now use making more classes and glue, it's awesome.

# ? Jul 21, 2009 16:03

CanSpice: Jan 12, 2002; GO CANUCKS GO

Yeah, Moose is all fine and good if you're writing new classes, but when you have legacy classes and you can't commit the time to re-doing them with Moose, all the Moose talk gets a little old.

My latest foray into exciting Perl stuff is Shipwright. Anybody ever used it? Whenever I had to release our software I had to spend a week installing all the Perl stuff. Now I should just have to check out my Shipwright repository on each platform, build it, and away I go. Should only take a day, hopefully.

# ? Jul 21, 2009 20:04

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

CanSpice posted:

Yeah, Moose is all fine and good if you're writing new classes, but when you have legacy classes and you can't commit the time to re-doing them with Moose, all the Moose talk gets a little old.

I don't really buy this. Moose classes and legacy classes can coexist just fine.

CanSpice posted:

My latest foray into exciting Perl stuff is Shipwright. Anybody ever used it? Whenever I had to release our software I had to spend a week installing all the Perl stuff. Now I should just have to check out my Shipwright repository on each platform, build it, and away I go. Should only take a day, hopefully.

My coworker built it and I hacked a bit on the prototype. Let me know if you need help.

# ? Jul 21, 2009 20:20

CanSpice: Jan 12, 2002; GO CANUCKS GO

Sartak posted:

I don't really buy this. Moose classes and legacy classes can coexist just fine.

Yeah, I just don't write new classes much these days. I'm also a little hesitant to bring in yet another dependency, although it looks like Moose's dependencies are fairly lightweight. Hell, it's no Tk or PDL.

quote:

My coworker built it and I hacked a bit on the prototype. Let me know if you need help.

Once I worked around a couple of bugs (I filed three with RT, now I just need to reply to sunnavy's replies to them) it seems to be going fairly well. Once you figure out what's going on (which isn't easy -- the documentation isn't all that great) it's fairly smooth sailing. I'm writing a blog post outlining the steps I took to create my vessel in hopes that it helps other people.

# ? Jul 21, 2009 21:42

Triple Tech: Jul 28, 2006; So what, are you quitting to join Homo Explosion?

Ships? Sailing? :roflolmao:

# ? Jul 21, 2009 21:54

Filburt Shellbach: Nov 6, 2007; Apni tackat say tujay aaj mitta juu gaa!

CanSpice posted:

Yeah, I just don't write new classes much these days. I'm also a little hesitant to bring in yet another dependency, although it looks like Moose's dependencies are fairly lightweight. Hell, it's no Tk or PDL.

That's entirely fair. If you're not writing new classes, converting to Moose would be a tough sell. I do think Moosing your code would ultimately improve code quality and robustness, but it is hard to justify that on any budget.

CanSpice posted:

it seems to be going fairly well [...] I'm writing a blog post outlining the steps I took to create my vessel in hopes that it helps other people.

Great! It's cool to see Shipwright start getting users.

# ? Jul 21, 2009 21:56

Adbot: ADBOT LOVES YOU

# ? May 17, 2024 14:08

SA Support Robot: Apr 20, 2007

Does anyone here use AnyEvent? I am madly in love with it and wish to project my love upon others.

:swoon:

# ? Jul 23, 2009 18:02

The Something Awful Forums > Discussion > Serious Hardware/Software Crap > The Cavern of COBOL > The Perl Short Questions Megathread: executable line noise

«‹›72 »