Avery Pennarun [Wed, 30 Dec 2009 08:28:36 +0000 (03:28 -0500)]
Use t# instead of et# for hashsplit parameter type.
This lets us work with any kind of buffer object, which means there's no
unnecessary data copying when coming into our module. Causes a bit of
speedup.
Also refactored the checksum code for easier experimentation.
Avery Pennarun [Wed, 30 Dec 2009 07:33:35 +0000 (02:33 -0500)]
Add a C module to do the rolling checksum.
This is about 80x faster than the old speed (27megs/sec instead of 330k/sec)
but still quite a lot slower than the 60+megs/sec I get *without* the
checksum stuff. There are a few inefficiencies remaining, but not such easy
ones as before...
Avery Pennarun [Wed, 30 Dec 2009 06:08:27 +0000 (01:08 -0500)]
datagen.c: a quick program to generate a repeatable series of bytes.
Useful for testing. Note that we *don't* see the random number generator,
so every time you generate the bytes, you get the same sequence.
This is also vastly faster than /dev/urandom, since it doesn't try to be
cryptographically secure. It generates about 200 megs/sec on my computer,
which is much faster than a disk and thus useful for testing the speed of
hashsplit.