DESIGN: update mentions of stupidsum to reflect new rollsum algorithm.

Pointed out by Gabriel Filion.

Signed-off-by: Avery Pennarun <apenwarr@gmail.com>

diff --git a/DESIGN b/DESIGN
index c0d9fb7d9fdd8813221c43a90405777d91e80f25..bd1576e353a4fa62718cbc3d48231f2cc64231be 100644 (file)
--- a/DESIGN
+++ b/DESIGN
@@ -125,28 +125,35 @@ anything over a few megabytes in size, bup's compression will actually
 *work*, which is a bit advantage over xdelta.
 
 How does hashsplitting work?  It's deceptively simple.  We read through the
-file one byte at a time, calculating a rolling checksum of the last 32
-bytes.  (Why 32?  No reason.  Literally.  We picked it out of the air. 
+file one byte at a time, calculating a rolling checksum of the last 128
+bytes.  (Why 128?  No reason.  Literally.  We picked it out of the air. 
 Probably some other number is better.  Feel free to join the mailing list
 and tell us which one and why.)  (The rolling checksum idea is actually
-stolen from rsync and xdelta, although we use it differently.)
+stolen from rsync and xdelta, although we use it differently.  And they use
+some kind of variable window size based on a formula we don't totally
+understand.)
 
-The particular rolling checksum algorithm we use is called "stupidsum,"
-because it's based on the only checksum Avery remembered how to calculate at
+The original rolling checksum algorithm we used was called "stupidsum,"
+because it was based on the only checksum Avery remembered how to calculate at
 the time.  He also remembered that it was the introductory checksum
 algorithm in a whole article about how to make good checksums that he read
 about 15 years ago, and it was thoroughly discredited in that article for
 being very stupid.  But, as so often happens, Avery couldn't remember any
-better algorithms from the article.  So what we get is stupidsum.  (If
-you're a computer scientist and can demonstrate that some other rolling
+better algorithms from the article.  So what we got is stupidsum.
+
+Since then, we have replaced the stupidsum algorithm with what we call
+"rollsum," based on code in librsync.  It's essentially the same as what
+rsync does, except we use a fixed window size.
+
+(If you're a computer scientist and can demonstrate that some other rolling
 checksum would be faster and/or better and/or have fewer screwy edge cases,
 we need your help!  Avery's out of control!  Join our mailing list!  Please! 
 Save us! ...  oh boy, I sure hope he doesn't read this)
 
-In any case, stupidsum, although stupid, seems to do pretty well at its job. 
-You can find it in bupsplit.c.  Basically, it converts the last 32 bytes
-of the file into a 32-bit integer.  What we then do is take the lowest 13
-bits of the checksum, and if they're all 1's, we consider that to be the end
+In any case, rollsum seems to do pretty well at its job. 
+You can find it in bupsplit.c.  Basically, it converts the last 128 bytes
+read into a 32-bit integer.  What we then do is take the lowest 13
+bits of the rollsum, and if they're all 1's, we consider that to be the end
 of a chunk.  This happens on average once every 2^13 = 8192 bytes, so the
 average chunk size is 8192 bytes.
 
@@ -181,7 +188,7 @@ file, all the chunks *before* and *after* the affected chunk are absolutely
 the same.  All that matters to the hashsplitting algorithm is the 32-byte
 "separator" sequence, and a single change can only affect, at most, one
 separator sequence or the bytes between two separator sequences.  And
-because of stupidsum, about one in 8192 possible 32-byte sequences is a
+because of rollsum, about one in 8192 possible 128-byte sequences is a
 separator sequence.  Like magic, the hashsplit chunking algorithm will chunk
 your file the same way every time, even without knowing how it had chunked
 it previously.