Merge branch 'master' of git://github.com/thatch/bup

[bup.git] / DESIGN

diff --git a/DESIGN b/DESIGN
index bd1576e353a4fa62718cbc3d48231f2cc64231be..dfe30f36062ffbf8cc2fb623792ef1b87271cf8a 100644 (file)
--- a/DESIGN
+++ b/DESIGN
@@ -55,15 +55,9 @@ Essentially, copying data from the filesystem to your repository is called
 a backup using the 'bup save' command, but that's getting ahead of
 ourselves.
 
-As most backup experts know, backing stuff up is normally about 100x more
-common than restoring stuff, ie.  copying from the repository to your
-filesystem.  For that reason, and also because bup is so new, there is no
-actual 'bup restore' command that does the obvious inverse operation to 'bup
-save'.  There are 'bup ftp' and 'bup fuse', which let you access your
-backed-up data, but they aren't as efficient as a fully optimized restore
-tool intended for high-volume restores.  There's nothing stopping us from
-writing one; we just haven't written it yet.  Feel free to pester us about
-it on the bup mailing list (see the README to find out about the list).
+For the inverse operation, ie. copying from the repository to your
+filesystem, you have several choices; the main ones are 'bup restore', 'bup
+ftp', 'bup fuse', and 'bup web'.
 
 Now, those are the basics of backups.  In other words, we just spent about
 half a page telling you that bup backs up and restores data.  Are we having
@@ -287,7 +281,7 @@ they're written.
 But that leads us to our next problem.
 
 
-Huge numbers of huge packfiles (git.PackMidx, cmd/midx)
+Huge numbers of huge packfiles (midx.py, bloom.py, cmd/midx, cmd/bloom)
 ------------------------------
 
 Git isn't actually designed to handle super-huge repositories.  Most git
@@ -360,9 +354,13 @@ You generate midx files with 'bup midx'.  The downside of midx files is that
 generating one takes a while, and you have to regenerate it every time you
 add a few packs.
 
-(Computer Sciency observers will note that there are some interesting data
-structures out there that could help make things even better.  A very
-promising sounding one is called a "bloom filter." Look it up in Wikipedia.)
+UPDATE: Brandon Low contributed an implementation of "bloom filters", which
+have even better characteristics than midx for certain uses.  Look it up in
+Wikipedia.  He also massively sped up both midx and bloom by rewriting the
+key parts in C.  The nicest thing about bloom filters is we can update them
+incrementally every time we get a new idx, without regenerating from
+scratch.  That makes the update phase much faster, and means we can also get
+away with generating midxes less often.
 
 midx files are a bup-specific optimization and git doesn't know what to do
 with them.  However, since they're stored as separate files, they don't
@@ -568,8 +566,9 @@ compare the files in the index against the ones in the backup set, and
 update only the ones that have changed.  (Even more interesting things
 happen if people are using the files on the restored system and you haven't
 updated the index yet; the net result would be an automated merge of all
-non-conflicting files.)  This would be a poor man's distributed filesystem. 
-The only catch is that nobody has written 'bup restore' yet.  Someday!
+non-conflicting files.) This would be a poor man's distributed filesystem. 
+The only catch is that nobody has written this feature for 'bup restore'
+yet.  Someday!
 
 
 How 'bup save' works (cmd/save)