Store metadata in the index, in bupindex.meta; only store unique values.

[bup.git] / DESIGN

diff --git a/DESIGN b/DESIGN
index 8a00073e454cbfe42191dc2a79154a2122845f0d..7d2b64d6307f2fbc509c116ccf160696f056cc61 100644 (file)
--- a/DESIGN
+++ b/DESIGN
@@ -395,9 +395,27 @@ it did before the addition of metadata, and restore files using the
 tree information.
 
 The nice thing about this design is that you can walk through each
-file in a tree just by opening the tree and the .bupmeta contents, and
+file in a tree just by opening the tree and the .bupm contents, and
 iterating through both at the same time.
 
+Since the contents of any .bupm file should match the state of the
+filesystem when it was *indexed*, bup must record the detailed
+metadata in the index.  To do this, bup records four values in the
+index, the atime, mtime, and ctime (as timespecs), and an integer
+offset into a secondary "metadata store" which has the same name as
+the index, but with ".meta" appended.  This secondary store contains
+the encoded Metadata object corresponding to each path in the index.
+
+Currently, in order to decrease the storage required for the metadata
+store, bup only writes unique values there, reusing offsets when
+appropriate across the index.  The effectiveness of this approach
+relies on the expectation that there will be many duplicate metadata
+records.  Storing the full timestamps in the index is intended to make
+that more likely, because it makes it unnecessary to record those
+values in the secondary store.  So bup clears them before encoding the
+Metadata objects destined for the index, and timestamp differences
+don't contribute to the uniqueness of the metadata.
+
 Bup supports recording and restoring hardlinks, and it does so by
 tracking sets of paths that correspond to the same dev/inode pair when
 indexing.  This information is stored in an optional file with the