1 % bup-midx(1) Bup %BUP_VERSION%
2 % Avery Pennarun <apenwarr@gmail.com>
7 bup-midx - create a multi-index (.midx) file from several .idx files
11 bup midx [-o *outfile*] <-a|-f|*idxnames*...>
15 `bup midx` creates a multi-index (.midx) file from one or more
16 git pack index (.idx) files.
18 Note: you should no longer need to run this command by hand.
19 It gets run automatically by `bup-save`(1) and similar
24 -o, --output=*filename.midx*
25 : use the given output filename for the .midx file.
26 Default is auto-generated.
29 : automatically generate new .midx files for any .idx
30 files where it would be appropriate.
33 : force generation of a single new .midx file containing
34 *all* your .idx files, even if other .midx files
35 already exist. This will result in the fastest backup
36 performance, but may take a long time to run.
39 : specify the directory containing the .idx/.midx files
40 to work with. The default is $BUP_DIR/objects/pack and
41 $BUP_DIR/indexcache/*.
44 : maximum number of .idx files to open at a time. You
45 can use this if you have an especially small number of file
46 descriptors available, so that midx can complete
47 (though possibly non-optimally) even if it can't open
48 all your .idx files at once. The default value of this
49 option should be fine for most people.
52 : validate a .midx file by ensuring that all objects in
53 its contained .idx files exist inside the .midx. May
54 be useful for debugging.
60 Merging 21 indexes (2278559 objects).
61 Table size: 524288 (17 bits)
62 Reading indexes: 100.00% (2278559/2278559), done.
63 midx-b66d7c9afc4396187218f2936a87b865cf342672.midx
67 By default, bup uses git-formatted pack files, which
68 consist of a pack file (containing objects) and an idx
69 file (containing a sorted list of object names and their
70 offsets in the .pack file).
72 Normal idx files are convenient because it means you can use
73 `git`(1) to access your backup datasets. However, idx
74 files can get slow when you have a lot of very large packs
75 (which git typically doesn't have, but bup often does).
77 bup .midx files consist of a single sorted list of all the objects
78 contained in all the .pack files it references. This list
79 can be binary searched in about log2(m) steps, where m is
80 the total number of objects.
82 To further speed up the search, midx files also have a
83 variable-sized fanout table that reduces the first n
84 steps of the binary search. With the help of this fanout
85 table, bup can narrow down which page of the midx file a
86 given object id would be in (if it exists) with a single
87 lookup. Thus, typical searches will only need to swap in
88 two pages: one for the fanout table, and one for the object
91 midx files are most useful when creating new backups, since
92 searching for a nonexistent object in the repository
93 necessarily requires searching through *all* the index
94 files to ensure that it does not exist. (Searching for
95 objects that *do* exist can be optimized; for example,
96 consecutive objects are often stored in the same pack, so
97 we can search that one first using an MRU algorithm.)
102 `bup-save`(1), `bup-margin`(1), `bup-memtest`(1)
106 Part of the `bup`(1) suite.