1 % bup-midx(1) Bup %BUP_VERSION%
2 % Avery Pennarun <apenwarr@gmail.com>
7 bup-midx - create a multi-index (`.midx`) file from several `.idx` files
11 bup midx [-o *outfile*] \<-a|-f|*idxnames*...\>
15 `bup midx` creates a multi-index (`.midx`) file from one or more
16 git pack index (`.idx`) files.
18 Note: you should no longer need to run this command by hand.
19 It gets run automatically by `bup-save`(1) and similar
24 -o, \--output=*filename.midx*
25 : use the given output filename for the `.midx` file.
26 Default is auto-generated.
29 : automatically generate new `.midx` files for any `.idx`
30 files where it would be appropriate.
33 : force generation of a single new `.midx` file containing
34 *all* your `.idx` files, even if other `.midx` files
35 already exist. This will result in the fastest backup
36 performance, but may take a long time to run.
39 : specify the directory containing the `.idx`/`.midx` files
40 to work with. The default is $BUP_DIR/objects/pack and
41 $BUP_DIR/indexcache/*.
44 : maximum number of `.idx` files to open at a time. You
45 can use this if you have an especially small number of file
46 descriptors available, so that midx can complete
47 (though possibly non-optimally) even if it can't open
48 all your `.idx` files at once. The default value of this
49 option should be fine for most people.
52 : validate a `.midx` file by ensuring that all objects in
53 its contained `.idx` files exist inside the `.midx`. May
54 be useful for debugging.
59 Merging 21 indexes (2278559 objects).
60 Table size: 524288 (17 bits)
61 Reading indexes: 100.00% (2278559/2278559), done.
62 midx-b66d7c9afc4396187218f2936a87b865cf342672.midx
66 By default, bup uses git-formatted pack files, which
67 consist of a pack file (containing objects) and an idx
68 file (containing a sorted list of object names and their
69 offsets in the .pack file).
71 Normal idx files are convenient because it means you can use
72 `git`(1) to access your backup datasets. However, idx
73 files can get slow when you have a lot of very large packs
74 (which git typically doesn't have, but bup often does).
76 bup `.midx` files consist of a single sorted list of all the objects
77 contained in all the .pack files it references. This list
78 can be binary searched in about log2(m) steps, where m is
79 the total number of objects.
81 To further speed up the search, midx files also have a
82 variable-sized fanout table that reduces the first n
83 steps of the binary search. With the help of this fanout
84 table, bup can narrow down which page of the midx file a
85 given object id would be in (if it exists) with a single
86 lookup. Thus, typical searches will only need to swap in
87 two pages: one for the fanout table, and one for the object
90 midx files are most useful when creating new backups, since
91 searching for a nonexistent object in the repository
92 necessarily requires searching through *all* the index
93 files to ensure that it does not exist. (Searching for
94 objects that *do* exist can be optimized; for example,
95 consecutive objects are often stored in the same pack, so
96 we can search that one first using an MRU algorithm.)
101 `bup-save`(1), `bup-margin`(1), `bup-memtest`(1)
105 Part of the `bup`(1) suite.