1 % bup-memtest(1) Bup %BUP_VERSION%
2 % Avery Pennarun <apenwarr@gmail.com>
7 bup-memtest - test bup memory usage statistics
11 bup memtest [options...]
15 `bup memtest` opens the list of pack indexes in your bup
16 repository, then searches the list for a series of
17 nonexistent objects, printing memory usage statistics after
20 Because of the way Unix systems work, the output will
21 usually show a large (and unchanging) value in the VmSize
22 column, because mapping the index files in the first place
23 takes a certain amount of virtual address space. However, this
24 virtual memory usage is entirely virtual; it doesn't take
25 any of your RAM. Over time, bup uses *parts* of the
26 indexes, which need to be loaded from disk, and this is
27 what causes an increase in the VmRSS column.
31 -n, \--number=*number*
32 : set the number of objects to search for during each
33 cycle (ie. before printing a line of output)
35 -c, \--cycles=*cycles*
36 : set the number of cycles (ie. the number of lines of
37 output after the first). The first line of output is
38 always 0 (ie. the baseline before searching for any
42 : ignore any `.midx` files created by `bup midx`. This
43 allows you to compare memory performance with and
47 : search for existing objects instead of searching for
48 random nonexistent ones. This can greatly affect
49 memory usage and performance. Note that most of the
50 time, `bup save` spends most of its time searching for
51 nonexistent objects, since existing ones are probably
52 in unmodified files that we won't be trying to back up
53 anyway. So the default behaviour reflects real bup
54 performance more accurately. But you might want this
55 option anyway just to make sure you haven't made
56 searching for existing objects much worse than before.
61 $ bup memtest -n300 -c5
62 PackIdxList: using 1 index.
63 VmSize VmRSS VmData VmStk
64 0 20824 kB 4528 kB 1980 kB 84 kB
65 300 20828 kB 5828 kB 1984 kB 84 kB
66 600 20828 kB 6844 kB 1984 kB 84 kB
67 900 20828 kB 7836 kB 1984 kB 84 kB
68 1200 20828 kB 8736 kB 1984 kB 84 kB
69 1500 20828 kB 9452 kB 1984 kB 84 kB
71 $ bup memtest -n300 -c5 --ignore-midx
72 PackIdxList: using 361 indexes.
73 VmSize VmRSS VmData VmStk
74 0 27444 kB 6552 kB 2516 kB 84 kB
75 300 27448 kB 15832 kB 2520 kB 84 kB
76 600 27448 kB 17220 kB 2520 kB 84 kB
77 900 27448 kB 18012 kB 2520 kB 84 kB
78 1200 27448 kB 18388 kB 2520 kB 84 kB
79 1500 27448 kB 18556 kB 2520 kB 84 kB
84 When optimizing bup indexing, the first goal is to keep the
85 VmRSS reasonably low. However, it might eventually be
86 necessary to swap in all the indexes, simply because
87 you're searching for a lot of objects, and this will cause
88 your RSS to grow as large as VmSize eventually.
90 The key word here is *eventually*. As long as VmRSS grows
91 reasonably slowly, the amount of disk activity caused by
92 accessing pack indexes is reasonably small. If it grows
93 quickly, bup will probably spend most of its time swapping
94 index data from disk instead of actually running your
95 backup, so backups will run very slowly.
97 The purpose of `bup memtest` is to give you an idea of how
98 fast your memory usage is growing, and to help in
99 optimizing bup for better memory use. If you have memory
100 problems you might be asked to send the output of `bup
101 memtest` to help diagnose the problems.
103 Tip: try using `bup midx -a` or `bup midx -f` to see if it
104 helps reduce your memory usage.
106 Trivia: index memory usage in bup (or git) is only really a
107 problem when adding a large number of previously unseen
108 objects. This is because for each object, we need to
109 absolutely confirm that it isn't already in the database,
110 which requires us to search through *all* the existing pack
111 indexes to ensure that none of them contain the object in
112 question. In the more obvious case of searching for
113 objects that *do* exist, the objects being searched for are
114 typically related in some way, which means they probably
115 all exist in a small number of packfiles, so memory usage
116 will be constrained to just those packfile indexes.
118 Since git users typically don't add a lot of files in a
119 single run, git doesn't really need a program like `bup
120 midx`. bup, on the other hand, spends most of its time
121 backing up files it hasn't seen before, so its memory usage
122 patterns are different.
131 Part of the `bup`(1) suite.