1 % bup-memtest(1) Bup %BUP_VERSION%
2 % Avery Pennarun <apenwarr@gmail.com>
7 bup-memtest - test bup memory usage statistics
11 bup memtest [options...]
15 `bup memtest` opens the list of pack indexes in your bup
16 repository, then searches the list for a series of
17 nonexistent objects, printing memory usage statistics after
20 Because of the way Unix systems work, the output will
21 usually show a large (and unchanging) value in the VmSize
22 column, because mapping the index files in the first place
23 takes a certain amount of virtual address space. However, this
24 virtual memory usage is entirely virtual; it doesn't take
25 any of your RAM. Over time, bup uses *parts* of the
26 indexes, which need to be loaded from disk, and this is
27 what causes an increase in the VmRSS column.
31 -n, \--number=*number*
32 : set the number of objects to search for during each
33 cycle (ie. before printing a line of output)
35 -c, \--cycles=*cycles*
36 : set the number of cycles (ie. the number of lines of
37 output after the first). The first line of output is
38 always 0 (ie. the baseline before searching for any
42 : ignore any `.midx` files created by `bup midx`. This
43 allows you to compare memory performance with and
47 : search for existing objects instead of searching for
48 random nonexistent ones. This can greatly affect
49 memory usage and performance. Note that most of the
50 time, `bup save` spends most of its time searching for
51 nonexistent objects, since existing ones are probably
52 in unmodified files that we won't be trying to back up
53 anyway. So the default behaviour reflects real bup
54 performance more accurately. But you might want this
55 option anyway just to make sure you haven't made
56 searching for existing objects much worse than before.
60 $ bup memtest -n300 -c5
61 PackIdxList: using 1 index.
62 VmSize VmRSS VmData VmStk
63 0 20824 kB 4528 kB 1980 kB 84 kB
64 300 20828 kB 5828 kB 1984 kB 84 kB
65 600 20828 kB 6844 kB 1984 kB 84 kB
66 900 20828 kB 7836 kB 1984 kB 84 kB
67 1200 20828 kB 8736 kB 1984 kB 84 kB
68 1500 20828 kB 9452 kB 1984 kB 84 kB
70 $ bup memtest -n300 -c5 --ignore-midx
71 PackIdxList: using 361 indexes.
72 VmSize VmRSS VmData VmStk
73 0 27444 kB 6552 kB 2516 kB 84 kB
74 300 27448 kB 15832 kB 2520 kB 84 kB
75 600 27448 kB 17220 kB 2520 kB 84 kB
76 900 27448 kB 18012 kB 2520 kB 84 kB
77 1200 27448 kB 18388 kB 2520 kB 84 kB
78 1500 27448 kB 18556 kB 2520 kB 84 kB
83 When optimizing bup indexing, the first goal is to keep the
84 VmRSS reasonably low. However, it might eventually be
85 necessary to swap in all the indexes, simply because
86 you're searching for a lot of objects, and this will cause
87 your RSS to grow as large as VmSize eventually.
89 The key word here is *eventually*. As long as VmRSS grows
90 reasonably slowly, the amount of disk activity caused by
91 accessing pack indexes is reasonably small. If it grows
92 quickly, bup will probably spend most of its time swapping
93 index data from disk instead of actually running your
94 backup, so backups will run very slowly.
96 The purpose of `bup memtest` is to give you an idea of how
97 fast your memory usage is growing, and to help in
98 optimizing bup for better memory use. If you have memory
99 problems you might be asked to send the output of `bup
100 memtest` to help diagnose the problems.
102 Tip: try using `bup midx -a` or `bup midx -f` to see if it
103 helps reduce your memory usage.
105 Trivia: index memory usage in bup (or git) is only really a
106 problem when adding a large number of previously unseen
107 objects. This is because for each object, we need to
108 absolutely confirm that it isn't already in the database,
109 which requires us to search through *all* the existing pack
110 indexes to ensure that none of them contain the object in
111 question. In the more obvious case of searching for
112 objects that *do* exist, the objects being searched for are
113 typically related in some way, which means they probably
114 all exist in a small number of packfiles, so memory usage
115 will be constrained to just those packfile indexes.
117 Since git users typically don't add a lot of files in a
118 single run, git doesn't really need a program like `bup
119 midx`. bup, on the other hand, spends most of its time
120 backing up files it hasn't seen before, so its memory usage
121 patterns are different.
130 Part of the `bup`(1) suite.