cmd/midx: add a new --max-files parameter.

Zoran reported that 'bup midx -f' on his system tried to open 3000 files at
a time and wouldn't work.  That's no good, so let's limit the maximum files
to open; the default is 500 for now, since that ought to be usable for
normal people.  Arguably we could use getrlimit() or something to find out
the actual maximum, or just keep opening stuff until we get an error, but
maybe there's no point.

Unfortunately this patch isn't really perfect, because it limits the
usefulness of midx files.  If you could merge midx files into other midx
files, then you could at least group them all together after multiple runs,
but that's not currently supported.

Signed-off-by: Avery Pennarun <apenwarr@gmail.com>

diff --git a/cmd/midx-cmd.py b/cmd/midx-cmd.py
index bbfd2acc0d2a562b26c755f64636f8debdd0467d..579f78a1504c2bbda55f9d7d5020872c9932b6c4 100755 (executable)
--- a/cmd/midx-cmd.py
+++ b/cmd/midx-cmd.py
@@ -7,6 +7,11 @@ PAGE_SIZE=4096
 SHA_PER_PAGE=PAGE_SIZE/20.
 
 
+def _group(l, count):
+    for i in xrange(0, len(l), count):
+        yield l[i:i+count]
+
+
 def merge(idxlist, bits, table):
     count = 0
     for e in git.idxmerge(idxlist):
@@ -81,6 +86,7 @@ bup midx [options...] <idxnames...>
 o,output=  output midx filename (default: auto-generated)
 a,auto     automatically create .midx from any unindexed .idx files
 f,force    automatically create .midx from *all* .idx files
+max-files= maximum number of idx files to open at once [500]
 """
 o = options.Options('bup midx', optspec)
 (opt, flags, extra) = o.parse(sys.argv[1:])
@@ -98,7 +104,9 @@ elif opt.auto or opt.force:
     for path in paths:
         log('midx: scanning %s\n' % path)
         if opt.force:
-            do_midx(path, opt.output, glob.glob('%s/*.idx' % path))
+            idxs = glob.glob('%s/*.idx' % path)
+            for sublist in _group(idxs, opt.max_files):
+                do_midx(path, opt.output, sublist)
         elif opt.auto:
             m = git.PackIdxList(path)
             needed = {}