Reduce default max objects per pack to 200,000 to save memory.

After some testing, it seems each object sha1 we need to cache while writing
a pack costs us about 83 bytes of memory.  (This isn't so great, so
optimizing it in C later could cut this down a lot.)  The new limit of 200k
objects takes about 16.6 megs of RAM, which nowadays is pretty acceptable.
It also corresponds to roughly 1GB of packfile for my random select of
sample data, so (since the default packfile limit is about 1GB anyway), this
*mostly* won't matter.

It will have an effect if your data is highly compressible, however; an
8192-byte object could compress down to a very small size and you'd end up
with a large number of objects.  The previous default limit of 10 million
objects was ridiculous, since that would take 830 megs of RAM.

diff --git a/git.py b/git.py
index f79fa3dfe931ddcb8309b9ccf2c82a7a1c3aa49e..288f87553f28d30211b402d0fa8571c9970e196d 100644 (file)
--- a/git.py
+++ b/git.py
@@ -238,6 +238,7 @@ class PackWriter:
         f = self.file
         if not f: return None
         self.file = None
+        self.objcache = None
 
         # update object count
         f.seek(8)
@@ -255,7 +256,6 @@ class PackWriter:
         f.write(sum.digest())
         
         f.close()
-        self.objcache = None
 
         p = subprocess.Popen(['git', 'index-pack', '-v',
                               '--index-version=2',

diff --git a/hashsplit.py b/hashsplit.py
index 278abfc05c05c5d79eb29e77f4be8814c6018d5d..7dee3d6ae91586cf3912e49a35b3a737034a1565 100644 (file)
--- a/hashsplit.py
+++ b/hashsplit.py
@@ -6,8 +6,8 @@ BLOB_LWM = 8192*2
 BLOB_MAX = BLOB_LWM*2
 BLOB_HWM = 1024*1024
 split_verbosely = 0
-max_pack_size = 1000*1000*1000
-max_pack_objects = 10*1000*1000
+max_pack_size = 1000*1000*1000  # larger packs will slow down pruning
+max_pack_objects = 200*1000  # cache memory usage is about 83 bytes per object
 fanout = 4096
 
 class Buf: