Avery Pennarun [Thu, 23 Dec 2010 02:08:58 +0000 (18:08 -0800)]
Skip over invalid .idx files if we find any.
There's no particular reason to make it fatal; just pretend they're not
there.
Zoran reported a bug where he had (it seems) some zero-length .idx files,
which is weird, but nothing worth aborting a backup over.
Also, fix _mmap_do() to be able to handle mmap'ing a zero-length file
without an error. It's a trivial and somewhat pointless operation, but it
shouldn't throw an exception.
Avery Pennarun [Wed, 22 Dec 2010 18:49:20 +0000 (10:49 -0800)]
cmd/server: find .idx filenames more efficiently when needed.
Rather than mapping *all* the .idx files into memory at once just to look up
a single object, just open/read/close them sequentially. This should
significantly increase the total repo size on a 32-bit system. (Of course,
it's still not very ideal; we really should have some kind of fallback mode
for when our total set of indexes starts getting too big.)
Avery Pennarun [Sat, 4 Dec 2010 14:13:16 +0000 (06:13 -0800)]
cmd/memtest: stop using weird mmap() and /dev/urandom tricks.
I'll just write a C function that can rapidly generate random sha1s. This
should make it more portable, hopefully fixing a problem reported by Michael
Budde on a Linux/SPARC system.
Avery Pennarun [Thu, 2 Dec 2010 00:02:03 +0000 (16:02 -0800)]
cmd/midx: differentiate the log message from the index.py merging.
It's a curse (inherited from git) that .idx files are called "indexes" and
the bupindex is called an "index." Let's change the message in cmd/midx so
at least we'll know which kind of index people are complaining about.
Avery Pennarun [Wed, 1 Dec 2010 10:44:18 +0000 (02:44 -0800)]
midx: auto-remove midx files that refer to missing .idx files.
Normally an .idx file doesn't ever disappear, but it could happen if you run
'git gc' on your repository. Which I thought would be a terrible idea, but
apparently it can actually save a lot of space for some people (although it
takes a pretty long time to run). And when that happens, all your .idx
files move around. So let's be polite when that happens. We'll print a
warning the first time, but then shut up after that since the flawed midx
file will just go away.
Gabriel Filion [Fri, 26 Nov 2010 11:00:35 +0000 (06:00 -0500)]
add a tag command
Currently implemented: list all tags, add a tag on a specific commit or
head, delete a known tag.
Also, make vfs expose a new directory called '/.tag' which contains a
link for each tag to its associated commit directory situated in
'/.commit'. Finally, make tags appear as symlinks on branch directories
on which the tagged commit is visible.
Gabriel Filion [Fri, 26 Nov 2010 11:00:34 +0000 (06:00 -0500)]
Move commit directories in /.commit/??/
Currently, directories in which we can access files of a particular
commit are placed in each branch directory by which it is reachable.
To avoid possible repetitions of commit directories, move the
directories in a new top level hidden directory named /.commit.
This hidden directory is structured as a two level-deep directory
structure, wherein the first level represents the first byte (two
hexadecimal characters) of commit hashes, and the second level
represents the remainder of the hash.
With this movement, branch directories now contain only symlinks to the
commit directories in /.commit/??/
Also, in BranchList (formerly CommitList), the 'latest' commit was
computed on every iteration over a commit. I moved this calculation up
one level so that it is computed only once.
Avery Pennarun [Sat, 13 Nov 2010 05:58:03 +0000 (21:58 -0800)]
t/test.sh: use /bin/pwd instead of just pwd.
$(pwd) seems to sometimes lie, because the shell uses the $PWD environment
variable. If your PWD is a symlink, this can cause the test to fail since
bup figures out the path using a real call to getcwd().
Problem reported by Zenaan Harkness, though he never did acknowledge if this
fixes his problem :(
Requiring a colon seems to be too fascist; it makes people think that you
can't use local repositories anymore, which wasn't true: you could just
refer to them as ":/path/to/repo". But that's just too weird and
non-obvious. It already resulted in a query on the mailing list, the
avoidance of which is why we added this patch in the first place. So let's
take it back out.
I kept some minor clarifications and unit test improvements, however.
Gabriel Filion [Mon, 11 Oct 2010 18:41:26 +0000 (14:41 -0400)]
Add a coding style document.
The document is largely inspired by the one in Scott Chacon's "HACKING"
file [1] in his 'agitmemnon-server' repository on GitHub with some
precision on the docstring style that was adopted for bup.
Gabriel Filion [Mon, 11 Oct 2010 18:30:51 +0000 (14:30 -0400)]
Revert new-style classes
Some classes were changed to "new-style" Python classes in c7a0f06.
Following a discussion on the mailing list about the relevance of such a
change, it was noted that the features that new-style classes bring are
not used in bup, and considering their slightly higher cost in
instantiating them and accessing their attributes, it is decided that we
don't change to using them.
Revert the changed clases back to old-style classes so that all code is
consistent.
Avery Pennarun [Sat, 16 Oct 2010 23:55:16 +0000 (17:55 -0600)]
cmd/save: if file.read() returns an error, don't abort.
Apparently some mis-implemented Linux filesystems (selinuxfs) have regular
files that can be opened for read, but return EINVAL when you try to read
them. We would throw a fatal exception in that case (since we're not
supposed to have read errors ever, and thus that implies something happened
that we didn't think of) but I guess we'd better make this into a non-fatal
error. It still makes the exit code nonzero so you can see that something
didn't work, though.
Avery Pennarun [Mon, 4 Oct 2010 03:41:09 +0000 (20:41 -0700)]
cmd/web: stream large files asynchronously.
We had a nice chunkyreader() loop for writing files, but unfortunately,
Tornado captured the full content of those files before writing them to the
client. Oops.
Change things around so we don't end up buffering some multiple of the
ENTIRE FILE in memory.
We don't know how many bytes we're going to split in total, but we can at
least print the total number of bytes we've seen so far.
Also fix cmd/random to *not* print progress messages by default, since my
test situation is
bup random 100M | bup split -b
and they scribble over each other when they both print progress output. bup
random now gets a '-v' option.
This lets you provide a list of git object ids on stdin instead of the raw
content. bup-split then uses a CatPipe to retrieve the objects from git and
hashsplit them. You could use this as a helper for converting a git repo
that contains a bunch of large files into one that uses bup-style hashsplit
files.
If you provide multiple input files on the command line, sometimes you want
to merge them togther into a single file before re-chunking them (the
default). But sometimes you want all the files to be treated separately for
chunking purposes, ie. when you know that some of the files will never
change so there's never any point in merging it with previous/subsequent
files.
David Roda [Tue, 14 Sep 2010 01:09:16 +0000 (21:09 -0400)]
Add simple styling to bup web.
This adds a wrapper set to 960px to the bup web layout. It also
sets widths on the table and columns.
I added a wrapper div to list-directory.html but I think that must
have snuck in with a previous commit. I am not sure how to fix that
so I will leave it for now. Sorry!
David Roda [Wed, 22 Sep 2010 01:50:27 +0000 (18:50 -0700)]
If we are showing hidden files, continue to do so.
This appends ?hidden=1 to all url's outputted to the template
if we are currently showing hidden files. I added a variable
url_append which is appended to the urls outside of the escaping
David Roda [Tue, 14 Sep 2010 01:09:13 +0000 (21:09 -0400)]
Enable static resources. Move css to external file.
Add to the settings variable in web-cmd.py to set
/web/static to be servced as static resources. This is for
css, javascript, and images.
Move the current styles from the head to static/css/styles.css.
Remove a few unnecessary styles and change the tab stop
to 4 spaces to match the rest of the code.
David Roda [Wed, 8 Sep 2010 12:58:09 +0000 (08:58 -0400)]
--remote parameter requires a colon
This patch checks for the presence of a colon if the --remote option
is used in bup save, bup split, bup join, and bup init. Even though
specifying *only* a pathname without a hostname: is perfectly valid,
it's confusing to allow users to do so, because if they specify
"-r hostname" it will be treated as a path and thus give them a
confusing error message. Requiring a colon will avoid this.
It adds a few test cases to demonstrate that the code
works properly.
It also wraps the remote connection in a try except to prevent
a traceback if there is an error (so far I have only seen this
happen with an invalid bup dir parameter)
And I added the netbeans project folder to gitignore
cmd/restore: embarrassingly slow implementation of 'bup restore'
Well, that was easy, since vfs.py already existed and is doing most of the
hard work. Only 103 lines including all the log message handling and
whatnot.
Only one catch: the restoring code is definitely not optimized. Among other
things (like the probably-excessive-for-our-restoring-needs layering in
vfs.py), we're still calling into 'git cat-file --stdin' to retrieve our
objects. This involves lots and lots of context switches, plus it can't use
midx files for its lookups. The result is that restoring takes much more
CPU time and memory than it really should. But oh well, we have to start
somewhere.
cmd/save: always print a progress() message after a log() message.
An earlier commit (634df2f8b26a1439f22dc9f6a23d55a006bf0429) made 'bup save'
update the progress line much less frequently. Unfortunately, if you used
-v or -vv, this would mean that there was *no* progress bar for a short time
after every log() message (directory or filename). That made the progress
bar flicker annoyingly.
To fix it, make sure we reset the progress bar timer after every filename we
print with log(). It's subtle, but it makes a very visible difference.
options.py: get the real tty width for word wrapping purposes.
Previously we just assumed it was 70 chars, which was safe enough, but not
as elegant as actually reading the real value and adjusting the word wrap of
the usage string accordingly.
options.py: remove extra newlines in usage string.
If the first line after the "--" was a comment (started with whitespace),
then we'd end up printing a double newline instead of a single one after the
synopsis string.
It would also look weird if we had a multi-line comment; the lines would be
separated by blank lines.
options.py: handle optspecs that include inline square brackets.
We recently made it so if the last thing on an options line was [defval],
then the value in brackets became the default for that option. However, we
inadvertently matched *any* bracketed value on that line, not just the one
at the end of the line, which basically prevents us from using square
brackets anywhere on the line. That's no fun.
options.py: better support for explicit no-* options.
If a declared option name starts with no-xxx, then the 'xxx' option starts
off set to True by default, so that no-xxx is False by default, so that
passing --no-xxx as an option will have the desired effect of setting
--no-xxx=True (and thus --xxx=False).
Previously, trying to list a --no-xxx option in the argument list would
trigger an assertion failure.
client.py,git.py: run 'bup midx -a' automatically sometimes.
Now that 'bup midx -a' is smarter, we should run it automatically after
creating a new index file. This should remove the need for running it by
hand.
Thus, we also remove 'bup midx' from the lists of commonly-used subcommands.
(While we're here, let's take out 'split' and 'join' too; you should be
using 'index' and 'save' most of the time.)
They were generated by catting bunches of bup source code together, which,
as it turns out, makes 'git grep' super annoying. Let's rot13 them so
grepping doesn't do anything interesting but the other characteristics are
the same.
cmd/midx: --auto mode can combine existing midx files now.
Previously, --auto would *only* create a midx from not-already-midxed .idx
files. This wasn't optimal since you'd eventually end up with a tonne of
.midx files, which is just as bad as a tonne of .idx files.
Now we'll try to maintain a maximum number of midx files using a
highwater/lowwater mark. That means the number of active midx files should
now stay between 2 and 5, and you can run 'bup midx -a' as often as you
want.
'bup midx -f' will still make sure everything is in a single .midx file,
which is an efficient thing to run every now and then.
'bup midx -af' is the same, but uses existing midx files rather than forcing
bup to start from only .idx files. Theoretically this should always be
faster than, and never be worse than, 'bup midx -f'.
Bonus: 'bup midx -a' now works when there's a limited number of file
descriptors. The previous fix only worked properly with 'bup midx -f'.
(This was rarely a problem since 'bup midx -a' would only ever touch the
last few .idx files, so it didn't need many file descriptors.)
cmd/midx: use getrlimit() to find the max open files.
It turns out the default file limit on MacOS is 256, which is less than our
default of 500. I guess this means trouble after all, so let's auto-detect
it.
* maint:
index.py: handle uid/gid == -1 on cygwin
cmd/memtest: use getrusage() instead of /proc/self/stat.
cmd/index: catch exception for paths that don't exist.
Don't use $(wildcard) during 'make install'.
Don't forget to install _helpers.dll on cygwin.
On cygwin, the uid or gid might be -1 for some reason. struct.pack()
complains about a DeprecationWarning when packing a negative number into an
unsigned int, so fix it up first.
cmd/memtest: use getrusage() instead of /proc/self/stat.
Only Linux has /proc/self/stat, so 'bup memtest' didn't work on anything
except Linux. Unfortunately, getrusage() on *Linux* doesn't have a valid
RSS field (sigh), so we have to use /proc/self/stat as a fallback if it's
zero.
Now memtest works on MacOS as well, which means 'make test' passes again.
(It stopped passing because 'bup memtest' recently got added to one of the
tests.)
All our man pages end up in section 1 of man anyway, and it looks like that
will probably never change. So let's make our filenames simpler and easier
to understand.
Even if we do end up adding a page in (say) section 5 someday, it's no big
deal; we can just add an exception to the Makefile for it or something.
It seems the $(wildcard) is evaluated once at make's startup, so any changes
made *during* build don't get noticed.
That means 'make install' would fail if you ran it without first running
'make all', because $(wildcard cmd/bup-*) wouldn't match anything at startup
time; the files we were copying only got created during the build.
We were installing *.so, but not *$(SOEXT) like we should have. Now we do,
which should fix some cygwin install problems reported by David Roda.
Also, when installing *.so and *.dll files, make them 0755 instead of 0644,
also to prevent permissions problems on cygwin, also reported by David Roda.
* guesser:
_helpers.extract_bits(): rewrite git.extract_bits() in C.
_helpers.firstword(): a new function to extract the first 32 bits.
git.py: when seeking inside a midx, use statistical guessing.
git.py: recover more elegantly if a MIDX file has the wrong version.
Previously we'd throw an assertion for any too-new-format MIDX file, which
isn't so good. Let's recover more politely (and just ignore the file in
question) if that happens.
Noticed by Zoran Zaric who was testing my midx3 branch.
Zoran reported that 'bup midx -f' on his system tried to open 3000 files at
a time and wouldn't work. That's no good, so let's limit the maximum files
to open; the default is 500 for now, since that ought to be usable for
normal people. Arguably we could use getrlimit() or something to find out
the actual maximum, or just keep opening stuff until we get an error, but
maybe there's no point.
Unfortunately this patch isn't really perfect, because it limits the
usefulness of midx files. If you could merge midx files into other midx
files, then you could at least group them all together after multiple runs,
but that's not currently supported.
Avery Pennarun [Fri, 27 Aug 2010 03:19:49 +0000 (20:19 -0700)]
_helpers.extract_bits(): rewrite git.extract_bits() in C.
That makes our memtest run just slightly faster: 2.8 seconds instead of 3.0
seconds, which catches us back up with the pre-interpolation-search code.
Thus we should now be able to release this patch without feeling embarrassed
:)
Avery Pennarun [Fri, 27 Aug 2010 02:31:24 +0000 (19:31 -0700)]
git.py: when seeking inside a midx, use statistical guessing.
Instead of using a pure binary search (where we seek to the middle of the
area and do a greater/lesser comparison) we now use an "interpolation
search" (http://en.wikipedia.org/wiki/Interpolation_search), which means we
seek to where we statistically *expect* the desired value to be.
In my test data, this reduces the number of typical search steps in my test
midx from 8.7 steps/object to 4.8 steps/object.
This reduces memory churn when using a midx, since sometimes a given search
region spans two pages, and this technique allows us to more quickly
eliminate one of the two pages sometimes, allowing us to dirty one fewer
page.
Unfortunately the implementation requires some futzing, so this actually
makes memtest run about 35% *slower*. Will try to fix that next.
The original link to this algorithm came from this article:
http://sna-projects.com/blog/2010/06/beating-binary-search/
Avery Pennarun [Fri, 27 Aug 2010 02:16:34 +0000 (19:16 -0700)]
cmd/memtest: add a --existing option to test with existing objects.
This is useful for testing behaviour when we're looking for objects
that *do* exist. Of course, it just goes through the objects in order, so
it's not actually that realistic.
Avery Pennarun [Thu, 26 Aug 2010 04:06:46 +0000 (21:06 -0700)]
cmd/midx: fix SHA_PER_PAGE calculation.
For some reason we were dividing by 200 instead of by 20, which was way off.
Switch to 20 instead. Suspiciously, this makes memory usage slightly worse
in my current (smallish) set of test data, so we might need to revert it
later...? But if we're going to have an adjustment, we should at least make
it clear what for, rather than hiding it in something that looks
suspiciously like a typo.
Avery Pennarun [Thu, 26 Aug 2010 03:40:34 +0000 (20:40 -0700)]
cmd/margin: add a new --predict option.
When --predict is given, it tries to guess the offset in the indexfile of
each hash, based on assumption that the hashes are distributed evenly
throughout the file. Then it prints the maximum amount by which this guess
deviates from reality.
I was hoping the results would show that the maximum deviation in a typical
midx was less than a page's worth of hashes; that would mean the toplevel
lookup table could be redundant, which means fewer pages hit in the
common case. No such luck, unfortunately; with 1.6 million objects, my
maximum deviation was 913 hashes (about 18 kbytes, or 5 pages).
By comparison, midx files should hit about 2 pages in the common case (1
lookup table + 1 data page). Or 3 pages if we're unlucky and the search
spans two data pages.
Avery Pennarun [Mon, 23 Aug 2010 03:27:03 +0000 (20:27 -0700)]
Rename _faster.so to _helpers.so.
Okay, _faster.so wasn't a good choice of names. Partly because not
everything in there is just to make stuff faster, and partly because some
*proposed* changes to it don't just make stuff faster. So let's rename it
one more time. Hopefully the last time for a while!
Gabriel Filion [Mon, 16 Aug 2010 01:29:29 +0000 (21:29 -0400)]
import cleanup
Remove unused imported modules.
I started using the pyflakes.vim plugin and it automagically shows a
bunch of problems/uncleanliness in the code. It helped me pull this out
in 15mins.
This change shouldn't have any impact on performance or functionality
but it makes the code cleaner.
Avery Pennarun [Sun, 22 Aug 2010 06:44:49 +0000 (23:44 -0700)]
cmd/ftp: don't die if we can't import the ctypes module.
It's only needed on some rare broken versions of readline anyway. If we
can't find the module, chances are the system doesn't have that broken
version of readline.
Based on suggestions by Gabriel Filion and Aaron Ucko.
Gabriel Filion [Fri, 20 Aug 2010 06:24:57 +0000 (02:24 -0400)]
lib/bup/vfs: bring back Python 2.4 support
There is currently one test failure when running tests against Python
2.4: a try..except..finally block that's interpreted as a syntax error.
The commit introducing this incompatibility with 2.4 is f77a0829
This is a well known python 2.4 limitation and the workaround, although
ugly, is easy.
With this test passing, Python 2.4 support is back.
Gabriel Filion [Mon, 2 Aug 2010 06:20:06 +0000 (02:20 -0400)]
lib/bup/vfs: Add docstrings
Since the vfs module uses the function git._treeparse, it should not be
named as if it was a private function. Rename git._treeparse to
git.treeparse and document it (add a docstring to it).
Also, transform _ChunkReader, _FileReader and Node into new-style
classes.
Finally, remove trailing spaces from lib/bup/vfs.py .
If you ran 'bup save' in an ssh sessio, you could end up sending huge
amounts of data back over ssh *just* to update the progress meter after
every single block! Oops. Limit the updates to only about 5 per second,
which is much better.
Rename _hashsplit.so to _faster.so, and move bupsplit into its own source file.
A lot of stuff in _hashsplit.c wasn't actually about hashsplitting; it was
just a catch-all for all our C accelerator functions. Now the module name
reflects that.
Also move the bupsplit functions into their own non-python-dependent C
source file so they can be used as part of other projects.
_hashsplit.c: replace the stupidsum algorithm with rsync's adler32-based one.
I've been meaning to do this for a while, but a particular test dataset that
really caused problems with stupidsum() (ie. it split things into way more
chunks than it should have) finally screwed me over. Let's change over to a
"real" checksum algorithm.
Non-annoying datasets shouldn't be noticeably affected, but bad ones (such
as my test case from EQL Data) can be 10x more sensible. Typical backup
sets now have about 20% fewer chunks, although this has little affect on the
overall repository size.
WARNING: After this patch, all your chunk boundaries will be different from
before! That means your incremental backups won't be terribly incremental
and your backup repositories will jump in size. This should only happen
once.
_hashsplit.c: switch rollsum_roll() to a macro instead of an inline function.
gcc 4.3's optimizer manages to fail at optimizing the inline, but works okay
with the macro.
Mysteriously, if find_ofs() is *not* static (and therefore presumably
*harder* to optimize), the optimizer works either way. But removing the
static is just wrong, so use the macro instead.
The difference in speed is about 53 megs/sec vs 80 megs/sec on my machine
for this command:
bup random 100M 2>/dev/null | bup split -N --bench
projects / bup.git / log