Gabriel Filion [Fri, 28 Jan 2011 06:14:08 +0000 (01:14 -0500)]
t/test.sh: Fix a test for 'split' on solaris
When looking at output from a test run on Solaris, one test in the
'split' suite showed up as OK but was actually showing a diff
invocation error.
The -q argument (for quiet) does not exist on the version of diff that
is installed on Solaris. Since wvtest intercepts output from tested
commands, the -q argument is actually not needed. Remove the argument in
order to make the test execute correctly under all operating systems
that were tested thus far.
Brandon Low [Mon, 24 Jan 2011 03:31:51 +0000 (19:31 -0800)]
Combine and speed up idx->midx and bupindex merge
These two processes used almost identical algorithms, but were
implemented separately. The main difference was one was ascending and
the other was descending.
This patch reverses the cmp on index.Entry so that both can share an
algorithm.
It also cuts some overhead in the algorithm by using it.next() instead of
the next() wrapper, yielding a ~6% speedup on midx generation and index merging.
Avery Pennarun [Wed, 26 Jan 2011 03:32:27 +0000 (19:32 -0800)]
auto_midx(): report args when failing to call a subprocess.
The exception from subprocess.call() doesn't report the path it tried to use
when it prints the "No such file or directory" error, which isn't helpful
when trying to debug problems.
Gabriel Filion [Mon, 17 Jan 2011 01:38:56 +0000 (20:38 -0500)]
options: remove unused 'exe' parameter
The 'exe' parameter was added in the hope of using it for additional
contextual information in the help text that Options generates. It was
till then abandoned and was judged as superflous information.
Remove the 'exe' parameter from Options' constructor.
Brandon Low [Wed, 12 Jan 2011 01:15:53 +0000 (17:15 -0800)]
save: handle backup write errors properly
bup-save was catching all IOErrors and treating them as data-read
failures, but some of them could be backup-write errors. Have git.py
and client.py raise distinctive errors when pack write raises IOError.
Gabriel Filion [Mon, 17 Jan 2011 02:19:25 +0000 (21:19 -0500)]
cmd/init: don't spit out a traceback on init error
When an error occurs during repository creation, 'bup init' currently
lets GitError exceptions leak out, printing a backtrace to unsuspecting
users in the process.
Intercept GitError exceptions that come out of git.init_repo() and print
out the message that it contains in a more friendly manner.
Gabriel Filion [Mon, 17 Jan 2011 02:19:24 +0000 (21:19 -0500)]
git.py: error when repo's parent dir absent
Currently, when you try to initialize a bup repository inside an
unexistant directory (e.g. BUP_DIR=some_dir/bup_repo, and some_dir does
not exist), bup has to call "git init" to then obtain an error code
which is not very significant to users.
Add a check for the existence of the repository's parent directory and
throw an exception with a more meaningful error message when that
happens.
Gabriel Filion [Sun, 9 Jan 2011 02:19:37 +0000 (21:19 -0500)]
Add some notes on how to install bup on FreeBSD
I've given bup a go on FreeBSD 8.1 and everything seemed to be
functional.
Some package names are not really obvious, and the default 'make'
command doesn't like bup's GNU Make-ish Makefile. Add some notes in the
README so that people can have some pointers on what to do to get bup
fully functional under FreeBSD.
Gabriel Filion [Sun, 9 Jan 2011 02:59:48 +0000 (21:59 -0500)]
Update ls man page for new -a option
Commit 74d28e77366dba1eefbfa2beeda34bcaa835dc58, while modifying the
default behaviour for 'bup ls', introduced a new option to obtain the
old default behaviour.
Brandon Low [Sat, 8 Jan 2011 19:56:00 +0000 (11:56 -0800)]
client/server:Handle multiple suggestions and misc
There was a fixme in the code, I was doing cleanups and fixed it. There
are therefor some misc. cleanups in here along with the handling of
multiple suggested packs.
Avery Pennarun [Sat, 8 Jan 2011 09:59:55 +0000 (01:59 -0800)]
Merge branch 'next' into 'master'
* 'next':
Change server_mode=='dumb' to just a dumb_server_mode bool.
Teach bup about URLs and non-ssh remotes
Don't generate midx files in dumb server mode
Add optional dumb-server mode
git.CatPipe: set a buffer size on the subprocess to increase performance.
Improve test pass condition
Adds examples for strip, strip-prefix and graft to bup save's documentation
Adds --graft option to bup save.
Zoran Zaric [Fri, 7 Jan 2011 11:16:24 +0000 (12:16 +0100)]
Fix a bug in strip_path when prefix is a symlink
helpers.realpath() wasn't the right choice for path normalization.
The prefix itself can be a symlink, too. Now we use os.path.realpath(),
which also follows symlinks for the last element.
Avery Pennarun [Thu, 6 Jan 2011 23:27:04 +0000 (15:27 -0800)]
Merge branch 'bl/dumbserver' into next
* bl/dumbserver:
Change server_mode=='dumb' to just a dumb_server_mode bool.
Teach bup about URLs and non-ssh remotes
Don't generate midx files in dumb server mode
Add optional dumb-server mode
Brandon Low [Wed, 5 Jan 2011 17:28:23 +0000 (09:28 -0800)]
Add optional dumb-server mode
In dumb server mode, the server tells the client to load all .idx files
up front. Puts the burden of deciding what .idxs a client should work
from more squarely in the server side. This mode is activated by
putting a bup-dumb-server file in the bup repodir.
Carsten Bormann [Thu, 6 Jan 2011 21:38:10 +0000 (13:38 -0800)]
git.CatPipe: set a buffer size on the subprocess to increase performance.
apenwarr: I modified Carsten's patch slightly, since "line mode" is not
really appropriate. On my system, this patch (or Carsten's) can read 111
megabytes in 1.7 seconds instead of 2.1 seconds, or 65MB/sec instead of 52
MB/sec.
Avery Pennarun [Mon, 3 Jan 2011 04:58:35 +0000 (20:58 -0800)]
Merge branches 'gf/ls', 'gf/tag', 'zz/import-rsnapshot' and 'bl/selfindex'
* gf/ls:
ls-cmd: hide files with a leading dot by default
* gf/tag:
Refuse branch/tag names that start with a dot
tag-cmd: Some fixups
* zz/import-rsnapshot:
Adds a testcase for import-rsnapshot.
Makes import-rsnapshot use save's -f option.
Adds -f option to save to use a given indexfile.
Makefile: handle shell commands (cmd/*-cmd.sh)
Adds documentation for bup-import-rsnapshot
Adds import-rsnapshot command.
Adds documentation for save's strip option.
Adds testcases for --strip and --strip-path.
Adds a strip and strip-path option to bup save.
* bl/selfindex:
Rename receive-objects command to receive-objects-v2.
Write idxs directly rather than using git-index-pack.
Send SHAs from the client to reduce server load
Use chunkyreader() instead of manually reading multiple blocks.
Avery Pennarun [Mon, 3 Jan 2011 04:38:44 +0000 (20:38 -0800)]
Rename receive-objects command to receive-objects-v2.
...since it's incompatible with the old one. That will make it die more
spectacularly when talking to an old-style server, rather than failing in
more confusing ways.
Theoretically we could do fancy things like making our server support both
variants of receive-objects, but hey, bup is a pre-release, it shouldn't be
acquiring backwards compatibility cruft *already* :)
Avery Pennarun [Thu, 23 Dec 2010 02:08:58 +0000 (18:08 -0800)]
Skip over invalid .idx files if we find any.
There's no particular reason to make it fatal; just pretend they're not
there.
Zoran reported a bug where he had (it seems) some zero-length .idx files,
which is weird, but nothing worth aborting a backup over.
Also, fix _mmap_do() to be able to handle mmap'ing a zero-length file
without an error. It's a trivial and somewhat pointless operation, but it
shouldn't throw an exception.
Avery Pennarun [Wed, 22 Dec 2010 18:49:20 +0000 (10:49 -0800)]
cmd/server: find .idx filenames more efficiently when needed.
Rather than mapping *all* the .idx files into memory at once just to look up
a single object, just open/read/close them sequentially. This should
significantly increase the total repo size on a 32-bit system. (Of course,
it's still not very ideal; we really should have some kind of fallback mode
for when our total set of indexes starts getting too big.)
Avery Pennarun [Sat, 4 Dec 2010 14:13:16 +0000 (06:13 -0800)]
cmd/memtest: stop using weird mmap() and /dev/urandom tricks.
I'll just write a C function that can rapidly generate random sha1s. This
should make it more portable, hopefully fixing a problem reported by Michael
Budde on a Linux/SPARC system.
Avery Pennarun [Thu, 2 Dec 2010 00:02:03 +0000 (16:02 -0800)]
cmd/midx: differentiate the log message from the index.py merging.
It's a curse (inherited from git) that .idx files are called "indexes" and
the bupindex is called an "index." Let's change the message in cmd/midx so
at least we'll know which kind of index people are complaining about.
Avery Pennarun [Wed, 1 Dec 2010 10:44:18 +0000 (02:44 -0800)]
midx: auto-remove midx files that refer to missing .idx files.
Normally an .idx file doesn't ever disappear, but it could happen if you run
'git gc' on your repository. Which I thought would be a terrible idea, but
apparently it can actually save a lot of space for some people (although it
takes a pretty long time to run). And when that happens, all your .idx
files move around. So let's be polite when that happens. We'll print a
warning the first time, but then shut up after that since the flawed midx
file will just go away.
Gabriel Filion [Fri, 26 Nov 2010 11:00:35 +0000 (06:00 -0500)]
add a tag command
Currently implemented: list all tags, add a tag on a specific commit or
head, delete a known tag.
Also, make vfs expose a new directory called '/.tag' which contains a
link for each tag to its associated commit directory situated in
'/.commit'. Finally, make tags appear as symlinks on branch directories
on which the tagged commit is visible.
Gabriel Filion [Fri, 26 Nov 2010 11:00:34 +0000 (06:00 -0500)]
Move commit directories in /.commit/??/
Currently, directories in which we can access files of a particular
commit are placed in each branch directory by which it is reachable.
To avoid possible repetitions of commit directories, move the
directories in a new top level hidden directory named /.commit.
This hidden directory is structured as a two level-deep directory
structure, wherein the first level represents the first byte (two
hexadecimal characters) of commit hashes, and the second level
represents the remainder of the hash.
With this movement, branch directories now contain only symlinks to the
commit directories in /.commit/??/
Also, in BranchList (formerly CommitList), the 'latest' commit was
computed on every iteration over a commit. I moved this calculation up
one level so that it is computed only once.
Avery Pennarun [Sat, 13 Nov 2010 05:58:03 +0000 (21:58 -0800)]
t/test.sh: use /bin/pwd instead of just pwd.
$(pwd) seems to sometimes lie, because the shell uses the $PWD environment
variable. If your PWD is a symlink, this can cause the test to fail since
bup figures out the path using a real call to getcwd().
Problem reported by Zenaan Harkness, though he never did acknowledge if this
fixes his problem :(
Requiring a colon seems to be too fascist; it makes people think that you
can't use local repositories anymore, which wasn't true: you could just
refer to them as ":/path/to/repo". But that's just too weird and
non-obvious. It already resulted in a query on the mailing list, the
avoidance of which is why we added this patch in the first place. So let's
take it back out.
I kept some minor clarifications and unit test improvements, however.
Gabriel Filion [Mon, 11 Oct 2010 18:41:26 +0000 (14:41 -0400)]
Add a coding style document.
The document is largely inspired by the one in Scott Chacon's "HACKING"
file [1] in his 'agitmemnon-server' repository on GitHub with some
precision on the docstring style that was adopted for bup.
Gabriel Filion [Mon, 11 Oct 2010 18:30:51 +0000 (14:30 -0400)]
Revert new-style classes
Some classes were changed to "new-style" Python classes in c7a0f06.
Following a discussion on the mailing list about the relevance of such a
change, it was noted that the features that new-style classes bring are
not used in bup, and considering their slightly higher cost in
instantiating them and accessing their attributes, it is decided that we
don't change to using them.
Revert the changed clases back to old-style classes so that all code is
consistent.
Avery Pennarun [Sat, 16 Oct 2010 23:55:16 +0000 (17:55 -0600)]
cmd/save: if file.read() returns an error, don't abort.
Apparently some mis-implemented Linux filesystems (selinuxfs) have regular
files that can be opened for read, but return EINVAL when you try to read
them. We would throw a fatal exception in that case (since we're not
supposed to have read errors ever, and thus that implies something happened
that we didn't think of) but I guess we'd better make this into a non-fatal
error. It still makes the exit code nonzero so you can see that something
didn't work, though.
Avery Pennarun [Mon, 4 Oct 2010 03:41:09 +0000 (20:41 -0700)]
cmd/web: stream large files asynchronously.
We had a nice chunkyreader() loop for writing files, but unfortunately,
Tornado captured the full content of those files before writing them to the
client. Oops.
Change things around so we don't end up buffering some multiple of the
ENTIRE FILE in memory.
We don't know how many bytes we're going to split in total, but we can at
least print the total number of bytes we've seen so far.
Also fix cmd/random to *not* print progress messages by default, since my
test situation is
bup random 100M | bup split -b
and they scribble over each other when they both print progress output. bup
random now gets a '-v' option.
This lets you provide a list of git object ids on stdin instead of the raw
content. bup-split then uses a CatPipe to retrieve the objects from git and
hashsplit them. You could use this as a helper for converting a git repo
that contains a bunch of large files into one that uses bup-style hashsplit
files.
If you provide multiple input files on the command line, sometimes you want
to merge them togther into a single file before re-chunking them (the
default). But sometimes you want all the files to be treated separately for
chunking purposes, ie. when you know that some of the files will never
change so there's never any point in merging it with previous/subsequent
files.
projects / bup.git / log