Gabriel Filion [Sun, 9 Jan 2011 02:19:37 +0000 (21:19 -0500)]
Add some notes on how to install bup on FreeBSD
I've given bup a go on FreeBSD 8.1 and everything seemed to be
functional.
Some package names are not really obvious, and the default 'make'
command doesn't like bup's GNU Make-ish Makefile. Add some notes in the
README so that people can have some pointers on what to do to get bup
fully functional under FreeBSD.
Gabriel Filion [Sun, 9 Jan 2011 02:59:48 +0000 (21:59 -0500)]
Update ls man page for new -a option
Commit 74d28e77366dba1eefbfa2beeda34bcaa835dc58, while modifying the
default behaviour for 'bup ls', introduced a new option to obtain the
old default behaviour.
Brandon Low [Sat, 8 Jan 2011 19:56:00 +0000 (11:56 -0800)]
client/server:Handle multiple suggestions and misc
There was a fixme in the code, I was doing cleanups and fixed it. There
are therefor some misc. cleanups in here along with the handling of
multiple suggested packs.
Avery Pennarun [Sat, 8 Jan 2011 09:59:55 +0000 (01:59 -0800)]
Merge branch 'next' into 'master'
* 'next':
Change server_mode=='dumb' to just a dumb_server_mode bool.
Teach bup about URLs and non-ssh remotes
Don't generate midx files in dumb server mode
Add optional dumb-server mode
git.CatPipe: set a buffer size on the subprocess to increase performance.
Improve test pass condition
Adds examples for strip, strip-prefix and graft to bup save's documentation
Adds --graft option to bup save.
Zoran Zaric [Fri, 7 Jan 2011 11:16:24 +0000 (12:16 +0100)]
Fix a bug in strip_path when prefix is a symlink
helpers.realpath() wasn't the right choice for path normalization.
The prefix itself can be a symlink, too. Now we use os.path.realpath(),
which also follows symlinks for the last element.
Avery Pennarun [Thu, 6 Jan 2011 23:27:04 +0000 (15:27 -0800)]
Merge branch 'bl/dumbserver' into next
* bl/dumbserver:
Change server_mode=='dumb' to just a dumb_server_mode bool.
Teach bup about URLs and non-ssh remotes
Don't generate midx files in dumb server mode
Add optional dumb-server mode
Brandon Low [Wed, 5 Jan 2011 17:28:23 +0000 (09:28 -0800)]
Add optional dumb-server mode
In dumb server mode, the server tells the client to load all .idx files
up front. Puts the burden of deciding what .idxs a client should work
from more squarely in the server side. This mode is activated by
putting a bup-dumb-server file in the bup repodir.
Carsten Bormann [Thu, 6 Jan 2011 21:38:10 +0000 (13:38 -0800)]
git.CatPipe: set a buffer size on the subprocess to increase performance.
apenwarr: I modified Carsten's patch slightly, since "line mode" is not
really appropriate. On my system, this patch (or Carsten's) can read 111
megabytes in 1.7 seconds instead of 2.1 seconds, or 65MB/sec instead of 52
MB/sec.
Avery Pennarun [Mon, 3 Jan 2011 04:58:35 +0000 (20:58 -0800)]
Merge branches 'gf/ls', 'gf/tag', 'zz/import-rsnapshot' and 'bl/selfindex'
* gf/ls:
ls-cmd: hide files with a leading dot by default
* gf/tag:
Refuse branch/tag names that start with a dot
tag-cmd: Some fixups
* zz/import-rsnapshot:
Adds a testcase for import-rsnapshot.
Makes import-rsnapshot use save's -f option.
Adds -f option to save to use a given indexfile.
Makefile: handle shell commands (cmd/*-cmd.sh)
Adds documentation for bup-import-rsnapshot
Adds import-rsnapshot command.
Adds documentation for save's strip option.
Adds testcases for --strip and --strip-path.
Adds a strip and strip-path option to bup save.
* bl/selfindex:
Rename receive-objects command to receive-objects-v2.
Write idxs directly rather than using git-index-pack.
Send SHAs from the client to reduce server load
Use chunkyreader() instead of manually reading multiple blocks.
Avery Pennarun [Mon, 3 Jan 2011 04:38:44 +0000 (20:38 -0800)]
Rename receive-objects command to receive-objects-v2.
...since it's incompatible with the old one. That will make it die more
spectacularly when talking to an old-style server, rather than failing in
more confusing ways.
Theoretically we could do fancy things like making our server support both
variants of receive-objects, but hey, bup is a pre-release, it shouldn't be
acquiring backwards compatibility cruft *already* :)
Avery Pennarun [Thu, 23 Dec 2010 02:08:58 +0000 (18:08 -0800)]
Skip over invalid .idx files if we find any.
There's no particular reason to make it fatal; just pretend they're not
there.
Zoran reported a bug where he had (it seems) some zero-length .idx files,
which is weird, but nothing worth aborting a backup over.
Also, fix _mmap_do() to be able to handle mmap'ing a zero-length file
without an error. It's a trivial and somewhat pointless operation, but it
shouldn't throw an exception.
Avery Pennarun [Wed, 22 Dec 2010 18:49:20 +0000 (10:49 -0800)]
cmd/server: find .idx filenames more efficiently when needed.
Rather than mapping *all* the .idx files into memory at once just to look up
a single object, just open/read/close them sequentially. This should
significantly increase the total repo size on a 32-bit system. (Of course,
it's still not very ideal; we really should have some kind of fallback mode
for when our total set of indexes starts getting too big.)
Avery Pennarun [Sat, 4 Dec 2010 14:13:16 +0000 (06:13 -0800)]
cmd/memtest: stop using weird mmap() and /dev/urandom tricks.
I'll just write a C function that can rapidly generate random sha1s. This
should make it more portable, hopefully fixing a problem reported by Michael
Budde on a Linux/SPARC system.
Avery Pennarun [Thu, 2 Dec 2010 00:02:03 +0000 (16:02 -0800)]
cmd/midx: differentiate the log message from the index.py merging.
It's a curse (inherited from git) that .idx files are called "indexes" and
the bupindex is called an "index." Let's change the message in cmd/midx so
at least we'll know which kind of index people are complaining about.
Avery Pennarun [Wed, 1 Dec 2010 10:44:18 +0000 (02:44 -0800)]
midx: auto-remove midx files that refer to missing .idx files.
Normally an .idx file doesn't ever disappear, but it could happen if you run
'git gc' on your repository. Which I thought would be a terrible idea, but
apparently it can actually save a lot of space for some people (although it
takes a pretty long time to run). And when that happens, all your .idx
files move around. So let's be polite when that happens. We'll print a
warning the first time, but then shut up after that since the flawed midx
file will just go away.
Gabriel Filion [Fri, 26 Nov 2010 11:00:35 +0000 (06:00 -0500)]
add a tag command
Currently implemented: list all tags, add a tag on a specific commit or
head, delete a known tag.
Also, make vfs expose a new directory called '/.tag' which contains a
link for each tag to its associated commit directory situated in
'/.commit'. Finally, make tags appear as symlinks on branch directories
on which the tagged commit is visible.
Gabriel Filion [Fri, 26 Nov 2010 11:00:34 +0000 (06:00 -0500)]
Move commit directories in /.commit/??/
Currently, directories in which we can access files of a particular
commit are placed in each branch directory by which it is reachable.
To avoid possible repetitions of commit directories, move the
directories in a new top level hidden directory named /.commit.
This hidden directory is structured as a two level-deep directory
structure, wherein the first level represents the first byte (two
hexadecimal characters) of commit hashes, and the second level
represents the remainder of the hash.
With this movement, branch directories now contain only symlinks to the
commit directories in /.commit/??/
Also, in BranchList (formerly CommitList), the 'latest' commit was
computed on every iteration over a commit. I moved this calculation up
one level so that it is computed only once.
Avery Pennarun [Sat, 13 Nov 2010 05:58:03 +0000 (21:58 -0800)]
t/test.sh: use /bin/pwd instead of just pwd.
$(pwd) seems to sometimes lie, because the shell uses the $PWD environment
variable. If your PWD is a symlink, this can cause the test to fail since
bup figures out the path using a real call to getcwd().
Problem reported by Zenaan Harkness, though he never did acknowledge if this
fixes his problem :(
Requiring a colon seems to be too fascist; it makes people think that you
can't use local repositories anymore, which wasn't true: you could just
refer to them as ":/path/to/repo". But that's just too weird and
non-obvious. It already resulted in a query on the mailing list, the
avoidance of which is why we added this patch in the first place. So let's
take it back out.
I kept some minor clarifications and unit test improvements, however.
Gabriel Filion [Mon, 11 Oct 2010 18:41:26 +0000 (14:41 -0400)]
Add a coding style document.
The document is largely inspired by the one in Scott Chacon's "HACKING"
file [1] in his 'agitmemnon-server' repository on GitHub with some
precision on the docstring style that was adopted for bup.
Gabriel Filion [Mon, 11 Oct 2010 18:30:51 +0000 (14:30 -0400)]
Revert new-style classes
Some classes were changed to "new-style" Python classes in c7a0f06.
Following a discussion on the mailing list about the relevance of such a
change, it was noted that the features that new-style classes bring are
not used in bup, and considering their slightly higher cost in
instantiating them and accessing their attributes, it is decided that we
don't change to using them.
Revert the changed clases back to old-style classes so that all code is
consistent.
Avery Pennarun [Sat, 16 Oct 2010 23:55:16 +0000 (17:55 -0600)]
cmd/save: if file.read() returns an error, don't abort.
Apparently some mis-implemented Linux filesystems (selinuxfs) have regular
files that can be opened for read, but return EINVAL when you try to read
them. We would throw a fatal exception in that case (since we're not
supposed to have read errors ever, and thus that implies something happened
that we didn't think of) but I guess we'd better make this into a non-fatal
error. It still makes the exit code nonzero so you can see that something
didn't work, though.
Avery Pennarun [Mon, 4 Oct 2010 03:41:09 +0000 (20:41 -0700)]
cmd/web: stream large files asynchronously.
We had a nice chunkyreader() loop for writing files, but unfortunately,
Tornado captured the full content of those files before writing them to the
client. Oops.
Change things around so we don't end up buffering some multiple of the
ENTIRE FILE in memory.
We don't know how many bytes we're going to split in total, but we can at
least print the total number of bytes we've seen so far.
Also fix cmd/random to *not* print progress messages by default, since my
test situation is
bup random 100M | bup split -b
and they scribble over each other when they both print progress output. bup
random now gets a '-v' option.
This lets you provide a list of git object ids on stdin instead of the raw
content. bup-split then uses a CatPipe to retrieve the objects from git and
hashsplit them. You could use this as a helper for converting a git repo
that contains a bunch of large files into one that uses bup-style hashsplit
files.
If you provide multiple input files on the command line, sometimes you want
to merge them togther into a single file before re-chunking them (the
default). But sometimes you want all the files to be treated separately for
chunking purposes, ie. when you know that some of the files will never
change so there's never any point in merging it with previous/subsequent
files.
David Roda [Tue, 14 Sep 2010 01:09:16 +0000 (21:09 -0400)]
Add simple styling to bup web.
This adds a wrapper set to 960px to the bup web layout. It also
sets widths on the table and columns.
I added a wrapper div to list-directory.html but I think that must
have snuck in with a previous commit. I am not sure how to fix that
so I will leave it for now. Sorry!
David Roda [Wed, 22 Sep 2010 01:50:27 +0000 (18:50 -0700)]
If we are showing hidden files, continue to do so.
This appends ?hidden=1 to all url's outputted to the template
if we are currently showing hidden files. I added a variable
url_append which is appended to the urls outside of the escaping
David Roda [Tue, 14 Sep 2010 01:09:13 +0000 (21:09 -0400)]
Enable static resources. Move css to external file.
Add to the settings variable in web-cmd.py to set
/web/static to be servced as static resources. This is for
css, javascript, and images.
Move the current styles from the head to static/css/styles.css.
Remove a few unnecessary styles and change the tab stop
to 4 spaces to match the rest of the code.
David Roda [Wed, 8 Sep 2010 12:58:09 +0000 (08:58 -0400)]
--remote parameter requires a colon
This patch checks for the presence of a colon if the --remote option
is used in bup save, bup split, bup join, and bup init. Even though
specifying *only* a pathname without a hostname: is perfectly valid,
it's confusing to allow users to do so, because if they specify
"-r hostname" it will be treated as a path and thus give them a
confusing error message. Requiring a colon will avoid this.
It adds a few test cases to demonstrate that the code
works properly.
It also wraps the remote connection in a try except to prevent
a traceback if there is an error (so far I have only seen this
happen with an invalid bup dir parameter)
And I added the netbeans project folder to gitignore
cmd/restore: embarrassingly slow implementation of 'bup restore'
Well, that was easy, since vfs.py already existed and is doing most of the
hard work. Only 103 lines including all the log message handling and
whatnot.
Only one catch: the restoring code is definitely not optimized. Among other
things (like the probably-excessive-for-our-restoring-needs layering in
vfs.py), we're still calling into 'git cat-file --stdin' to retrieve our
objects. This involves lots and lots of context switches, plus it can't use
midx files for its lookups. The result is that restoring takes much more
CPU time and memory than it really should. But oh well, we have to start
somewhere.
cmd/save: always print a progress() message after a log() message.
An earlier commit (634df2f8b26a1439f22dc9f6a23d55a006bf0429) made 'bup save'
update the progress line much less frequently. Unfortunately, if you used
-v or -vv, this would mean that there was *no* progress bar for a short time
after every log() message (directory or filename). That made the progress
bar flicker annoyingly.
To fix it, make sure we reset the progress bar timer after every filename we
print with log(). It's subtle, but it makes a very visible difference.
options.py: get the real tty width for word wrapping purposes.
Previously we just assumed it was 70 chars, which was safe enough, but not
as elegant as actually reading the real value and adjusting the word wrap of
the usage string accordingly.
options.py: remove extra newlines in usage string.
If the first line after the "--" was a comment (started with whitespace),
then we'd end up printing a double newline instead of a single one after the
synopsis string.
It would also look weird if we had a multi-line comment; the lines would be
separated by blank lines.
options.py: handle optspecs that include inline square brackets.
We recently made it so if the last thing on an options line was [defval],
then the value in brackets became the default for that option. However, we
inadvertently matched *any* bracketed value on that line, not just the one
at the end of the line, which basically prevents us from using square
brackets anywhere on the line. That's no fun.
options.py: better support for explicit no-* options.
If a declared option name starts with no-xxx, then the 'xxx' option starts
off set to True by default, so that no-xxx is False by default, so that
passing --no-xxx as an option will have the desired effect of setting
--no-xxx=True (and thus --xxx=False).
Previously, trying to list a --no-xxx option in the argument list would
trigger an assertion failure.
client.py,git.py: run 'bup midx -a' automatically sometimes.
Now that 'bup midx -a' is smarter, we should run it automatically after
creating a new index file. This should remove the need for running it by
hand.
Thus, we also remove 'bup midx' from the lists of commonly-used subcommands.
(While we're here, let's take out 'split' and 'join' too; you should be
using 'index' and 'save' most of the time.)
They were generated by catting bunches of bup source code together, which,
as it turns out, makes 'git grep' super annoying. Let's rot13 them so
grepping doesn't do anything interesting but the other characteristics are
the same.
cmd/midx: --auto mode can combine existing midx files now.
Previously, --auto would *only* create a midx from not-already-midxed .idx
files. This wasn't optimal since you'd eventually end up with a tonne of
.midx files, which is just as bad as a tonne of .idx files.
Now we'll try to maintain a maximum number of midx files using a
highwater/lowwater mark. That means the number of active midx files should
now stay between 2 and 5, and you can run 'bup midx -a' as often as you
want.
'bup midx -f' will still make sure everything is in a single .midx file,
which is an efficient thing to run every now and then.
'bup midx -af' is the same, but uses existing midx files rather than forcing
bup to start from only .idx files. Theoretically this should always be
faster than, and never be worse than, 'bup midx -f'.
Bonus: 'bup midx -a' now works when there's a limited number of file
descriptors. The previous fix only worked properly with 'bup midx -f'.
(This was rarely a problem since 'bup midx -a' would only ever touch the
last few .idx files, so it didn't need many file descriptors.)
cmd/midx: use getrlimit() to find the max open files.
It turns out the default file limit on MacOS is 256, which is less than our
default of 500. I guess this means trouble after all, so let's auto-detect
it.
* maint:
index.py: handle uid/gid == -1 on cygwin
cmd/memtest: use getrusage() instead of /proc/self/stat.
cmd/index: catch exception for paths that don't exist.
Don't use $(wildcard) during 'make install'.
Don't forget to install _helpers.dll on cygwin.
projects / bup.git / log