Rob Browning [Sat, 10 Nov 2012 19:06:33 +0000 (13:06 -0600)]
Change index; include atime; pack time as xstat timespec; use ns in memory.
Update the index format header to 'BUPI\0\0\0\4' (version 4).
Change the index to include atimes, and write all times to bupindex as
xstat timespecs. This is in preparation for indexing all metadata.
After moving all of the times to the index, there should be far fewer
unique instances of the remaining metadata in many/most cases.
Change the index and index-related code to handle all time
values as integer nanoseconds since the epoch, excepting the packed
format, which (as mentioned above) is now an xstat timespec.
After these changes the index-related in-memory and on-disk time value
handling should match that of the metadata code (i.e. metadata.py,
.bupm, etc.).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: change 10e8 to 10**9 for ns per second.]
Rob Browning [Sun, 23 Sep 2012 23:26:48 +0000 (18:26 -0500)]
Cache all password (pwd) and group (grp) database lookups.
Thanks to Jann Horn <jannhorn@googlemail.com> for determining that
even just caching user names and group names could provide a notable
performance improvement in some cases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 18 Aug 2012 20:26:28 +0000 (15:26 -0500)]
Overhaul restore destination handling, and stripping/grafting behavior.
Change restore to respond to source paths like this (assume outdir
corresponds to "." if there no -C argument or to -C outdir):
/foo/what/ever - extract ever to outdir/ever
/foo/what/ever/ - extract ever/* to outdir/*
/foo/what/ever/. - extract ever/. to outdir/. (i.e. outdir == ever).
Also fix handling of top-level commit symlinks. Previously bup would
just restore /foo/latest as a dummy symlink like this:
latest -> ../.commit/SHA
Instead, dereference latest and restore the target instead.
Tighten up stripping/grafting with additional argument checks, and
handle any root collisions by creating a fake root dir (see comments
in save-cmd.py). Bup still doesn't handle other path collisions yet,
i.e. if both /foo/bar and /bar are remapped to /bar.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 5 May 2012 17:45:01 +0000 (12:45 -0500)]
Add preliminary hardlink support for review.
Thanks to Tim Riemenschneider <git@tim-riemenschneider.de> for
pointing out a compatibility problem (with older versions of Python)
in an earlier version of this patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Zoran Zaric [Sat, 26 Jan 2013 18:44:34 +0000 (19:44 +0100)]
Print a more meaningful error message when help can't run man.
Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: elaborated on the error message a bit] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Alexander Barton [Sat, 26 Jan 2013 18:48:47 +0000 (19:48 +0100)]
Explicitly use "du -k" to set block size in t/test.sh.
Commit e3514812 (Explicitly select the "du" block size in t/test.sh)
used the parameter "-B 1024" which isn't portable. Now use "du -k"
to explicitly get output in kilobytes.
At least du(1) on Linux (GNU coreutils), Mac OS X, FreeBSD, NetBSD,
and OpenBSD support this option.
Signed-off-by: Alexander Barton <alex@barton.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 23 Jan 2013 01:19:20 +0000 (19:19 -0600)]
Explicitly select the "du" block size in t/test.sh.
Since du's default output can be affected by several environment
variables (DU_BLOCK_SIZE, BLOCK_SIZE and BLOCKSIZE), be explicit about
what we want via -B 1024.
Thanks to Michael Ekstrand <michael@elehack.net> for the original
report, and help with the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Yung-Chin Oei [Wed, 24 Oct 2012 02:26:35 +0000 (03:26 +0100)]
Remove dependency on linux/ext2_fs.h
Due to a missing header in (some versions of) linux/ext2_fs.h,
configure's AC_CHECK_HEADERS test fails for this header, making it
appear "not found", and causing Linux-attr support to be disabled in bup
for no good reason.
Given that all the flags bup needs are available in linux/fs.h, we can
safely remove the requirement for linux/ext2_fs.h altogether, to work
around the problem.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Jan 2013 16:23:04 +0000 (10:23 -0600)]
Check the arguments passed to ./configure.
For now, don't allow arguments, since ./configure doesn't actually
support any. Previously "./configure --help" would print misleading
information, and any other arguments were silently ignored.
Thanks to Alexander Barton <alex@barton.de> for the report.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 21 Jul 2012 20:26:17 +0000 (15:26 -0500)]
Add a metadata() method to Node, File, Symlink, and Dir VFS objects.
The current implementation just populates the self._metadata field of
all of the objects in a directory (and of directory itself) from the
relevant .bupm file whenever metadata() is called on any one of those
objects.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Reviewed-by: Gabriel Filion <lelutin@gmail.com>
t/test.sh: fix test failure when $top_dir has a dot-name
Fix the admittedly rare case where we're running tests from a tree
that has a dot-name at its base, which only 'bup ls -a' will print.
This problem was originally encountered on a system that keeps mounts
under '/.autofs/'.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 1 Oct 2011 17:24:36 +0000 (12:24 -0500)]
Save metadata during "bup save".
Record metadata in a hidden/mangled file named .bupm in each
directory, so that the metadata for /foo/ is stored in /foo/.bupm,
with the first entry being the metadata for foo/ itself.
Record an empty index file for each special file so that index entries
and .bupm entries correspond correctly.
Rework the strip/graft functions to return both the save-name, and the
underlying filesystem path to the save-name (when there is one) for
each component. There may not be corresponding filesystem paths if
graft options rewrite the path prefix.
For now, record "/" metadata only when there are no strip/graft
options.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Tested-by: Alexander Barton <alex@barton.de>
Rob Browning [Fri, 23 Nov 2012 20:32:39 +0000 (14:32 -0600)]
Since 'latest' is computed by revs[0], it must exist, so don't test it.
Remove the unnecessary "if latest:" conditional from
BranchList._mksubs(). Given the fact that latest used to be computed
via max(revs), and is now computed by revs[0], there's no way it
wouldn't exist.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 23 Nov 2012 20:18:52 +0000 (14:18 -0600)]
Use branch tip, not newest date to choose 'latest' in BranchList._mksubs().
Previously 'latest' was chosen in the VFS by taking max(revs), which
would pick the most recent commit by rev_list date, which only has a
resolution of integer seconds.
That approach could fail if two commits (i.e. two "bup save" runs)
executed close enough together that both commits ended up with the
same rev_list date. In that case, the sort order was determined by
the commit hash, which is effectively random.
Since I couldn't think of a case where you wouldn't want 'latest' to
refer to the most recent commit (save) to the relevant branch,
'latest' is now computed as revs[0], i.e. the first commit returned by
rev_list.
Thanks to Zoran and Gabriel for some helpful sanity checks while
tracking this down.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 7 Aug 2011 17:01:08 +0000 (18:01 +0100)]
Make "bup meta -tvv" output identical to "bup xstat".
Move the bits of xstat-cmd.py that generate the detailed metadata
representation to metadata.py and xstat.py to support their use from
both "bup xstat" and "bup meta -tvv".
Add size to detailed metadata description when available.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Gabriel Filion [Tue, 9 Oct 2012 03:52:35 +0000 (23:52 -0400)]
Rectify bup-split documentation for the fanout option.
The --fanout option is no longer the maximum number of objects in a
tree, but an average. The documentation, however was never updated and
this can lead to misunderstandings.
Also add a "bold" delimiter that was forgotten in the command summary in
its documentation page.
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Gabriel Filion [Mon, 1 Oct 2012 07:13:53 +0000 (03:13 -0400)]
Documentation: Protect file extensions from start of line.
In the documentation files, we use file extensions as words to simplify
the text. When compiling man pages from the Markdown files, it is
possible that an extension lands at the beginning of a line.
In such a case, the extension is mistakenly identified as a Groff macro.
It seems as though Groff simply ignores it since it is not a known
macro, but emits a warning about the syntax.
This was caught thanks to the debian package's lintian output at:
Since we're putting highlighting on file extensions, we should add it to
all cases, even though there's not risk of it landing at the beginning
of a line. This way, the documentation looks better standardized.
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
git.py: avoid repeated string-copying in tree_decode()
git.tree_decode showed bad perfomance when dealing with large trees,
because it required string-copying quadratically in the number of tree
elements. By removing unnecessary copying, performance is improved at
all tree sizes, and significantly so for larger trees.
The problem became particularly apparent in combination with another bug
in bup (patch for which forthcoming), that allowed trees to grow without
bound when backing up sparse files.
Calling bup-server with "-r host:" fails to activate dumb mode.
When no remote directory is supplied to the -r option, no set-dir
command is sent to the server. This has the weird side effect that the
server actually does not check whether it needs to be in "smart" or
"dumb" mode.
By forcing all commands to make that verification, we'll ensure that the
server mode is correct.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 24 Sep 2012 00:08:38 +0000 (19:08 -0500)]
Change userfullname() default back to "user%d" for the moment.
This was changed in the previous GECOS patch, but since username() has
the "user%d" fallback too, I think we should change both, or neither,
and should probably discuss it a bit more first.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Michael Witten [Mon, 18 Jun 2012 06:06:36 +0000 (06:06 +0000)]
Fallbacks for missing GECOS data; this solves a test issue.
See the thread starting here:
Subject: tests fail while trying to compile & install on arch linux
Date: Sun, 20 May 2012 23:08:20 +0300
From: Alper Kanat <tunix@raptiye.org>
Message-ID: <CAPMuxnSmeHLcP=9M7uYPn6LQeCGNfZOF9DgCFQBHCzwR_DQ0Cg@mail.gmail.com>
When an entry for the current user has been successfully retrieved from the
Unix password database by the function:
bup.helpers.usersfullname
then solely the GECOS field has been used to formulate an author's and
committer's full name for constructing a git commit object. Unfortunately,
this field may well be empty for a great many users; for such a user, the
result has been a full name that is the empty string.
This had not been a problem until the following commit was made in the course
of the development of `git' itself:
commit 4b340cfab9c7a18e39bc531d6a6ffaffdf95f62d
Author: Junio C Hamano <gitster@pobox.com>
AuthorDate: Sun Mar 11 01:25:43 2012 -0800
Commit: Junio C Hamano <gitster@pobox.com>
CommitDate: Sun Mar 11 03:56:50 2012 -0700
ident.c: add split_ident_line() to parse formatted ident line
The commit formatting logic format_person_part() in pretty.c
implements the logic to split an author/committer ident line into
its parts, intermixed with logic to compute its output using these
piece it computes.
Separate the former out to a helper function split_ident_line() so
that other codepath can use the same logic, and rewrite the function
using the helper function.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The new `split_ident_line()' function added by that commit is written
under the stricter assumption that the user's name is *not* the empty
string; clearly, this assumption is broken by what `bup' has been doing
when the user's GECOS field is empty.
Consequently, the easiest solution (as far as `bup' development is
concerned) is to make sure that the full name is never the empty
string. This is achieved by using fallbacks when the GECOS field
yields the empty string:
0. The user's login name is tried.
1. The string "user <uid>" is composed, where `<uid>' is the
current process's user identifier.
Essentially, this seems to solve the problem, as it allows tests to
pass on my system. However, both:
bup.helpers.username
bup.helpers.usersfullname
still rely on Unix-specific functionality, which is not really acceptable.
Signed-off-by: Michael Witten <mfwitten@gmail.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Tested-by: Zoran Zaric <zz@zoranzaric.de>
Gabriel Filion [Sat, 28 Jul 2012 03:02:10 +0000 (23:02 -0400)]
Add BUP_DIR to the subprocess environment during set-dir on the server.
In the normal flow of a remote backup, the client indicates to the
server end where it wants to save data (e.g. where the bup repository
is) by issuing a "set-dir path" command.
This command on the server end records the path in a global variable.
But it doesn't place it in the environment. This causes any subprocess
forked on the server end to be ignorant about the location of the
bup repository and causes errors such as this one:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-midx", line 231, in <module>
git.check_repo_or_die()
File "/usr/lib/bup/bup/git.py", line 851, in check_repo_or_die
init_repo()
File "/usr/lib/bup/bup/git.py", line 828, in init_repo
_git_wait('git init', p)
File "/usr/lib/bup/bup/git.py", line 887, in _git_wait
raise GitError('%s returned %d' % (cmd, rv))
bup.git.GitError: git init returned 1
['/usr/bin/bup', 'midx', '--auto', '--dir',
'/backup/test/objects/pack']: returned 1
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
the subcmd_env variable is never used in main.py. However, when I
removed that part, the -d option stopped working and bup used ~/.bup
instead: so it _is_ doing what we want it to.
The reason why it's working is that line 115 is actually not creating a
copy of the dict, but rather simply pointing to the same dict. so the
call to update() actually changes the environment for the main program,
which is actually quite alright (e.g. it supercedes the environment
variable and ensures that the path given to -d is inherited into
subprocesses)
Now the problem is that this code is very not obvious about what it
does. Plus, it's a couple of useless lines that we need to maintain.
Let's just remove any extraneous work and make the addition to the
environment triggered by the -d option as obvious and concise as
possible.
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Greg Troxel [Fri, 16 Dec 2011 14:53:47 +0000 (09:53 -0500)]
Extend README for NetBSD.
Add NetBSD to the list of systems on which bup is known to work. Give
hints for bup usage on NetBSD, including the location of the fuse
bindings and the pkgsrc entry. Caution about incorrect cycle
detection on fuse mounts. Add pkgsrc URLs.
Signed-off-by: Greg Troxel <gdt@lexort.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Aneurin Price [Thu, 12 Jan 2012 16:15:23 +0000 (16:15 +0000)]
cmd/drecurse: correctly pass excluded_paths to recursive_dirlist
The excluded_paths argument was being passed as a positional argument,
but its position actually corresponded to the 'bup_dir' argument, so
'bup drecurse --exclude=/foo/bar /foo' has never worked.
Signed-off-by: Aneurin Price <aneurin.price@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log