t/test.sh: fix test failure when $top_dir has a dot-name
Fix the admittedly rare case where we're running tests from a tree
that has a dot-name at its base, which only 'bup ls -a' will print.
This problem was originally encountered on a system that keeps mounts
under '/.autofs/'.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 1 Oct 2011 17:24:36 +0000 (12:24 -0500)]
Save metadata during "bup save".
Record metadata in a hidden/mangled file named .bupm in each
directory, so that the metadata for /foo/ is stored in /foo/.bupm,
with the first entry being the metadata for foo/ itself.
Record an empty index file for each special file so that index entries
and .bupm entries correspond correctly.
Rework the strip/graft functions to return both the save-name, and the
underlying filesystem path to the save-name (when there is one) for
each component. There may not be corresponding filesystem paths if
graft options rewrite the path prefix.
For now, record "/" metadata only when there are no strip/graft
options.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Tested-by: Alexander Barton <alex@barton.de>
Rob Browning [Fri, 23 Nov 2012 20:32:39 +0000 (14:32 -0600)]
Since 'latest' is computed by revs[0], it must exist, so don't test it.
Remove the unnecessary "if latest:" conditional from
BranchList._mksubs(). Given the fact that latest used to be computed
via max(revs), and is now computed by revs[0], there's no way it
wouldn't exist.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 23 Nov 2012 20:18:52 +0000 (14:18 -0600)]
Use branch tip, not newest date to choose 'latest' in BranchList._mksubs().
Previously 'latest' was chosen in the VFS by taking max(revs), which
would pick the most recent commit by rev_list date, which only has a
resolution of integer seconds.
That approach could fail if two commits (i.e. two "bup save" runs)
executed close enough together that both commits ended up with the
same rev_list date. In that case, the sort order was determined by
the commit hash, which is effectively random.
Since I couldn't think of a case where you wouldn't want 'latest' to
refer to the most recent commit (save) to the relevant branch,
'latest' is now computed as revs[0], i.e. the first commit returned by
rev_list.
Thanks to Zoran and Gabriel for some helpful sanity checks while
tracking this down.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 7 Aug 2011 17:01:08 +0000 (18:01 +0100)]
Make "bup meta -tvv" output identical to "bup xstat".
Move the bits of xstat-cmd.py that generate the detailed metadata
representation to metadata.py and xstat.py to support their use from
both "bup xstat" and "bup meta -tvv".
Add size to detailed metadata description when available.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Gabriel Filion [Tue, 9 Oct 2012 03:52:35 +0000 (23:52 -0400)]
Rectify bup-split documentation for the fanout option.
The --fanout option is no longer the maximum number of objects in a
tree, but an average. The documentation, however was never updated and
this can lead to misunderstandings.
Also add a "bold" delimiter that was forgotten in the command summary in
its documentation page.
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Gabriel Filion [Mon, 1 Oct 2012 07:13:53 +0000 (03:13 -0400)]
Documentation: Protect file extensions from start of line.
In the documentation files, we use file extensions as words to simplify
the text. When compiling man pages from the Markdown files, it is
possible that an extension lands at the beginning of a line.
In such a case, the extension is mistakenly identified as a Groff macro.
It seems as though Groff simply ignores it since it is not a known
macro, but emits a warning about the syntax.
This was caught thanks to the debian package's lintian output at:
Since we're putting highlighting on file extensions, we should add it to
all cases, even though there's not risk of it landing at the beginning
of a line. This way, the documentation looks better standardized.
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
git.py: avoid repeated string-copying in tree_decode()
git.tree_decode showed bad perfomance when dealing with large trees,
because it required string-copying quadratically in the number of tree
elements. By removing unnecessary copying, performance is improved at
all tree sizes, and significantly so for larger trees.
The problem became particularly apparent in combination with another bug
in bup (patch for which forthcoming), that allowed trees to grow without
bound when backing up sparse files.
Calling bup-server with "-r host:" fails to activate dumb mode.
When no remote directory is supplied to the -r option, no set-dir
command is sent to the server. This has the weird side effect that the
server actually does not check whether it needs to be in "smart" or
"dumb" mode.
By forcing all commands to make that verification, we'll ensure that the
server mode is correct.
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 24 Sep 2012 00:08:38 +0000 (19:08 -0500)]
Change userfullname() default back to "user%d" for the moment.
This was changed in the previous GECOS patch, but since username() has
the "user%d" fallback too, I think we should change both, or neither,
and should probably discuss it a bit more first.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Michael Witten [Mon, 18 Jun 2012 06:06:36 +0000 (06:06 +0000)]
Fallbacks for missing GECOS data; this solves a test issue.
See the thread starting here:
Subject: tests fail while trying to compile & install on arch linux
Date: Sun, 20 May 2012 23:08:20 +0300
From: Alper Kanat <tunix@raptiye.org>
Message-ID: <CAPMuxnSmeHLcP=9M7uYPn6LQeCGNfZOF9DgCFQBHCzwR_DQ0Cg@mail.gmail.com>
When an entry for the current user has been successfully retrieved from the
Unix password database by the function:
bup.helpers.usersfullname
then solely the GECOS field has been used to formulate an author's and
committer's full name for constructing a git commit object. Unfortunately,
this field may well be empty for a great many users; for such a user, the
result has been a full name that is the empty string.
This had not been a problem until the following commit was made in the course
of the development of `git' itself:
commit 4b340cfab9c7a18e39bc531d6a6ffaffdf95f62d
Author: Junio C Hamano <gitster@pobox.com>
AuthorDate: Sun Mar 11 01:25:43 2012 -0800
Commit: Junio C Hamano <gitster@pobox.com>
CommitDate: Sun Mar 11 03:56:50 2012 -0700
ident.c: add split_ident_line() to parse formatted ident line
The commit formatting logic format_person_part() in pretty.c
implements the logic to split an author/committer ident line into
its parts, intermixed with logic to compute its output using these
piece it computes.
Separate the former out to a helper function split_ident_line() so
that other codepath can use the same logic, and rewrite the function
using the helper function.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The new `split_ident_line()' function added by that commit is written
under the stricter assumption that the user's name is *not* the empty
string; clearly, this assumption is broken by what `bup' has been doing
when the user's GECOS field is empty.
Consequently, the easiest solution (as far as `bup' development is
concerned) is to make sure that the full name is never the empty
string. This is achieved by using fallbacks when the GECOS field
yields the empty string:
0. The user's login name is tried.
1. The string "user <uid>" is composed, where `<uid>' is the
current process's user identifier.
Essentially, this seems to solve the problem, as it allows tests to
pass on my system. However, both:
bup.helpers.username
bup.helpers.usersfullname
still rely on Unix-specific functionality, which is not really acceptable.
Signed-off-by: Michael Witten <mfwitten@gmail.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Tested-by: Zoran Zaric <zz@zoranzaric.de>
Gabriel Filion [Sat, 28 Jul 2012 03:02:10 +0000 (23:02 -0400)]
Add BUP_DIR to the subprocess environment during set-dir on the server.
In the normal flow of a remote backup, the client indicates to the
server end where it wants to save data (e.g. where the bup repository
is) by issuing a "set-dir path" command.
This command on the server end records the path in a global variable.
But it doesn't place it in the environment. This causes any subprocess
forked on the server end to be ignorant about the location of the
bup repository and causes errors such as this one:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-midx", line 231, in <module>
git.check_repo_or_die()
File "/usr/lib/bup/bup/git.py", line 851, in check_repo_or_die
init_repo()
File "/usr/lib/bup/bup/git.py", line 828, in init_repo
_git_wait('git init', p)
File "/usr/lib/bup/bup/git.py", line 887, in _git_wait
raise GitError('%s returned %d' % (cmd, rv))
bup.git.GitError: git init returned 1
['/usr/bin/bup', 'midx', '--auto', '--dir',
'/backup/test/objects/pack']: returned 1
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
the subcmd_env variable is never used in main.py. However, when I
removed that part, the -d option stopped working and bup used ~/.bup
instead: so it _is_ doing what we want it to.
The reason why it's working is that line 115 is actually not creating a
copy of the dict, but rather simply pointing to the same dict. so the
call to update() actually changes the environment for the main program,
which is actually quite alright (e.g. it supercedes the environment
variable and ensures that the path given to -d is inherited into
subprocesses)
Now the problem is that this code is very not obvious about what it
does. Plus, it's a couple of useless lines that we need to maintain.
Let's just remove any extraneous work and make the addition to the
environment triggered by the -d option as obvious and concise as
possible.
Signed-off-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Greg Troxel [Fri, 16 Dec 2011 14:53:47 +0000 (09:53 -0500)]
Extend README for NetBSD.
Add NetBSD to the list of systems on which bup is known to work. Give
hints for bup usage on NetBSD, including the location of the fuse
bindings and the pkgsrc entry. Caution about incorrect cycle
detection on fuse mounts. Add pkgsrc URLs.
Signed-off-by: Greg Troxel <gdt@lexort.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Aneurin Price [Thu, 12 Jan 2012 16:15:23 +0000 (16:15 +0000)]
cmd/drecurse: correctly pass excluded_paths to recursive_dirlist
The excluded_paths argument was being passed as a positional argument,
but its position actually corresponded to the 'bup_dir' argument, so
'bup drecurse --exclude=/foo/bar /foo' has never worked.
Signed-off-by: Aneurin Price <aneurin.price@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
The function now returns immediately if the two arguments are the same
Python object, otherwise it compares the full path name (rather than
just the file name).
Avery Pennarun [Mon, 19 Mar 2012 18:58:45 +0000 (14:58 -0400)]
options.py: clean up handling of --no-* options.
The particular bug that triggered this (in a project other than bup) was of
the form:
n,no-stupid don't be stupid
Where it would actually end up setting stupid=1 by accident, and -n would
mean --stupid, not --no-stupid. As part of fixing it, you can now also do
this:
n,no-stupid,smart don't be stupid (ie. be smart)
and it'll work as it should: n == smart == no-stupid == not stupid.
Avery Pennarun [Mon, 19 Mar 2012 18:34:51 +0000 (14:34 -0400)]
options.py: don't crash given semi-invalid optspecs.
It's kind of weird to provide an argument without a description, but it's
not crash-worthy (especially when the crash was a totally unhelpful
exception stack trace). While we're here, test for a couple of other ones
that didn't cause a crash, but we want to keep it that way.
And fix the copyright message; actually options.py started in 2010.
Avery Pennarun [Thu, 19 Jan 2012 23:36:13 +0000 (15:36 -0800)]
cmd/ftp: fix tab completion on MacOS.
MacOS doesn't use the "real" readline, and the clone it uses is slightly
incompatible in its bindings. Just bind both and it seems to work on both
MacOS and Linux.
Avery Pennarun [Mon, 31 Oct 2011 21:49:55 +0000 (17:49 -0400)]
bupsplit.c: remove extra-large stack-allocated array from selftest().
In some rare cases involving userspace threads (where you're running the
selftest function for some reason?) this could cause stack overflow or
excess memory usage. Let's just do it with plain malloc().
Avery Pennarun [Thu, 9 Jun 2011 03:18:17 +0000 (23:18 -0400)]
Disable t/test-meta.sh in 'make test' unless TEST_META=1
In other words:
make test # doesn't run metadata tests
TEST_META=1 make test # does run metadata tests
The metadata tests still fail randomly on some people's computers, but we're
falling too far behind and it's time to make a release. The metadata stuff
isn't used anywhere critical in bup yet, so it's okay to leave it in but not
test it for now.
Avery Pennarun [Thu, 9 Jun 2011 03:15:48 +0000 (23:15 -0400)]
Merge branch 'meta'
* meta:
Add utimes/lutimes implementations of _helpers utime() and lutime().
Replace _helpers.utimensat() with utime() and lutime().
Test for available nanosecond stat timestamp members.
Add config.h dependency to _helpers in csetup.py.
Add -*-shell-script-*- to configure.inc.
Use FS_IOC_GETFLAGS/FS_IOC_SETFLAGS directly as the preprocessor guards.
Verify the expected length of saved_errors in tmetadata.py.
Don't use xstat.lutime() in test-meta.sh when xstat.utime() will do.
Add meta support for restoring filesystem sockets.
Add _recognized_file_types(); defer error for unrecognized restore.
index.py: new format (V3), with inodes, link counts, and 64-bit times.
Cap timestamps in index to avoid needing to worry about fractional parts.
index.py: factor out an Entry._fixup_time method.
Rely on options.parse() for more of the meta and xstat argument processing.
Remove vestigal clean target comment regarding pybuptest.tmp permissions.
Add initial timespec behavior tests.
Return None from bup_set_linux_file_attr() and bup_utimensat().
Replace os.*stat() with xstat.*stat(); use integer ns for all fs times.
Drop xstat floating point timestamp support -- use integer ns.
xstst-cmd.py: test for _have_utimensat rather than _have_ns_fs_timestamps.
Rob Browning [Wed, 1 Jun 2011 00:49:33 +0000 (19:49 -0500)]
Replace _helpers.utimensat() with utime() and lutime().
Rework utimensat() handling in preparation for the addition of
utimes/lutimes based fallbacks.
Publish lutime() and utime() at the Python level from _helpers.c
rather than utimensat() itself.
Drop the _have_utimensat tests in favor of testing xstat.lutime which
will be false when xstat.lutime() is not available.
Move bup_utimensat() Python argument parsing to
bup_parse_xutime_args() and use it to implement bup_utime_ns() and
bup_lutime_ns(). This argument parsing will eventually be shared by
the utimes/lutimes based fallbacks.
Remove _helpers.AT_FDCWD and _helpers.AT_SYMLINK_NOFOLLOW since
utimensat is no longer published on the Python side.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 1 Jun 2011 00:49:32 +0000 (19:49 -0500)]
Test for available nanosecond stat timestamp members.
Use st_atim, st_mtim, and st_ctim when available, and fall back to
st_atimensec, st_mtimensec, and st_ctimensec. If neither are
available, return 0 ns values.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 1 Jun 2011 00:49:25 +0000 (19:49 -0500)]
Add _recognized_file_types(); defer error for unrecognized restore.
Defer an error if an unrecognized file type is encountered during
metadata restoration -- whether during path creation
(i.e. --start-extract) or metadata application
(i.e. --finish-extract).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Aneurin Price [Wed, 1 Jun 2011 16:20:36 +0000 (17:20 +0100)]
configure.inc: strip trailing characters from 'uname -s' output
On Cygwin, 'uname -s' includes the version of the underlying operating
system; here it is 'CYGWIN_NT-6.0'. The configure script attempts to
define this in config/config.h, but '#define OS_CYGWIN_NT-6.0 1' is an
invalid macro definition.
This truncates the value to just 'CYGWIN', to match $OS in the Makefile.
Avery Pennarun [Tue, 7 Jun 2011 02:09:39 +0000 (22:09 -0400)]
Update bupsplit.[ch] to have a less restrictive BSD-style license.
Nobody else owns any copyright on those files, so I can do this without
asking anyone else for permission :)
Some people have asked to use these files in their own non-GPL/LGPL
projects, and although the LGPL permits this, the files are so small that
there's no reason to be obnoxious about it. So let's just let them do it
the easy way.
Aaron M. Ucko [Mon, 30 May 2011 23:03:01 +0000 (19:03 -0400)]
index.py: new format (V3), with inodes, link counts, and 64-bit times.
To allow unambiguous preservation of hard-link structure, index device
numbers, inode numbers (new) and link counts (new) at 64 bits apiece
per GNU libc, which uses uint64_t, uint64_t, and unsigned long respectively.
Take the opportunity to use 64 bits for mtime and ctime as well, both
to be ready for Y2038 and to handle NTFS's zero value (Y1600).
Aaron M. Ucko [Mon, 30 May 2011 23:03:00 +0000 (19:03 -0400)]
Cap timestamps in index to avoid needing to worry about fractional parts.
Avoid a potential race condition by which bup's use of whole-second
granularity for timestamps in the index could let it theoretically
miss some last-second changes by capping timestamps to at most one
second before the start of indexing per a newly introduced mandatory
parameter to bup.index.Writer.
Aaron M. Ucko [Mon, 30 May 2011 23:02:35 +0000 (19:02 -0400)]
Improve formatting of error and warning messages.
log() trailing newlines as appropriate. Fix a format string typo in
lib/bup/git.py encountered when verifying that exceptions' string
values already end with newlines.
Avery Pennarun [Mon, 30 May 2011 00:50:25 +0000 (20:50 -0400)]
Merge branch 'master' into meta
* master: (27 commits)
t/test.sh: 'ls' on NetBSD sets -A by default as root; work around it.
README: add a list of binary packages
README: rework the title hierarchy
Clarify the message when the BUP_DIR doesn't exist.
Refactor: unify ls/ftp-ls code
ftp/ls: Adjust documentation
ls: include hidden files when explicitly requested
ftp: implement ls -s (show hashes)
ftp/ls: columnate output attached to a tty, else don't
ftp: don't output trailing line for 'ls'
ftp: output a newline on EOF when on a tty
config: more config stuff to config/ subdir, call it from Makefile.
cmd/{split,save}: support any compression level using the new -# feature.
options.py: add support for '-#' style compression options.
Add documentation for compression levels
Add test case for compression level
Add compression level options to bup save and bup split
Make zlib compression level a parameter for Client
Make zlib compression level a parameter of git.PackWriter
Use is_superuser() rather than checking euid directly
...
Gabriel Filion [Mon, 16 May 2011 05:13:28 +0000 (01:13 -0400)]
README: add a list of binary packages
Debian/Ubuntu are known to have bup packages in their archives, thanks
to Jon Dowland.
Also, a NetBSD package is currently being built, as was shared by Thomas
Klausner. However, it is still not found in the official NetBSD packages
search engine.
Gabriel Filion [Mon, 16 May 2011 05:13:27 +0000 (01:13 -0400)]
README: rework the title hierarchy
In Markdown, a line underlining another one with '=' characters
represents a first level title, while a line underlining another one
with '-' characters represents a second level title.
Rework the title levels to gain visibility on the different sections and
to allow to split "Getting started" more easily (see my next commit for
additions to this section).
Gabriel Filion [Mon, 16 May 2011 04:27:24 +0000 (00:27 -0400)]
Refactor: unify ls/ftp-ls code
Both the 'ls' command and the 'ls' subcommand of the 'ftp' command use
some code that is very similar. Modifications must be done in two places
instead of one and this can lead to inconsistencies.
Refactor code so that both paths use the same function with the same opt
spec.
Gabriel Filion [Mon, 16 May 2011 04:27:22 +0000 (00:27 -0400)]
ls: include hidden files when explicitly requested
The current code of 'bup ls' insists on hiding a file from its listing
even if the file was explicitly requested as an argument. This is not
what users would expect. Remove the condition and always list files
(not directories) starting with a dot when they were given in the
argument list.
Gabriel Filion [Mon, 16 May 2011 04:27:21 +0000 (00:27 -0400)]
ftp: implement ls -s (show hashes)
'bup ls' has a -s flag that can be used to show file hashes on the left
of each file name. 'bup ftp ls' doesn't have that feature.
Implement the feature by copying code from 'bup ls'. This is the last
feature difference between 'bup ls' and 'bup ftp ls' and bringing them
to the same level will make it possible to unify the code that is used
by both.
Gabriel Filion [Mon, 16 May 2011 04:27:20 +0000 (00:27 -0400)]
ftp/ls: columnate output attached to a tty, else don't
'bup ftp ls' and 'bup ls' currently behave in a different manner.
'bup ftp ls' always formats its output in columns regardless of whether
the program's stdout is a tty or not.
'bup ls' always prints one name on each line.
Make both of those commands behave the same. By using lib/bup/helpers'
istty1 variable, decide to format in columns when outputting to a tty,
and to output one file name per line when the output is not a tty.
Gabriel Filion [Mon, 16 May 2011 04:27:19 +0000 (00:27 -0400)]
ftp: don't output trailing line for 'ls'
'ls' is currently the only 'ftp' subcommand that outputs a trailing
newline before the prompt is re-displayed. This is cause by the use of
"print" to output a string that already contains an ending newline.
For a matter of consistency of output, make 'ls' output without that
extra trailing newline.
projects / bup.git / log