Greg Troxel [Fri, 16 Dec 2011 14:53:47 +0000 (09:53 -0500)]
Extend README for NetBSD.
Add NetBSD to the list of systems on which bup is known to work. Give
hints for bup usage on NetBSD, including the location of the fuse
bindings and the pkgsrc entry. Caution about incorrect cycle
detection on fuse mounts. Add pkgsrc URLs.
Signed-off-by: Greg Troxel <gdt@lexort.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Aneurin Price [Thu, 12 Jan 2012 16:15:23 +0000 (16:15 +0000)]
cmd/drecurse: correctly pass excluded_paths to recursive_dirlist
The excluded_paths argument was being passed as a positional argument,
but its position actually corresponded to the 'bup_dir' argument, so
'bup drecurse --exclude=/foo/bar /foo' has never worked.
Signed-off-by: Aneurin Price <aneurin.price@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
The function now returns immediately if the two arguments are the same
Python object, otherwise it compares the full path name (rather than
just the file name).
Avery Pennarun [Mon, 19 Mar 2012 18:58:45 +0000 (14:58 -0400)]
options.py: clean up handling of --no-* options.
The particular bug that triggered this (in a project other than bup) was of
the form:
n,no-stupid don't be stupid
Where it would actually end up setting stupid=1 by accident, and -n would
mean --stupid, not --no-stupid. As part of fixing it, you can now also do
this:
n,no-stupid,smart don't be stupid (ie. be smart)
and it'll work as it should: n == smart == no-stupid == not stupid.
Avery Pennarun [Mon, 19 Mar 2012 18:34:51 +0000 (14:34 -0400)]
options.py: don't crash given semi-invalid optspecs.
It's kind of weird to provide an argument without a description, but it's
not crash-worthy (especially when the crash was a totally unhelpful
exception stack trace). While we're here, test for a couple of other ones
that didn't cause a crash, but we want to keep it that way.
And fix the copyright message; actually options.py started in 2010.
Avery Pennarun [Thu, 19 Jan 2012 23:36:13 +0000 (15:36 -0800)]
cmd/ftp: fix tab completion on MacOS.
MacOS doesn't use the "real" readline, and the clone it uses is slightly
incompatible in its bindings. Just bind both and it seems to work on both
MacOS and Linux.
Avery Pennarun [Mon, 31 Oct 2011 21:49:55 +0000 (17:49 -0400)]
bupsplit.c: remove extra-large stack-allocated array from selftest().
In some rare cases involving userspace threads (where you're running the
selftest function for some reason?) this could cause stack overflow or
excess memory usage. Let's just do it with plain malloc().
Avery Pennarun [Thu, 9 Jun 2011 03:18:17 +0000 (23:18 -0400)]
Disable t/test-meta.sh in 'make test' unless TEST_META=1
In other words:
make test # doesn't run metadata tests
TEST_META=1 make test # does run metadata tests
The metadata tests still fail randomly on some people's computers, but we're
falling too far behind and it's time to make a release. The metadata stuff
isn't used anywhere critical in bup yet, so it's okay to leave it in but not
test it for now.
Avery Pennarun [Thu, 9 Jun 2011 03:15:48 +0000 (23:15 -0400)]
Merge branch 'meta'
* meta:
Add utimes/lutimes implementations of _helpers utime() and lutime().
Replace _helpers.utimensat() with utime() and lutime().
Test for available nanosecond stat timestamp members.
Add config.h dependency to _helpers in csetup.py.
Add -*-shell-script-*- to configure.inc.
Use FS_IOC_GETFLAGS/FS_IOC_SETFLAGS directly as the preprocessor guards.
Verify the expected length of saved_errors in tmetadata.py.
Don't use xstat.lutime() in test-meta.sh when xstat.utime() will do.
Add meta support for restoring filesystem sockets.
Add _recognized_file_types(); defer error for unrecognized restore.
index.py: new format (V3), with inodes, link counts, and 64-bit times.
Cap timestamps in index to avoid needing to worry about fractional parts.
index.py: factor out an Entry._fixup_time method.
Rely on options.parse() for more of the meta and xstat argument processing.
Remove vestigal clean target comment regarding pybuptest.tmp permissions.
Add initial timespec behavior tests.
Return None from bup_set_linux_file_attr() and bup_utimensat().
Replace os.*stat() with xstat.*stat(); use integer ns for all fs times.
Drop xstat floating point timestamp support -- use integer ns.
xstst-cmd.py: test for _have_utimensat rather than _have_ns_fs_timestamps.
Rob Browning [Wed, 1 Jun 2011 00:49:33 +0000 (19:49 -0500)]
Replace _helpers.utimensat() with utime() and lutime().
Rework utimensat() handling in preparation for the addition of
utimes/lutimes based fallbacks.
Publish lutime() and utime() at the Python level from _helpers.c
rather than utimensat() itself.
Drop the _have_utimensat tests in favor of testing xstat.lutime which
will be false when xstat.lutime() is not available.
Move bup_utimensat() Python argument parsing to
bup_parse_xutime_args() and use it to implement bup_utime_ns() and
bup_lutime_ns(). This argument parsing will eventually be shared by
the utimes/lutimes based fallbacks.
Remove _helpers.AT_FDCWD and _helpers.AT_SYMLINK_NOFOLLOW since
utimensat is no longer published on the Python side.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 1 Jun 2011 00:49:32 +0000 (19:49 -0500)]
Test for available nanosecond stat timestamp members.
Use st_atim, st_mtim, and st_ctim when available, and fall back to
st_atimensec, st_mtimensec, and st_ctimensec. If neither are
available, return 0 ns values.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 1 Jun 2011 00:49:25 +0000 (19:49 -0500)]
Add _recognized_file_types(); defer error for unrecognized restore.
Defer an error if an unrecognized file type is encountered during
metadata restoration -- whether during path creation
(i.e. --start-extract) or metadata application
(i.e. --finish-extract).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Aneurin Price [Wed, 1 Jun 2011 16:20:36 +0000 (17:20 +0100)]
configure.inc: strip trailing characters from 'uname -s' output
On Cygwin, 'uname -s' includes the version of the underlying operating
system; here it is 'CYGWIN_NT-6.0'. The configure script attempts to
define this in config/config.h, but '#define OS_CYGWIN_NT-6.0 1' is an
invalid macro definition.
This truncates the value to just 'CYGWIN', to match $OS in the Makefile.
Avery Pennarun [Tue, 7 Jun 2011 02:09:39 +0000 (22:09 -0400)]
Update bupsplit.[ch] to have a less restrictive BSD-style license.
Nobody else owns any copyright on those files, so I can do this without
asking anyone else for permission :)
Some people have asked to use these files in their own non-GPL/LGPL
projects, and although the LGPL permits this, the files are so small that
there's no reason to be obnoxious about it. So let's just let them do it
the easy way.
Aaron M. Ucko [Mon, 30 May 2011 23:03:01 +0000 (19:03 -0400)]
index.py: new format (V3), with inodes, link counts, and 64-bit times.
To allow unambiguous preservation of hard-link structure, index device
numbers, inode numbers (new) and link counts (new) at 64 bits apiece
per GNU libc, which uses uint64_t, uint64_t, and unsigned long respectively.
Take the opportunity to use 64 bits for mtime and ctime as well, both
to be ready for Y2038 and to handle NTFS's zero value (Y1600).
Aaron M. Ucko [Mon, 30 May 2011 23:03:00 +0000 (19:03 -0400)]
Cap timestamps in index to avoid needing to worry about fractional parts.
Avoid a potential race condition by which bup's use of whole-second
granularity for timestamps in the index could let it theoretically
miss some last-second changes by capping timestamps to at most one
second before the start of indexing per a newly introduced mandatory
parameter to bup.index.Writer.
Aaron M. Ucko [Mon, 30 May 2011 23:02:35 +0000 (19:02 -0400)]
Improve formatting of error and warning messages.
log() trailing newlines as appropriate. Fix a format string typo in
lib/bup/git.py encountered when verifying that exceptions' string
values already end with newlines.
Avery Pennarun [Mon, 30 May 2011 00:50:25 +0000 (20:50 -0400)]
Merge branch 'master' into meta
* master: (27 commits)
t/test.sh: 'ls' on NetBSD sets -A by default as root; work around it.
README: add a list of binary packages
README: rework the title hierarchy
Clarify the message when the BUP_DIR doesn't exist.
Refactor: unify ls/ftp-ls code
ftp/ls: Adjust documentation
ls: include hidden files when explicitly requested
ftp: implement ls -s (show hashes)
ftp/ls: columnate output attached to a tty, else don't
ftp: don't output trailing line for 'ls'
ftp: output a newline on EOF when on a tty
config: more config stuff to config/ subdir, call it from Makefile.
cmd/{split,save}: support any compression level using the new -# feature.
options.py: add support for '-#' style compression options.
Add documentation for compression levels
Add test case for compression level
Add compression level options to bup save and bup split
Make zlib compression level a parameter for Client
Make zlib compression level a parameter of git.PackWriter
Use is_superuser() rather than checking euid directly
...
Gabriel Filion [Mon, 16 May 2011 05:13:28 +0000 (01:13 -0400)]
README: add a list of binary packages
Debian/Ubuntu are known to have bup packages in their archives, thanks
to Jon Dowland.
Also, a NetBSD package is currently being built, as was shared by Thomas
Klausner. However, it is still not found in the official NetBSD packages
search engine.
Gabriel Filion [Mon, 16 May 2011 05:13:27 +0000 (01:13 -0400)]
README: rework the title hierarchy
In Markdown, a line underlining another one with '=' characters
represents a first level title, while a line underlining another one
with '-' characters represents a second level title.
Rework the title levels to gain visibility on the different sections and
to allow to split "Getting started" more easily (see my next commit for
additions to this section).
Gabriel Filion [Mon, 16 May 2011 04:27:24 +0000 (00:27 -0400)]
Refactor: unify ls/ftp-ls code
Both the 'ls' command and the 'ls' subcommand of the 'ftp' command use
some code that is very similar. Modifications must be done in two places
instead of one and this can lead to inconsistencies.
Refactor code so that both paths use the same function with the same opt
spec.
Gabriel Filion [Mon, 16 May 2011 04:27:22 +0000 (00:27 -0400)]
ls: include hidden files when explicitly requested
The current code of 'bup ls' insists on hiding a file from its listing
even if the file was explicitly requested as an argument. This is not
what users would expect. Remove the condition and always list files
(not directories) starting with a dot when they were given in the
argument list.
Gabriel Filion [Mon, 16 May 2011 04:27:21 +0000 (00:27 -0400)]
ftp: implement ls -s (show hashes)
'bup ls' has a -s flag that can be used to show file hashes on the left
of each file name. 'bup ftp ls' doesn't have that feature.
Implement the feature by copying code from 'bup ls'. This is the last
feature difference between 'bup ls' and 'bup ftp ls' and bringing them
to the same level will make it possible to unify the code that is used
by both.
Gabriel Filion [Mon, 16 May 2011 04:27:20 +0000 (00:27 -0400)]
ftp/ls: columnate output attached to a tty, else don't
'bup ftp ls' and 'bup ls' currently behave in a different manner.
'bup ftp ls' always formats its output in columns regardless of whether
the program's stdout is a tty or not.
'bup ls' always prints one name on each line.
Make both of those commands behave the same. By using lib/bup/helpers'
istty1 variable, decide to format in columns when outputting to a tty,
and to output one file name per line when the output is not a tty.
Gabriel Filion [Mon, 16 May 2011 04:27:19 +0000 (00:27 -0400)]
ftp: don't output trailing line for 'ls'
'ls' is currently the only 'ftp' subcommand that outputs a trailing
newline before the prompt is re-displayed. This is cause by the use of
"print" to output a string that already contains an ending newline.
For a matter of consistency of output, make 'ls' output without that
extra trailing newline.
Gabriel Filion [Sat, 14 May 2011 23:07:56 +0000 (19:07 -0400)]
ftp: output a newline on EOF when on a tty
Using the 'quit' command with ftp while in interactive mode -- attached
to a tty -- ends up clearing the line for the shell to use a fresh one
for the next prompt.
Using Ctrl-D to send an EOF to the application's input while in
interactive mode currently does not clear the line in the same way.
Let's force a newline when an EOF is received from a tty so that the
program exits in a more aesthetic way.
Avery Pennarun [Sun, 15 May 2011 21:06:51 +0000 (17:06 -0400)]
Merge branch 'master' into config
* master:
cmd/{split,save}: support any compression level using the new -# feature.
options.py: add support for '-#' style compression options.
Add documentation for compression levels
Add test case for compression level
Add compression level options to bup save and bup split
Make zlib compression level a parameter for Client
Make zlib compression level a parameter of git.PackWriter
Use is_superuser() rather than checking euid directly
Add is_superuser() helper function
Makefile: add a PREFIX variable for locations other than /usr.
Avery Pennarun [Sun, 8 May 2011 19:09:04 +0000 (19:09 +0000)]
Earlier "negative timestamp" patch had a 64-bit timestamp in the test.
The date in the comment is correct - for -0x80000000. Sadly, the *code*
actually said -0x90000000. That works on 64-bit systems (and filesystems)
not not 32-bit ones, where python gives an encoding error.
In any case, based on the comment (June 10, 1893) it seems tat -0x80000000
must have been the intended value anyway. Now 'make test' passes on 32-bit
Linux again.
Rob Browning [Sun, 27 Mar 2011 17:01:45 +0000 (12:01 -0500)]
Replace os.*stat() with xstat.*stat(); use integer ns for all fs times.
Replace all calls of the os.*stat() functions with calls to the xstat
equivalents. This should leave bup with the xstat stat representation
(and integer ns timestamps) everywhere.
Remove FSTime, and add a few xstat conversion functions to replace the
bits we still want: timespec_to_nsecs(), nsecs_to_timespec(),
fstime_floor_secs(), fstime_to_timespec().
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 27 Mar 2011 17:01:44 +0000 (12:01 -0500)]
Drop xstat floating point timestamp support -- use integer ns.
Drop conditional support for floating point timestamps (the xstat
os.stat() fallback). Switch to integer nanosecond timestamps
everywhere except the metadata record encoding and _helpers.c.
The metadata encoding is still a timespec because separate s and ns
timespec vints compress much better, and timespecs are still returned
by _helpers because the conversion to integer nanoseconds is much more
convenient in Python.
Enforce timespec range expectations -- throw an exception if the
system returns a nanosecond value less than 0 or greater than 999999999. Remove _have_ns_fs_timestamps.
Depend on bup_stat(), bup_stat(), and bup_lstat() unconditionally, and
change the timespec return conversion from "(ll)" to "(Ll)", i.e. long
long range for secs.
This commit may break the build on some platforms -- we'll have to add
suitable conditionals once we see what's needed.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 27 Mar 2011 17:01:43 +0000 (12:01 -0500)]
xstst-cmd.py: test for _have_utimensat rather than _have_ns_fs_timestamps.
Test for _have_utimensat rather than _have_ns_fs_timestamps to decide
whether or not to print the atime and mtime, since the existence of
utimensat() is the real indicator, and since _have_ns_fs_timestamps is
going away.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Avery Pennarun [Sun, 8 May 2011 07:13:48 +0000 (03:13 -0400)]
Merge branch 'master' of git://github.com/thatch/bup
* 'master' of git://github.com/thatch/bup:
Missing space in optspec
Fix a bug where marginally old files couldn't be stored in the index
Show better errors with out-of-range Entry values
Gabriel Filion [Mon, 2 May 2011 23:12:34 +0000 (19:12 -0400)]
Doc: add some precisions for --remote and dumb mode
The -r/--remote argument to some of bup's commands currently doesn't
give enough information about how to customize options to SSH. Let's add
information about this so that users know how to customize options for
SSH connections.
Also, in bup-server's documentation, point out which mode is the default
one for more clarity.
Tim Hatch [Sat, 16 Apr 2011 00:18:51 +0000 (17:18 -0700)]
Fix a bug where marginally old files couldn't be stored in the index
Due to the struct having unsigned timestamps, files with dates between Dec 13,
1901 and Jan 1, 1970 were not representable. This change extends the struct to
be able to pack signed timestamps, which was the spirit of code in _fixup, and
extends the useful range back to 1901. Timestamps prior to 1901 are still
adjusted to zero, as they were before.
There should be no compatibility problems loading packed structures created
before this change, since positive values were truncated at 0x7fffffff.
Rob Browning [Mon, 21 Mar 2011 01:35:53 +0000 (20:35 -0500)]
tgit.py: provoke ENOTDIR rather than EACCES in test_check_repo_or_die().
Replace the objects/pack directory with an empty file to provoke an
ENOTDIR error from stat('objects/pack/.').
Previously the code changed the permissions of the test directory to
0000 in order to provoke an error other than ENOENT (i.e. EACCES), but
that doesn't work when the tests are run as root or fakeroot.
(As Gabriel Filion pointed out, the chmod of the testdir is no
longer necessary, so I removed it and squashed that into this patch.
-- apenwarr)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
Avery Pennarun [Sun, 20 Mar 2011 10:53:42 +0000 (03:53 -0700)]
cmd/xstat: don't report mtime/atime for symlinks if we don't have_ns_timestamps.
We can't set the atime/mtime on a symlink anyway if we don't
have_ns_timestamps, which means the values are meaningless. Report them as
0 in order to avoid triggering a unit test failure.
Avery Pennarun [Sun, 20 Mar 2011 10:46:32 +0000 (03:46 -0700)]
test-meta.sh: remove a bashism, and don't delete dirs on exit.
It's really annoying to have it wiping out directories that you want to
examine after a failed test. And "set -o pipefail" is not available in the
version of bash on MacOS 10.4.
Avery Pennarun [Sun, 20 Mar 2011 10:45:32 +0000 (03:45 -0700)]
metadata.py: be careful with the umask() when restoring symlinks.
On MacOS, the umask affects symlink permissions, although not in any sort of
useful way that I can see. Still, getting the permissions wrong breaks the
unit tests, so let's be careful about it.
projects / bup.git / log