Signed-off-by: Mark J Hewitt <m.hewitt@computer.org>
[rlb@defaultvalue.org: adjust commit message, and include a leading
single quote in what we're looking for, to match c8c1f0341c82f4abf4d3f8eed1fe4ddfa4a48493.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 22 Mar 2014 15:25:32 +0000 (10:25 -0500)]
test-ls.sh: get the group and gid from the filesystem.
Get the expected gid from the filesystem, not "id", because on some
platforms (BSDs, etc.) a new path's gid is taken from the parent
directory, not the effective gid.
Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem
and to him and Greg Troxel <gdt@ir.bbn.com> for helping craft the
solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 18 Feb 2014 18:30:31 +0000 (12:30 -0600)]
test-ls.sh: take only the first 10 chars from ls -l's mode string.
Since ls -l's mode string may not be separated from the next field by
a space (i.e. when ACLs, etc. are involved), take only the first 10
characters for now when retrieving the symlink mode string (cf. 30d9027cc5444f038d38927219dc59e3b69fa219).
Thanks to Mark J Hewitt <mjh@idnet.com> for pointing out the problem
and testing the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 17 Feb 2014 18:39:05 +0000 (12:39 -0600)]
test-compression.sh: use "tar ... | wc -c" instead of du.
This test was failing on a ZFS system with compression enabled because
du wasn't summing the actual file sizes. To fix that, use tar/wc so
that we'll calculate the actual tree size without having to worry
about the portability of du's --apparent-size argument.
Thanks to Björn Seifert <seifert@oern.de> for reporting the problem
and Greg Troxel <gdt@lexort.com> for suggesting this fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Always return a level 0 blob from _splitbuf() for BLOB_MAX sized blobs.
Previously if _splitbuf() returned an offset anywhere in the buffer we
would always use the 'level' of the offset for the resulting blob. So
when the offset was greater than BLOB_MAX, we'd still use the offset's
level, even though we'd be returning a blob that didn't reach that
offset. In some cases, this could cause bup to generate a sequence of
BLOB_MAX sized blobs, each in their own own chunk group.
To fix that, set the level of all BLOB_MAX sized blobs to 0, which
makes those blobs behave just as they would have before this patch,
whenever they were found at the end of the buffer.
Test the new behavior -- the last two of these four new tests will
fail if run before the changes, as they check that blobs split by
enforcing BLOB_MAX do not 'inherit' the value of level from a split
later in the buffer.
Signed-off-by: Aidan Hobson Sayers <aidanhs@cantab.net>
[rlb@defaultvalue.org: squash tests commit into _splitbuf() changes commit;
adjust commit message.] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 31 Jan 2014 00:49:36 +0000 (18:49 -0600)]
Fix drecurse relative --excludes, quash duplicates, and add tests.
Previously "bup drecurse --exclude bar foo" wouldn't work because bar
would be expanded to an absolute path, but drecurse would traverse
(and attempt to match against) relative paths.
Have parse_excludes() remove duplicates and sort its result.
Add some initial drecurse tests.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
This comment was originally added in reference to some confusion over
globbing, but the text was misleading because bup does apply
realpath() to all of the --exclude paths, arguably a form of
expansion.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
itxx00 [Wed, 8 Jan 2014 05:14:26 +0000 (13:14 +0800)]
bup-restore.md: always initialize root_meta in do_root.
Previously, using bup split with bup restore would cause an error:
UnboundLocalError: local variable 'root_meta' referenced before assignment
Signed-off-by: itxx00 <itxx00@gmail.com>
[rlb@defaultvalue.org: adjust commit message. Initialize root_meta
unconditionally before guard to match other code.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 3 Dec 2013 18:37:53 +0000 (12:37 -0600)]
Check for overflow when converting from Python to unsigned integer types.
Python's PyArg_ParseTuple() unsigned integer type conversions don't
check for overflow, so stop using them. Add bup_uint_from_py(),
bup_ulong_from_py(), and bup_ullong_from_py() functions that do check,
and use them instead.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 19 Dec 2013 04:16:44 +0000 (22:16 -0600)]
Don't require non-negative timespec ns; fix stat timestamp conversions.
Previously we required that the system timespec tv_nsec values be
non-negative -- stop that. POSIX doesn't specify, and there's no real
need for us to care, since all significant timestamp manipulations are
now handled as integer nanosecond values, not (sec, ns) pairs.
Change stat_struct_to_py() (used by bup_stat()/bup_lstat()) to use
INTEGER_TO_PY() to convert the timestamp st_atime/st_mtime/st_ctime
values so that we don't have to care about the sign/size of the
underlying type (which should be time_t).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 18 Dec 2013 03:12:06 +0000 (21:12 -0600)]
Always publish (l)utimes in helpers when available and fix type conversions.
Previously, bup assumed a timeval was (long, long), which agrees with
glibc.info and (Linux) utimes(2), but POSIX says it's (time_t,
suseconds_t).
Since the types vary (and even if they didn't, Python doesn't provide
safe "time_t" conversions), handle the type conversions more
generically, via ASSIGN_PYLONG_TO_INTEGRAL() and
INTEGRAL_ASSIGNMENT_FITS().
Optimistically assume the latter is sufficient for tv_usec (where we
convert it to a long long) because POSIX guarantees that tv_usec will
be a signed integral.
Whenever available, publish utimes and lutimes independently in
helpers, and run the relevant t/txstat.py tests. So now, we'll be
running at least some tests on utimensat, lutimes, and utimes whenever
they're available.
A more significant improvement would be to run some broader bup tests
twice, once with utimensat and once with utimes/lutimes, whenever both
are available.
Rob Browning [Sat, 28 Dec 2013 18:18:51 +0000 (12:18 -0600)]
Remove vestigial and inappropriate isnan()/isinf() timestamp tests.
I'm fairly sure this is an historical accident, but regardless, remove
the (my) obviously inappropriate floating point tests on integer
values from bup_parse_xutime_args(). Nothing to see here -- move
along.
I've broken this change out from a larger post-0.25 patch, where I'd
already fixed it, so that it'll be easy to pull into stable if we
like.
Thanks to Ed Maste <carpeddiem@gmail.com> for reporting the problem
(which prompted me to go ahead and create the separate fix).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 24 Dec 2013 21:34:44 +0000 (15:34 -0600)]
Add -n, -A, -F, --file-type, --numeric-ids and detailed -l to "bup ls".
With the addition of -A, change -a to behave more conventionally,
i.e. to show "." and "..".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Alexander Barton <alex@barton.de> Reviewed-By: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: fall back to n.mode when n.metadata() isn't
available in node_info() so that CommitDirs ("ls -AF /"), etc. will
still work; thanks to Gabriel Filion <gabster@lelutin.ca> for the
report; fix the rdiff-backup test to use -A.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 18 Dec 2013 02:56:52 +0000 (20:56 -0600)]
Always publish utimensat in helpers when available and fix type conversions.
Previously, bup assumed a timespec was (long, long), which agrees with
glibc.info, but utimensat(2) and POSIX say it's (time_t, long).
Since the types vary (and even if they didn't, Python doesn't provide
safe "time_t" conversions), add ASSIGN_PYLONG_TO_INTEGRAL(), which
should handle safely assigning a PyLong value to any given C integral
destination, whenever it will fit -- then use that macro to handle the
utimensat arguments.
Whenever it's available, publish utimensat independently in helpers.
For now, this allows us to test utimensat directly, and in the future,
it may allow us to test bup's behavior with either utimensat or
utimes/lutimes without recompilation, whenever both are available.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 18 Dec 2013 02:19:44 +0000 (20:19 -0600)]
Don't fail tests when the timestamp read resolution is higher than write.
Previously, if bup was able to read path timestamps at a higher
resolution than it could write them, tests would fail. This situation
can occur (for example) when the stat() resolution is 1ns, but either
the underlying filesystem's isn't, or bup wasn't able to find
utimensat at build time.
To fix this, compute the maximum resolution of the test filesystem
(via "t/ns-timestamp-resolutions HERE"), and then limit the "bup
xstat" timestamp resolution in the tests to match via a new
--mtime-resolution argument.
For completeness, also add --atime-resolution and --ctime-resolution
arguments.
Thanks to Tim Riemenschneider <t.riemenschneider@detco.de> for
reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 19 Dec 2013 02:19:19 +0000 (20:19 -0600)]
Stop testing negative timestamps -- they're not necessarily valid.
Both timevals and timespecs use a time_t for the seconds which POSIX
requires to be an "integer or real-floating type", so unsigned integer
types are perfectly valid. (See FUSE for example -- though it's not
even using a time_t; at the moment it's just using uint64_t...).
Thanks to Tim Riemenschneider <t.riemenschneider@detco.de> for
reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 16 Dec 2013 02:30:46 +0000 (20:30 -0600)]
Require utimensat or utimes/lutimes.
This was already true (tests would fail), so make it explicit, and if
we do encounter any platforms where none of these functions are
available, we can worry about it then.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 6 Dec 2013 16:18:37 +0000 (10:18 -0600)]
t/test-meta.sh: handle systems that have sgid directories.
Explicitly chown the test directory to a group we're in so that the
tests will work on systems that have sgid directories by default
(i.e. NetBSD, etc.).
Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem
and helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 3 Dec 2013 20:17:26 +0000 (14:17 -0600)]
Create t/force-delete from t/lib.sh force-delete and use it everywhere.
Move the t/lib.sh force-delete code to t/force-delete as a standalone
command, and use it everywhere -- particularly to delete t/tmp during
"make clean", given the recent move to run all tests inside t/ via
TMPDIR (18f838c9a66d521f9fa042eead3c5771ca99b03a).
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 25 Nov 2013 20:31:28 +0000 (14:31 -0600)]
Don't check ACLs in t/compare-trees on NetBSD.
Treat NetBSD like Cygwin and Darwin (OS X) in t/compare-trees. This
should finish the process of disabling POSIX ACL support on NetBSD
that was started here: 349ff15c7db09883abc20bdf3e8a4df6bff12cd3.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Nov 2013 18:05:00 +0000 (12:05 -0600)]
Don't check ACLs in t/compare-trees on FreeBSD.
Treat FreeBSD like Cygwin and Darwin (OS X) in t/compare-trees. This
should finish the process of disabling POSIX ACL support on FreeBSD
that was started here: 349ff15c7db09883abc20bdf3e8a4df6bff12cd3.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 20 Nov 2013 20:57:53 +0000 (14:57 -0600)]
Enforce MAX_PER_TREE by always _squish()ing in spit_to_shalist().
Previously bup would ignore MAX_PER_TREE whenever it hit a long run of
data that didn't produce a non-zero hashplit "level" (see the
discussion of fanout in DESIGN). That can happen, for example, when
traversing a file containing large zero regions (sparse or not).
As a result, bup could produce an arbitrarily large number of blobs at
level 0 of the hashsplit tree, causing it to consume increasing memory
during split/save, and to behave badly during join/restore.
To fix that, don't try to outsmart _squish() -- just call it every
time, and let it enforce MAX_PER_TREE when appropriate.
Thanks to trebor <robert.rebstock@tempelhof-projekt.de> for reporting
the problem a long while back, Yung-Chin Oei <yungchin@yungchin.nl>
for tracking down the cause and proposing a slightly different fix,
and Aidan Hobson Sayers <aidanphs@gmail.com> for suggesting this
particular approach.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 12 Nov 2013 17:38:35 +0000 (11:38 -0600)]
Don't crash; defer an error when a path changes while indexing it.
Mirror the error handling already used by save.
With this change the result for a path that changes or is deleted
between the drecurse traversal and the from_path() call should be the
same as the result for a path that changes or is deleted between index
and save.
Thanks to Yann Autissier <yann.autissier@gmail.com> for reporting the
problem and Gabriel Filion <gabster@lelutin.ca> for helping figure out
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Nov 2013 17:21:46 +0000 (11:21 -0600)]
_helpers.c: be smarter when converting unknown integer types to python.
For now assume that long long and unsigned long long are always the
the largest integer types, and convert unknown size integers to Python
via PyLong_FromUnsignedLongLong() or PyLong_FromLongLong(), depending
on the value's sign.
Thanks to Thomas Klausner <tk@giga.or.at> for reporting the problem
and helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Nov 2013 20:11:44 +0000 (14:11 -0600)]
Makefile: avoid using -Werror with clang for now.
Apparently the default compiler on OS X and FreeBSD is now clang, and
it doesn't like our CHECK_VALUE_FITS() definition. So for now, just
don't specify -Werror when we detect clang.
Thanks to Alexander Barton <alex@barton.de>, Thomas Klausner
<tk@giga.or.at>, Sebastian Schumb <sebastian@sebastians-site.de>, and
Zoran Zaric <zz@zoranzaric.de> for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Alexander Barton <alex@barton.de>
projects / bup.git / log