Gabriel Filion [Sun, 8 Sep 2013 03:26:15 +0000 (23:26 -0400)]
Update bup-split(1); document -d, clarify the split "modes", and reorganize.
Update the synopsis, document the "-d DATE" option, and make a clearer
distinction between normal options and the options that select the
split "mode".
Signed-off-by: Gabriel Filion <gabster@lelutin.ca>
[rlb@defaultvalue.org: add dashes to seconds-since-epoch; adjust the
MODES text a bit and move -b down; make a few other edits and adjust
commit message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 14 Sep 2013 18:47:23 +0000 (13:47 -0500)]
Fix path ownership restoration problems on Cygwin.
It turns out that Cygwin won't allow you to chown() a path to an
unknown uid or gid, even when "root".
For now, make that a deferred error on Cygwin, rework the tests to
avoid it when possible, and disable the tests (on Cygwin) that require
it.
For the record, it appears that tar doesn't normally hit this problem
on Cygwin because it uses "geteuid() == 0" to detect super-user
status, which won't be true in the normal case, even if the user is an
administrator.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 16 Sep 2013 17:56:27 +0000 (12:56 -0500)]
Always fall back to socket()/bind() when os.mknod(...S_IFSOCK) fails.
Previously bup would use socket()/bind() instead of os.mknod(... |
stat.S_IFSOCK) on Cygwin, but this issue isn't Cygwin specific.
Remove the platform conditionalization, and fall back to
socket()/bind() any time mknod() fails with EINVAL.
Thanks to Robert Edmonds <edmonds@debian.org> for reporting the
relevant failure on a Debian kFreeBSD buildd.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 23 Aug 2013 03:57:03 +0000 (22:57 -0500)]
Stop interleaving stream and mmap IO operations when writing the index.
Previously bup would write the index via a simultaneous combination of
stream operations and mmap writes to non-overlapping, but adjacent
regions of the same file. This was causing index corruption in some
cases.
Instead, make an extra pass over the data in memory to precompute the
size of the final index section, which contains any 31+-bit offsets.
Then mmap and write the entire set of tables directly, avoiding the
need for simultaneous stream operations.
Reported-by: Alan Jenkins <alan.christopher.jenkins@gmail.com> Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Greg Troxel <gdt@lexort.com> Reviewed-by: Greg Troxel <gdt@lexort.com>
Christopher Meng [Sun, 25 Aug 2013 14:32:27 +0000 (10:32 -0400)]
Preserve filesystem timestamps during "make install".
A good idea regardless, and recommended by the Fedora and Debian
packaging guidelines.
From debian-policy:
The rationale is that there is some information conveyed by knowing
the age of the file, for example, you could recognize that some
documentation is very old by looking at the modification time, so it
would be nice if the modification time of the upstream source would
be preserved.
It should also help prevent unnecessary backup churn after upgrades.
Signed-off-by: Christopher Meng <cickumqt@gmail.com> Reviewed-by: Zoran Zaric <zz@zoranzaric.de> Tested-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: adjust commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Kumar Appaiah [Sun, 25 Aug 2013 19:02:18 +0000 (15:02 -0400)]
Add -l and --human-readable options to "bup ls".
When -l is specified, include the size in bytes for each item.
When --human-readable is also specified, print sizes like 3.5K, 1.8G,
etc., instead of the exact byte count.
Signed-Off-By: Kumar Appaiah <a.kumar@alumni.iitm.ac.in> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: adjust commit message; adjust newlines; squash
two of Kumar's patches into this one.] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Kumar Appaiah [Sun, 25 Aug 2013 19:02:16 +0000 (15:02 -0400)]
Add --human-readable option to "bup web".
When --human-readable is specified, print sizes like 3.5K, 1.8G,
etc., instead of the exact byte count.
Signed-Off-By: Kumar Appaiah <a.kumar@alumni.iitm.ac.in> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: adjust commit message; squash two of Kumar's
patches into this one.] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 25 Aug 2013 22:33:42 +0000 (17:33 -0500)]
save-cmd.py: don't write an irrelevant and incomplete .bupm fragment.
When finishing up (leaving) a directory and preparing to store its
related tree in the repository, don't write the pending .bupm (which
is still incomplete) to the pack if we know the directory already
exists in the repository (i.e. when we're not going to write it
either).
This problem was discovered when Zoran noticed that two consecutive
saves without an intervening index could produce a repository with an
unreferenced blob (according to git fsck).
Reported-by: Zoran Zaric <zz@zoranzaric.de> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 23 Aug 2013 17:23:17 +0000 (12:23 -0500)]
main.py: forward SIGTSTP/SIGCONT so "C-z" will actually suspend everything.
Catch and forward SIGTSTP (as SIGSTOP) and SIGCONT to the subprocess
as we already do for SIGTERM and SIGINT so that the subprocess will
also suspend/resume.
This still leaves bup with potentially unexpected behavior since the
(detached) subprocess will never see a SIGSTOP delivered to the parent
(because SIGSTOP can't be intercepted and forwarded). This is due to
the os.setsid() call that was originally introduced to support current
newliner arrangement (cf. b7a524ccb662c9ed3ebd786da0f45f459929ef45).
Thanks to Kalle for the report.
Reported-by: krichter722@aol.de Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Zoran Zaric [Mon, 12 Aug 2013 14:20:12 +0000 (16:20 +0200)]
git.py: don't automatically initialize ~/.bup if it doesn't exist.
bup had a convenience feature where commands would automagically
initialize a repo in ~/.bup if it didn't exist and no other BUP_DIR
was given.
This had the odd effect that when one forgot to specify BUP_DIR, a bup
repo would be initialized in ~/.bup even though only a browsing
command was used.
This patch drops that behaviour. Now all repositories must be
explicitly intiialized via "bup init".
Signed-off-by: Zoran Zaric <zz@zoranzaric.de>
[rlb@defaultvalue.org: edit commit message; fix test ("set +e" during
init run).] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
web-cmd.py: add a tornado import guard like the one for fuse in fuse-cmd.py.
In Debian, we replace the embedded copy of tornado with a Recommends on
the python-tornado package. Print a nice error message instead of a
backtrace if the user doesn't have it installed.
Signed-off-by: Robert S. Edmonds <edmonds@debian.org>
[rlb@defaultvalue.org: edited commit and error message] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 4 Aug 2013 18:39:55 +0000 (13:39 -0500)]
t/compare-trees: check rsync capability support correctly.
Handle older versions of rsync (where nothing was printed), and for
newer versions, check for "no FOO" rather than "FOO", since FOO will
always be listed one way or the other.
Signed-off-by: Gonéri Le Bouder <goneri@rulezlan.org>
[rlb@defaultvalue.org: adjust indentation, comments, and message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 31 Jul 2013 16:32:17 +0000 (11:32 -0500)]
_create_via_common_rec: treat rmdir() EEXIST like ENOTEMPTY.
See rmdir(2):
ENOTEMPTY
pathname contains entries other than . and .. ; or,
pathname has .. as its final component. POSIX.1-2001
also allows EEXIST for this condition.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 4 Jul 2013 19:12:22 +0000 (14:12 -0500)]
config/configure: add and use bup_find_prog; remove unused MF_PATH_INCLUDEs.
Pull all the common program search code into bup_find_prog() and use
it everywhere. This also makes configure's output a bit more
consistent in style/content.
Make it obvious that at the moment bup doesn't pay any attention to a
PYTHON or GIT environment variable value. i.e. bup currently ignores a
setting like PYTHON=python2.7.
Bup also doesn't use MF_PATH_INCLUDE settings yet, so remove them.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 8 Jun 2013 15:25:33 +0000 (10:25 -0500)]
git.py: flush idx_map before close so FILE* operations will see changes.
Flush idx_map (msync()) before closing it, since it doesn't look like
POSIX guarantees that a matching FILE* (i.e. idx_f) will see the
parallel changes if we don't.
From the original report:
After `bup save`, `git fsck` would show messages like
"Packfile index for %s SHA1 mismatch"
This indicated a bad trailing checksum on the pack index file.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reported-by: Alan Jenkins <alan.christopher.jenkins@gmail.com> Reviewed-by: Alan Jenkins <alan.christopher.jenkins@gmail.com> Tested-by: Alan Jenkins <alan.christopher.jenkins@gmail.com>
Rob Browning [Sun, 26 May 2013 17:16:40 +0000 (12:16 -0500)]
Remove lib.sh actually-root in favor of t/root-status.
Create a t/root-status command that outputs root, fake, or none and
that uses the same cross-platform logic as helpers.py is_superuser().
Use it everywhere.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 26 May 2013 16:05:56 +0000 (11:05 -0500)]
metadata.py: use socket() instead of mknod(...IF_SOCK) on Cygwin.
Apparently os.mknod(...IF_SOCK) fails on Cygwin, but our t/mksock tool
(which uses socket()) works just fine, so use that in
_create_via_common_rec() when on Cygwin.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Catch errors when restoring Linux attrs into xfs or btrfs
Both xfs and btrfs turn out to return EOPNOTSUPP, rather than ENOTTY,
when we try to apply Linux attrs that they do not support. This patch
improves matters in that it ensures we can restore into these file
systems without crashing, but does not quite address all issues yet:
- if any of the attributes we try to apply is not supported, none of
them will be applied to the file at all
- the user will typically drown in error output; eg. for a backup
originating from an ext4 file system, we can expect endless error
output relating to the extents-flag not being supported on xfs/btrfs
Signed-off-by: Yung-Chin Oei <yungchin@yungchin.nl> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Gabriel Filion [Tue, 14 May 2013 06:53:05 +0000 (02:53 -0400)]
Documentation: replace ~/.bup with $BUP_DIR in bup-index.md and bup-save.md.
When one is not using the default bupdir placement, then those files
aren't found inside ~/.bup, but wherever "-d /path.." or $BUP_DIR are
pointing to.
Thanks to krichter722@aol.de for mentioning this.
Signed-off-by: Gabriel Filion <gabster@lelutin.ca> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Fix a timing issue that often caused the test to fail -- the (removed)
file "foo" was marked as invalid in the index when it shouldn't have
been because it was modified less than tmax before the index run.
Insert a "bup tick" just before "bup index" so that foo won't be
marked invalid by the second run.
Don't expect the second "bup save" to fail since foo will now be valid
in the index (and hence ignored), even though it is no longer
available. The removal will be noticed by the next index+save.
When removing directories, the outcome is different: since metadata is
saved per directory, "bup save" notices that the directory is missing
and logs an "errno 2" error, but does not crash, which is what the
tested patch is concerned with. A third index+save run completes
without error.
This also means that without a "bup tick" before an index run, calls,
"bup save" might exit with non-zero exitcode when files/directories
are removed after bup index, but bup is not guaranteed to notice
missing files.
Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Add tests for file and dir removal between index and save.
Make sure the removals don't cause bup save to immediately abort.
Signed-off-by: Tim Riemenschneider <git@tim-riemenschneider.de>
[rlb@defaultvalue.org: move tests to their own file.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Damien Robert [Tue, 20 Nov 2012 20:52:45 +0000 (21:52 +0100)]
save-cmd.py: don't crash when a path disappears between index and save.
Protect calls to metadata.from_path() with a try/catch and defer
errors via add_error() instead of just crashing.
Signed-off-by: Damien Robert <damien.olivier.robert@gmail.com>
[git@tim-riemenschneider.de: rebase to current tmp/pending/meta: since
metadata is (now) stored in the index, only 1 of 2 hunks still
applies (race condition between reading the file for its content and
stat-ing it for metadata).]
[rlb@defaultvalue.org: edit commit message; squash file and dir
removal fixes into this commit; limit the scope of the try/catch to
the from_path() call, and put the remaining code in an else clause.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 18 Aug 2012 20:35:23 +0000 (15:35 -0500)]
Add support for "bup restore --exclude-rx <pattern> ...".
When --exclude-rx <pattern> is provided to bup restore, don't restore
any path matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html). The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards". To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".
You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.
Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Rob Browning [Sat, 21 Jul 2012 20:09:47 +0000 (15:09 -0500)]
Add support for "bup index --exclude-rx <pattern> ...".
When --exclude-rx <pattern> is provided to bup index, exclude any path
matching <pattern>, which must be a Python regular expression
(http://docs.python.org/library/re.html). The pattern will be
compared against the full path, without anchoring, so "x/y" will match
"ox/yard" or "box/yards". To exclude the contents of /tmp, but not
the directory itself, use "^/tmp/.".
You may check the behavior at runtime by setting BUP_DEBUG=2 in the
environment.
Thanks to Zoran Zaric <zz@zoranzaric.de> for reporting a bug in an
earlier version of this patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Zoran Zaric <zz@zoranzaric.de>
Thomas Haller [Sun, 3 Mar 2013 19:31:57 +0000 (20:31 +0100)]
wvtest.py: add a fallback definition of relpath() for Python < 2.6.
Since bup still supports Python 2.5 and relpath was added to Python in
2.6, add an os.path.relpath replacement, adapted from CPython source
(tag v2.6, file Lib/posixpath.py, hg-commit 95fff5a6a276).
Signed-off-by: Thomas Haller <thom311@gmail.com>
[rlb@defaultvalue.org: update LICENSE and add comment to code about
updating LICENSE.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Zak Wilcox <iwilcox@iwilcox.me.uk>
[rlb@defaultvalue.org: change original --ignore-dev to --no-check-device to match tar.
Adjust code to work with current master.
Remove tests -- will be reintroduced shortly.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Gabriel Filion [Sun, 17 Mar 2013 22:24:08 +0000 (18:24 -0400)]
Ignore artefacts from sampledata
9b5ae5f53e introduced the creation of some symlinks to avoid issues on
Cygwin. "make" now leaves the repository in a dirty state since we
forgot to exclude those symlinks.
The symlinks in question are products of the build/test process so we
need to avoid commiting them.
Signed-off-by: Gabriel Filion <gabster@lelutin.ca> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Jakob Matthes [Sun, 10 Mar 2013 18:10:38 +0000 (19:10 +0100)]
bup drecurse/index: allow non-directory --exclude and --exclude-from paths.
Previously bup only allowed directory exclusions.
Signed-off-by: Jakob Matthes <jakob.matthes@gmail.com> Reviewed-by: Gabriel Filion <lelutin@gmail.com>
[rlb@defaultvalue.org: change test file from g to j to avoid conflict
with subsequent "exclude-from" test; add WVPASS to mkdir so problem
is evident; edit commit message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Mark J Hewitt [Mon, 1 Oct 2012 16:17:28 +0000 (17:17 +0100)]
Use the correct C types when converting between C and Python in _helpers.c.
Previously, bup might fail when handling large repositories. For
example:
Traceback (most recent call last):
File "/usr/lib/bup/cmd/bup-midx", line 258, in <module>
do_midx_dir(path)
File "/usr/lib/bup/cmd/bup-midx", line 199, in do_midx_dir
all = list(do_midx_group(path, part1)) + part2
File "/usr/lib/bup/cmd/bup-midx", line 216, in do_midx_group
rv = _do_midx(path, None, sublist, gprefix)
File "/usr/lib/bup/cmd/bup-midx", line 123, in _do_midx
count = merge_into(fmap, bits, total, inp)
OverflowError: size does not fit in an int
['/usr/bin/bup', 'midx', '--auto', '--dir',
'/backup/bup/repository/objects/pack']: returned 1
Signed-off-by: Mark J Hewitt <m.hewitt@computer.org>
[rlb@defaultvalue.org: Remove vestigal header includes.
Remove Py_ssize_t definition for Python < 2.5.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
save-cmd.py: insist on having an index; check for the primary index file.
Insist on having an index, and report an error (with a suggestion to
run "bup index") if one isn't found.
Previously (at least as recently as 0.25-rc1), bup save would just
create an empty tree if there was no index. The metadata work
unintentionally broke that behavior.
For now, treat a missing index as an error under the assumption that
it's not likely to be intentional. If a good use case for the
previous behavior is presented, we can always restore it.
Signed-off-by: Jean-Baptiste Denis <jbd@jbdenis.net>
[rlb@defaultvalue.org: adjust commit message and save-cmd.py error message.] Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 17 Feb 2013 16:33:08 +0000 (10:33 -0600)]
Normalize stat(2) timespecs in _helpers.c.
Previously, the code required timespecs to have a nanosecond value
between 0 and 999999999, because that's what Linux appeared to
produce, but other platforms don't (i.e. Cygwin) -- so normalize the
system values to match expectations.
In particular, look for negative nanosecond values (must be greater
than -10**9), and if found, rewrite the timespec with non-negative
nanoseconds (still less than 10**9).
Thanks to Will Rouesnel <w.rouesnel@gmail.com> for reporting the
problem and posting an initial patch.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Ryan Brown [Thu, 7 Mar 2013 02:48:23 +0000 (21:48 -0500)]
Add a force-delete function to t/test.sh
Replace "rm -rf" with force-delete because "rm -rf" will proceed even
if it doesn't succeed. This masked a problem (reported on Fedora)
where the root directory and other system directories had "r-xr-xr-x"
(0555) permissions, causing bup restore to create directories that "rm
-rf" couldn't remove. As a result, subsequent tests would fail.
Signed-off-by: Ryan Brown <ryansb@csh.rit.edu>
[rlb@defaultvalue.org: edited commit message.] Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 3 Mar 2013 22:51:14 +0000 (16:51 -0600)]
Fix use of Python assert() with respect to optimization.
Fix a number of places where bup's assertions had material
side-effects, or where other code expected to see the AssertionError,
neither of which happen when optimization is enabled.
Reported-by: Jon Dowland <jmtd@debian.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log