Rob Browning [Sun, 25 Dec 2016 17:44:56 +0000 (11:44 -0600)]
index: only collect metadata for stale paths
Stop collecting all of the metadata for every path, even if the path
hasn't changed since the last save. Further, rework the code to
short-circuit some other unnecessary work.
To support this, split Entry.from_stat() into two parts, stale(), which
tests to see if an entry has "materially" changed, and
update_from_stat() which updates the entry to match the stat information
provided.
This should substantially decrease the indexing cost for paths that
haven't changed since the last save.
While we're here, rename hashgen to fake_hash so it's clearer that's its
only purpose.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 17 Dec 2016 20:40:02 +0000 (14:40 -0600)]
git.cp(): don't repeatedly recompute default repo
Before, bup would end up calling repo() for every invocation of cp().
Given that constructions like cp().get(id) are common, this could cause
a lot of unecessary calls, given that we already have git.repodir, which
will have the relevant value once initialized.
Discovered after noticing "bup restore" was hammering
"$BUP_DIR"/.git (which didn't exist).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 10 Dec 2016 17:57:05 +0000 (11:57 -0600)]
gc: restart catpipe after each new pack
Without this, the space taken by the stale packfiles that gc removes
won't be available to the filesystem until the entire gc run ends (or
the catpipe is otherwise reset).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 15 Dec 2016 18:45:19 +0000 (12:45 -0600)]
test-prune-older: avoid duplicate save times
When creating the list of random save times make sure to avoid any
duplicates since the timestamp is also the trivial commit content.
Otherwise the git commit for the second save dies with "nothing to
commit, working tree clean".
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 30 Oct 2016 18:31:43 +0000 (13:31 -0500)]
Add bup prune-older command
prune-older removes (permanently deletes) all saves except those
preserved by various temporal keep arguments. It is equivalent to a
suitable "bup rm" invocation followed by "bup gc".
For example, this invocation keeps all saves on the BRANCH for the past
month, and any older monthlies for the past year, and deletes the
remainder:
Ben Kelly [Thu, 10 Nov 2016 15:01:08 +0000 (10:01 -0500)]
walk_object: ignore blob content if not requested
Previously, walk_object would read the contents of each object it
encountered, even if it subsequently threw away those contents.
With this change, it will skip reading the contents of blobs unless
the caller explicitly asked for blob contents. In particular, this
means operations that traverse the object graph without caring
about object contents, such as `bup gc`, now only need to read the
contents of tree and commit objects.
Tested: `make test`; local run against copy of production repo. Signed-off-by: Ben Kelly <btk@google.com>
[rlb@defaultvalue.org: shorten commit summary; adjust indentation;
adjust docstring and comment; test "not include_data" first; change to
type='blob']
Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 5 Nov 2016 19:36:20 +0000 (14:36 -0500)]
test-gc: sort ls output defensively before comm
Explicitly sort the output of ls before passing it to comm in order to
accommodate some systems that apparently used to change the default sort
order. The order must match LC_COLLATE for comm to work.
While we're here, improve error checking around the comm invocations.
Thanks to Nick Alcock for reporting the original problem, and proposing
an initial fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 10 Sep 2016 17:52:59 +0000 (12:52 -0500)]
Report unknown root status if getwindowsversion > 5
Apparently the situation is much more complicated in newer versions, so
until someone more knowledgable can help, don't try to determine
superuser status there.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 7 Sep 2016 04:34:24 +0000 (23:34 -0500)]
Handle ntfs-3g EINVAL for attr save/restore
It appears that ntfs-3g now returns EINVAL for unsupported ioctls
including FS_IOC_GET_FLAGS and FS_IOC_SET_FLAGS.
Treat EINVAL like the existing expected errors for these calls, but
since there's evidence to suggest that other devices/filesystems use
EINVAL to indicate more serious trouble and use ENOTTY to indicate
unsupported ioctls, include a request to report any occurrences of
EINVAL when ntfs-3g is not involved.
Thanks to Mark J Hewitt for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 12 Jun 2016 20:50:46 +0000 (15:50 -0500)]
gc: explicitly handle missing objects
Instead of just dying with a KeyError backtrace, detect missing objects
and print a friendlier message about the problem. Having the new
MissingObject exception derive from KeyError should avoid affecting
existing code.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Jonathan Wright [Tue, 28 Jun 2016 23:22:41 +0000 (16:22 -0700)]
Fix fsyncing on OSX on SMB file-systems
Fixes the error:
Traceback (most recent call last):
File "/usr/local/Cellar/bup/0.28.1/lib/bup/cmd/bup-server", line 209, in <module>
cmd(conn, rest)
File "/usr/local/Cellar/bup/0.28.1/lib/bup/cmd/bup-server", line 91, in receive_objects_v2
fullpath = w.close(run_midx=not dumb_server_mode)
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/git.py", line 790, in close
return self._end(run_midx=run_midx)
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/git.py", line 764, in _end
fdatasync(f.fileno())
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/helpers.py", line 47, in <lambda>
fdatasync = lambda fd : fcntl.fcntl(fd, fcntl.F_FULLFSYNC)
IOError: [Errno 45] Operation not supported
Traceback (most recent call last):
File "/usr/local/Cellar/bup/0.28.1/lib/bup/cmd/bup-save", line 460, in <module>
w.close() # must close before we can update the ref
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/client.py", line 317, in close
id = self._end()
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/client.py", line 314, in _end
return self.suggest_packs() # Returns last idx received
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/client.py", line 229, in _suggest_packs
self.check_ok()
File "/usr/local/Cellar/bup/0.28.1/lib/bup/bup/client.py", line 134, in check_ok
% rv)
Signed-off-by: Jonathan Wright <quaggy@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 22 May 2016 18:49:17 +0000 (13:49 -0500)]
Check save/split names early via valid_save_name
Previously, split and save would just wait to see if the final
update_ref failed. Instead, check the validity of the name early,
before doing any significant work.
Thanks to vi0oss for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 11 Jun 2016 17:59:38 +0000 (12:59 -0500)]
test-web: use path with unique unicode sequence
Change the test path from "éxcitement" to "¡excitement!" since the
former has multiple valid unicode encodings (e.g. given combining
characters, etc.).
With the previous string, the test might fail if something along the way
normalized it (which was the case on at least some versions of OS X).
Thanks to ilovezfs for reporting the problem and helping with the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 27 May 2016 00:30:40 +0000 (19:30 -0500)]
Fix release archives and add tests
Previously, an archive created by "git archive" would fail to build,
because the versioning code was broken (in part by 03d35993b87f49753542e705e37949a46964be17) for any build outside a git
clone. Although "bup version" may not really have been quite right
since the switch from "bup-VERSION" to "VERSION release tags.
Fix the problem by reintroducing the content of the older
_version.py.pre as a permanent _release.py file, which (as with the
previous incarnation) is a template that's expanded during "git archive"
via export-subst (cf. gitattributes(5)).
Add a version module as the public interface to versioning, and have it
sort out whether or not the current tree is an official release. Adjust
configure-version to manage _checkout.py instead of _version.py (drop
_version.py), so that the version module is effectively just
disambiguating between _release.py and _checkout.py.
Adjust "bup version" to use the new version module and to handle
unprefixed VERSION tags.
Finally, add test-release-archive.sh, invoked via "make distcheck", so
we (hopefully) don't unknowingly break this again.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 28 May 2016 14:37:09 +0000 (09:37 -0500)]
Add config.vars.sh for bup_make and bup_python
During configure create a config/config.vars.sh file including (for now)
properly shell escaped bup_make and bup_python definitions so that we
can use them in other schell scripts.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 24 May 2016 00:15:30 +0000 (19:15 -0500)]
Compute exact sizes via t/data-size in test-gc.sh
Add a t/data-size helper that computes the total data size of all named
paths and use it instead of du in test-gc.sh to determine the
before/after repository sizes.
This should fix spurious test failures on filesystems with a large block
size. (Observed on ppc64el and powerpc tmpfs filesystems.)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Apr 2016 16:58:04 +0000 (11:58 -0500)]
restore: fix --sparse corruption
Rework the restore --sparse implementation to fix a bug that might cause
it to produce output that didn't match the original input.
Rewrite the code to probe for sparse regions in min_sparse_len jumps,
searching backward from the jump target for a non-zero byte. If one is
found, move just past that byte and try again.
This should be substantially more efficient than the previous approach
for non-sparse regions. In the limiting case, it should read roughly
1/min_sparse_len bytes instead of all of them.
The original bug was masked by the fact that the test that would have
revealed it wasn't actually generating random data (across trials),
something that was fixed in "Use $RANDOM seed for --sparse random
tests".
Thanks to Marcus Schopen for reporting the problem and Frank Gevaerts
for helping track down the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Apr 2016 22:52:02 +0000 (17:52 -0500)]
restore: add generative --sparse testing
Add t/sparse-test-data to automatically generate cases that might trip
up "restore --sparse", and use it in test-sparse-files.sh. This change
causes the current --sparse code to fail if you run enough trials,
but not as quickly as the existing "bup random" based test.
As compared to the existing "bup random" test, this one should evaluate
larger sparse regions much more often.
Thanks to Frank Gevaerts for help evaluating the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 24 Apr 2016 22:07:39 +0000 (17:07 -0500)]
Use $RANDOM seed for --sparse random tests
Since bup random defaults to a fixed random seed of 1, the restore
--sparse test for random data was actually testing the same data every
time. Specify a $RANDOM --seed to fix that.
After this change, the affected test fails fairly often.
Thanks to Frank Gevaerts for help evaluating the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
stevelr [Sun, 7 Feb 2016 02:05:51 +0000 (18:05 -0800)]
Add add_error imports missed in import * removal
[rlb@defaultvalue.org: rebased on current master; squashed related
patches; adjusted commit message] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 27 Mar 2016 00:33:57 +0000 (19:33 -0500)]
Check saved_errors before and after every test
Add a new buptest module that includes a no_lingering_errors context
manager which provokes a wvtest failure if there are any saved_errors
before or after a test, and then clears them, and use that manager in
all of the Python tests so that unexpected errors will be detected
immediately, instead of causing an unrelated failure later, when testing
for expected saved_errors.
Add a temp_testdir context manager to replace all of the current
tmpdir-related boilerplate.
Rework all of the tests to use the two managers where appropriate, so
that this:
def test_bloom():
with no_lingering_errors(), test_tempdir('bup-tbloom-') as tmpdir:
...
While making the changes, clean up the imports to match the PEP 8 import
recommendations a bit more closely (excepting the suggestion to avoid
"import bar, baz", which seems odd given the blessing of "from foo
import bar, baz", where bar and baz could still be modules).
Thanks to Tadej Janež for reporting the initial issue.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 26 Mar 2016 23:40:57 +0000 (18:40 -0500)]
test_server_split_with_indexes: close packwriter
Otherwise there are random test failures when the tmpdir is removed
before the PackWriter's destructor calls close(). This doesn't cause an
immediate problem, but it adds an unexpected entry to saved_errors that
causes later tests to fail.
The related saved_errors entries looked like this:
['../../../bup', 'midx', '--auto', '--dir', ...]: returned 15
and the underlying (hidden) midx stderr output in that case looked like
this:
error: '...' is not a bup repository; run "bup init"
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 28 Feb 2016 16:21:57 +0000 (10:21 -0600)]
fuse: remove redundant custom Stat class
It looks like our Stat class was just reinitializing all of the things
that the built in fuse.Stat class already initializes, so just rely on
theirs, and use their keyword initializers to avoid a few more
reinitializations.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Mar 2016 17:45:08 +0000 (11:45 -0600)]
web: fix broken GETs and test
It appears that content GETs, i.e. "curl
http://localhost/foo/latest/bar" may have been broken for a while. To
addresss that, switch to the apparently preferred upstream coroutine
approach, and add some initial tests.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 10 Jan 2016 19:44:04 +0000 (13:44 -0600)]
Never omit explicitly named mounts during index -x
Assuming mounts like this:
/
/usr
and an index invocation like "bup index -x / /usr", don't skip indexing
/usr. Previously bup would skip /usr when -x was specified because
reduce_paths() would drop /usr from the list of paths to index, assuming
that /usr would be covered when indexing /. However, when /usr was
actually reached, -x would cause it to be skipped.
To fix that, exempt all mount points mentioned on the command line from
-x during filesystem traversal, expand the (root only) tests to cover
this situation, and check some corner cases.
Thanks to Wayne Scott for reporting the problem, and Yves-Alexis Perez
for a reminder.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 5 Mar 2016 17:21:50 +0000 (11:21 -0600)]
index: raise children_n limit to struct 'I' max
This assertion appears to be a sanity check, and one that's perhaps too
low (think giant Maildirs), so raise it to the real limit, which should
be UINT_MAX, given that the index header containing it is
encoded/decoded via 'I' (platform native unsigned int):
https://docs.python.org/2/library/struct.html
Since Python doesn't appear to provide access to UINT_MAX, add it (and
INT_MAX) to _helpers.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log