Luca Carlon [Thu, 21 May 2020 20:28:41 +0000 (22:28 +0200)]
git/midx: provide context managers for idx classes
Opening files and then mmap()ing them keeps the files open at the
filesystem level, and then they cannot be fully removed until the
fd is closed when giving up the mapping.
On most filesystems, the file still exists but is no longer visible
ion this case. However, at least on CIFS this results in the file
still being visible in the folder, but it can no longer be opened
again, or such. This leads to a crash in 'bup gc' because it wants
to re-evaluate the idx it just tried to delete.
Teach the PackIdx classes the context manager protocol so we can
easily unmap once they're no longer needed, and use that in bup gc
(for now only there).
For consistency, already add the context manager protocol also to
the midx, even if it's not strictly needed yet since bup gc won't
actually do this to an midx.
Signed-off-by: Luca Carlon <carlon.luca@gmail.com>
[add commit message based on error description, add midx part,
remove shatable to avoid live pointers into unmapped region] Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit e059fb80731c631d364f4c56e7202ed16e98ad84) Tested-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 16 May 2020 07:03:31 +0000 (09:03 +0200)]
save: close files immediately
Use a with statement to close all files immediately after
hashsplitting. There's also no need to have two except
clauses, so unify them to simplify this change.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 0c009932e7eae09badf847b951025f2bad4c271f)
[rlb@defaultvalue.org: adjusted for 0.30.x]
Johannes Berg [Sun, 17 May 2020 21:45:21 +0000 (23:45 +0200)]
helpers: use float for format_filesize()
In format_filesize(), we really do want float division,
in order to display the value correctly. For example, if
there's a file with 45200000 bytes, that should be shown
as 43.1 MB, not 43.0. Fix this by using proper float
division here, not int division.
Fixes: a5809723352c ("helpers: use // not / for division") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 3912ff3aa3ef75f79ccc6ec014c4a691d0e3803b)
Johannes Berg [Fri, 1 May 2020 20:16:39 +0000 (22:16 +0200)]
mincore: fix reading information
The _fmincore_chunk_size is typically set to 64MiB, which
makes sense to avoid doing very large mmap() operations
(to save already precious VM on 32-bit systems).
However, since that's in bytes, we cannot divide a size in
pages by it, and expect any useful outcome.
Calculate the number of chunks (chunk_count) properly based
on the size of the file, rather than its number of pages.
Otherwise, chunk_count typically ends up just 1 even for a
very large file (my test file was ~500MiB), and mincore()
is run just once, so we fill the presence information only
for the first 64MiB of the file, even if it was previously
completely in RAM.
Given a large enough test file (and enough RAM to keep it
there), the following should print about the same times
twice:
cat test > /dev/null ; \
time cat test > /dev/null ; \
bup split --noop test ; \
time cat test >/dev/null
Without the fix, it's evident that the file is evicted from
RAM almost entirely (apart from the first 64MiB) even in
this synthetic case.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit f41b3ab13ca1d740d30241692f6ec4e98c66cd27)
Wyatt Alt [Sun, 19 Apr 2020 21:46:52 +0000 (14:46 -0700)]
Fix bup-web error formatting when port unsupplied
Previously supplying the bup dir to bup web by mistake would result in a
python syntax error.
Signed-off-by: Wyatt Alt <wyatt.alt@gmail.com> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit d80eb198f4542489c5e318fc3322650adddb1931)
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: elaborate on rationale in commit message]
(cherry picked from commit 980e14a3bb6a9789764cb636b441c10bd91d99e6)
Johannes Berg [Sun, 19 Jan 2020 20:18:35 +0000 (21:18 +0100)]
git: create_commit_blob: allow timezones to be specified as 0
Checking "if adate_tz" means that if it's 0 (UTC) then we'll
actually use localtime, which is wrong. Do this only when it's
specified as None.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 4443772e4a6cbce5479554e881830c500d27b17c)
Johannes Berg [Sat, 18 Jan 2020 21:36:36 +0000 (22:36 +0100)]
git: add a test for not keeping midx files open
This test creates a few dummy idx files, generates an midx,
queries a PackIdxList for a non-existent object, unlinks the
midx and checks that we still have it open, but that we close
it at PackIdxList::refresh now.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 5ab20bb9d4bd403eb726fe6c5a7321a09ad2b21f) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 3 Jan 2020 12:53:20 +0000 (13:53 +0100)]
git: split out idx file writing to a separate class
Split the idx file writing into a separate class, to make that
kind of action available separately. This will be useful for the
next patch where we use it to test some idx/midx code.
In the future, it'll also be useful for encrypted repositories
since the idx format there will be useful for local caching to
take advantage of midx and bloom code as is, but the packwriter
will of course not be useful.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit bbeea3a3f5408d8752fa38a0da9d0de6d6aa30d8) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 18 Jan 2020 21:26:16 +0000 (22:26 +0100)]
git: fix PackIdxList keeping deleted files open
When an midx is deleted underneath bup, usually by itself running
'bup midx --auto', then PackIdxList may keep them open. This can
cause bup to run out of disk space easily since these files can
be fairly big, and can be recreated multiple times in a backup
run.
To fix this, remove any open PackMidx instances from the list and
close them explicitly.
Out of an abundance of caution, also explicitly close the bloom
instance if we have one - the same issue should apply here even if
I couldn't observe it, since the GC isn't guaranteed to clean up
the object immediately.
I remember debugging this issue years ago without coming to any
good conclusion, and it's been mentioned on the mailing list a few
times as well, e.g.
https://groups.google.com/d/msg/bup-list/AqIyv9n9WPE/-Wl2JVh5AQAJ
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 5c746e43600c059c52b5fd78212499e3e9700946) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Mon, 17 Feb 2020 20:02:11 +0000 (21:02 +0100)]
index: make --fake-valid match the man page
The index command currently clobbers the hash of a file when
marking it as valid, but the man page states:
--fake-valid
mark specified paths as up-to-date even if they aren't.
This can be useful for testing, or to avoid unnecessarily
backing up files that you know are boring.
The latter part ("avoid unnecessarily backing up [...]") cannot be
implemented with --fake-valid as is, because of the clobbering of
the hash: the fake invented hash will not exist in the repository,
and thus save checks and saves the file.
Fix this by clobbering the hash only if it's the invalid EMPTY_SHA.
Add a test for this to test-save-smaller, just because that's where
we discovered it.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit a79de5af5c6fcdf730a94ae8d1b839dfbbc2dc59) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Mon, 17 Feb 2020 19:03:22 +0000 (20:03 +0100)]
save: add test for --smaller, fix DESIGN document
Add a test for --smaller, in particular showing that the actual --smaller
behaviour doesn't match what's described in the DESIGN file, which says:
Another interesting trick is that you can skip backing up files even if
IX_HASHVALID *isn't* set, as long as you have that file's sha1 in the
repository. What that means is you've chosen not to backup the latest
version of that file; instead, your new backup set just contains the
most-recently-known valid version of that file. This is a good trick if you
want to do frequent backups of smallish files and infrequent backups of
large ones (as in 'bup save --smaller'). Each of your backups will be
"complete," in that they contain all the small files and the large ones, but
intermediate ones will just contain out-of-date copies of the large files.
This ("Each of your backups will be 'complete,' [...]") would seem to indicate
all files should be present, but in fact neither new nor old files are actually
saved by 'bup save --smaller'.
To avoid confusion, also update the DESIGN documentation here.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit fb74d94974aeea5a2c78d3e32afe2aaaaf0d6f61) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sun, 16 Feb 2020 21:13:03 +0000 (22:13 +0100)]
documentation: quote literal * in man pages
For "1024*1024" etc. we need to quote the * so that it
doesn't get used as bold markup, fix that.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 1d8470db4ce9bf62d7f8edef579d7590dac555d7)
Johannes Berg [Wed, 5 Feb 2020 19:36:46 +0000 (20:36 +0100)]
build: fix C-side dependencies
We need to depend on bupsplit.h, otherwise changes there
don't force a rebuild.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 82d951a8589b8bd5a3e61b4d0a7b70e8ef6f6231) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 23:17:38 +0000 (00:17 +0100)]
save/vfs: update comments wrt. tree/bupm ordering
After looking into this and thinking about it, the comments here
are a bit misleading - save states the entries must be in a given
order without a rationale, and vfs states that the order is wrong
but gives an explanation that's not quite right.
Update both comments to make this clearer, and to document that
there's no inherent reason, just happened to pick something when
the save code was written, which turned out to be not the best.
Johannes Berg [Sat, 25 Jan 2020 22:13:17 +0000 (23:13 +0100)]
tests: add test for save encountering duplicates
Add a test for save encountering duplicates in the index, both
for a file and a directory, which was fixed in the previous patch.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit bfc9e621ee5801fd597b3668a0a7f6747b7cba3a) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 22:41:56 +0000 (23:41 +0100)]
save: don't confuse metadata on duplicate files
If there are duplicate files, save removes the duplicates but
doesn't remove the duplicates from metadata, so the metadata
list gets messed up wrt. the file list. Fix this.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit bd0dc75dd161d544133cc80d4858bc8ef1511a7e)
[rlb@defaultvalue.org: adapt to 0.3x] Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:21:24 +0000 (22:21 +0100)]
test-save-errors: add tests for inaccessible metadata
Add test cases for when metadata for a file and directory cannot
be read (causing an IOError instead).
Note that this test fails if the previous two patches aren't
applied.
Note also that if such an error ever happened without the patch
to save, the repository is essentially corrupt and there no way
to figure out _which_ file didn't get metadata.
Note that this uses some python trickery to force an IOError in
metadata reading.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: name test in commit summary line] Tested-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 19b3b1b7acda5e507a4ec053cff6a2e139c24f31)
[rlb@defaultvalue.org: adjust bup-save test overrides for 0.3x] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:20:36 +0000 (22:20 +0100)]
vfs: deal with bupm containing no metadata for a file
If a file was for some reason saved without metadata, deal with
this and create an empty/unknown metadata for it instead.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit da552ea8d19d5da73c0d387d4963bc77841ab9ba) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:18:53 +0000 (22:18 +0100)]
save: add empty metadata if reading fails
If we cannot read a file's metadata, we must add an entry in the
repository nonetheless because the bupm contents must match the
tree contents.
Add a dummy (empty) metadata entry in this case, as is already
done when reading a directory's metadata.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 47891d8951a95b8e0d9ca94387107cdf12ca3d3c) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:20:09 +0000 (00:20 +0100)]
index: fix Entry comparison methods
Some of these methods are evidently not used today,
but they should have proper arguments anyway.
Fixes: 3ff7455dd474 ("Don't use cmp()") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit c7013fde92fc73e0064462954918b09541e9a84f) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 29 Jan 2020 20:40:31 +0000 (21:40 +0100)]
test-web: check that static/styles.css can be retrieved
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 188ccbafbceed4795fb7dddbce97d1acccca9242) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:23:18 +0000 (00:23 +0100)]
client: import socket
Fixes: 7ce8041f0345 ("Teach bup about URLs and non-ssh remotes") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 21fbd0c733fe2abdeddcf7a5aaf7bab85890de2c) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:56 +0000 (00:22 +0100)]
client: import atoi
Fixes: 22d01e1a8077 ("If you specified the port number on the command line, convert it to an int.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 041082521f8bdc46e6a36347a9f5183abbe4f272) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:33 +0000 (00:22 +0100)]
client: import DemuxConn
Fixes: fb3bd84cfd24 ("Add DemuxConn and `bup mux` for client-server") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 1ad93eca8d0e837263d9f29dd229f6835f452c04) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:19:33 +0000 (00:19 +0100)]
vfs: fix finish_extract()
The 'dir' variable doesn't exist here, must be 'meta' instead.
Fixes: 0962d3904735 ("Add initial support for metadata archives.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 9cbb51ec1423ef2e24f18f756585dcc39d8a54f2) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 18 Dec 2019 21:43:58 +0000 (22:43 +0100)]
get: remove extra src_repo
We already have a src_repo from the with statement, no need to
instantiate another one (that won't even be closed properly).
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit ab051b4620832ef8e2b4dd134b1b61241f53814e) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 2 Jan 2020 21:34:52 +0000 (22:34 +0100)]
tests: fuse: use stat instead of relying on "ls -l" output
On my Fedora 31 system, this currently fails because the test
expects the file mode bits to be printed as "-rw-r--r--" but
I get "-rw-r--r--." instead, perhaps due to selinux.
Fix this to use stat --format instead to extract the bits we
want to check here (user, group, mode, timestamp).
While at it, I noticed that this test has some confusion with
timezones and only worked because 'ls -l' doesn't show the
time in this case since the file is so old. Fix that by making
_everything_ here use UTC, instead of just some parts.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
(cherry picked from commit c38e02f0f5daa4ddf4d2c0a553d717d1f3ad02e5) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 13 Dec 2019 20:37:16 +0000 (21:37 +0100)]
man pages: fix some formatting
I noticed one place where -- was used instead of \--, leading to a
emdash instead of two hyphens, but then I looked and realized this
was not just a single place; fix all of those.
While at it, I saw some --keep options being referenced without
backticks, fix that too.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
(cherry picked from commit 27e6ee3680585fd445ce04955333a4bdf585db30) Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 11 Feb 2020 02:47:13 +0000 (20:47 -0600)]
git: overhaul git version checking
Rework the git version tetsting to handle versions like "1.5.2-rc3"
or (apparently) "1.5.2-rc3 (something ...)". Add tests for parsing of
all the version types in the current git tag history that we need to
support.
Support and document BUP_ASSUME_GIT_VERSION_IS_FINE=1 as an escape
hatch in case the parsing isn't sufficiently comprehensive, or
upstreeam changes their practices in future releases.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit c9227455fdbd6605c04075aa2ae0faad9aacbc62) Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 13 Dec 2019 20:36:04 +0000 (21:36 +0100)]
CatPipe: remove _ver_warned
This variable is no longer used, remove it entirely
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
(cherry picked from commit bcdc1c7ff1ba166aa1990f77d71bb410cae959c3) Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 12 Jan 2020 21:33:15 +0000 (15:33 -0600)]
bloom: fix logic controlling bloom regeneration
Add missing MAX_BLOOM_BITS index in the logic in bup bloom that
determines whether or not we should regenerate the filter. We never
noticed because:
$ python2
>>> 0 < {1 : 2}
True
$ python3
>>> 0 < {1 : 2}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'dict'
Also regnerate if the -k value differs from the existing filter's k.
Thanks to Johannes Berg for pointing out some nontrivial problems in
an earlier version.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit b86fb21ae670f8d1dc42d26bd69252cb9cc0ab7f) Tested-by: Rob Browning <rlb@defaultvalue.org>
The Python documentation [0] indicates that an 'O' passed to
Py_BuildValue will have its refcount incremented. Since some elements of
the tuple created in stat_struct_to_py are pre-converted in C to
PyObjects, they already have a refcount of 1 - use 'N' to avoid
incrementing it and ensure Python can deallocate them correctly.
Rob Browning [Sun, 20 Oct 2019 16:48:58 +0000 (11:48 -0500)]
vfs: rely on FakeLink's target in readlink
This could result in an error like:
File "/usr/local/lib/bup/bup/vfs.py", line 524, in _compute_item_size
return len(_readlink(repo, item.oid))
AttributeError: 'FakeLink' object has no attribute 'oid'
Thanks to Hartmut Krafft for reporting the problem and helping devise
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
(cherry picked from commit 023917c97cbf9bf6dbfad7120b41850dc5d7c7cd) Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 11 Sep 2019 02:54:15 +0000 (21:54 -0500)]
test-get: rm pax_global_header from git archive output
git-archive now adds a pax_global_header, and while GNU tar suppresses
it, some versions of tar don't, so remove it explicitly to avoid
spurious test failures.
Thanks to Greg Troxel for for reporting the problem and helping devise
the solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 21:02:18 +0000 (16:02 -0500)]
cirrus-ci: specify make -j3 on all platforms
Based on past experience, this should help -- locally it has appeared
to help our tests even with a greater concurrency level than the cpu
count, and it looks like all the current test hosts have at least two
cores.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 18:18:17 +0000 (13:18 -0500)]
prep-for-macos-build: get brew rsync
It looks like macos may ship with a more limited rsync (or at least
one that doesn't produce the --itemize-changes output we expect, so
just install brew's.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 24 Aug 2019 16:49:54 +0000 (11:49 -0500)]
fsck: only specify -t1 when par2 appears to support it
It looks like par2 may reject the "-t1" argument even when it's new
enough to support parallelism, so we can't rely on detecting the
version. Instead, before running the first real par2 command, test
-t1 in a sandbox to decide whether we can use it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 21 Aug 2019 06:23:30 +0000 (01:23 -0500)]
fsck: disable par2's internal concurrency
It looks like par2 added its own concurrency a bit back, and when
that's combined with bup's fsck -j parallelism, a simple
t/test-fsck.sh run completely swamps a machine here, launching what
looks like 32 threads total (for a 4/8 core machine), and fully
saturating the CPU.
The current test, which specifies -j99 actually ends up launching 4
par2 invocations, each processing a single file, and then each par2
appears to assume it has the entire machine to itself and launches 8
threads (one per hardware "core").
The resulting test takes 100s, but if we disable par2's parallelism
with -t1, putting bup's -j argument back in control of the overall
level of concurrency, the run time comes down to 4s.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 20 Aug 2019 01:34:11 +0000 (20:34 -0500)]
test-prune-older: disable background gc
The debian buildd's exposed a race between the default automatic
background gc and the explict gc in the test, resulting in:
git commit --date 1566025987 -qam 1566025987
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
git gc --aggressive
fatal: gc is already running on machine 'zandonai' pid 34323 (use --force if not)
Traceback (most recent call last):
File "t/test-prune-older", line 190, in <module>
save_utcs = create_older_random_saves(save_population, three_years_ago, now)
File "t/test-prune-older", line 74, in create_older_random_saves
exc(['git', 'gc', '--aggressive'])
File "t/test-prune-older", line 41, in exc
check_call(cmd, shell=shell)
File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'gc', '--aggressive']' returned non-zero exit status 128
Fix it by disabling gc.autoDetach in the git config for the test repo
-- something we may well need to do in other tests too.
Thanks to Robert Edmonds for reporting the problem and suggesting the
fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
If the watched process ends a push to the pipe without a newline at
the end, but with newlines in the middle, then sep_rx.split() will
return with multiple entries, the last of which will not end with a
newline and yet not be the empty string. This line prefix needs to be
stashed into the pending buffer, too.
This turns out to be exactly the same logic as if sep_rx.split had not
split the string, so eliminate one layer of conditionals.
This version incorporates feedback from Rob Browning to continue to
pass a list to extend().
Signed-off-by: Nathaniel Filardo <nwf20@cl.cam.ac.uk>
[rlb@defaultvalue.org: adjust commit summary and remove extra space in
"if split[0]" guard.] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Greg Troxel [Wed, 5 Jun 2019 20:55:32 +0000 (16:55 -0400)]
restore: create fifos with mkfifo, not mknod
I recently did a restore of a large bup backup, about 34G worth. All
worked well, including metadata, except that bup threw an exception on
restoring fifos (that I didn't need; they were in /var and were sockets
in use by daemons when the backup happened).
The problem was that mknod was being called for the fifo, and given only
two argumetns. mknod(2) on NetBSD says it takes three arguments.
mkfifo takes two. I am guessing that mknod in python calls mknod the OS
call, and on Linux somehow the third null argument works out somehow.
But it seems irregular to make a fifo with mknod.
I realize python is not POSIX, but mknod(2) requires three arguments:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/mknod.html
It would be nice to have a test of backing up and restoring a fifo; that
would have caught this.
The following patch makes my restore go smoothly.
Signed-off-by: Greg Troxel <gdt@lexort.com>
[rlb@defaultvalue.org: adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 13 Apr 2019 17:07:18 +0000 (12:07 -0500)]
rev_list: handle multiple results/ref from remote for custom formats
Previously a remote rev_list invocation would crash if a custom format
was provided and the remote produced multiple results for any input
ref. Change the client/server code to use a blank line to indicate
the end of the rev-list results. Of course, that means that the parse
function provided must be able to handle (consume) any blank lines
that its format string produces, which may preclude the use of some
format strings, but should be sufficient for now.
Adjust test-ls to (trivially) cover this case, and broaden the use of
the commit hash length tests in the code.
Thanks to Alex Roper for reporting the problem, providing an easy way
to reproduce it, and proposing a fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 12 Apr 2019 19:55:13 +0000 (14:55 -0500)]
Handle commit mergetags (at all)
Previously bup would just crash (i.e. bup ls) if it encountered a
commit with a mergetag header (apparently a new thing). For now,
adjust git.parse_commit to accept and ignore them as long as they only
appear as an optional, final header in the commit. That may or may
not turn out to be sufficient, and it does mean that for now we won't
be able to preserve mergetags (if we want to) whenever we rewrite
commits via bup gc, get, etc.
Rob Browning [Sun, 23 Mar 2014 17:41:06 +0000 (12:41 -0500)]
Add bup get; see the documentation for further information
WARNING: this is a new EXPERIMENTAL command that can (intentionally)
modify your data in destructive ways. Treat with caution.
Thanks to Karl Kiniger <karl.kiniger@med.ge.com> for helping track
down various bugs in earlier versions, and for noting that we might
want --verbose to be a little more effusive. And thanks to Patryck
Rouleau <prouleau72@gmail.com> for suggesting improvements to the
documentation.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 20:32:52 +0000 (15:32 -0500)]
repo: avoid cyclic dependency with is_remote method
The current VFS operations (like resolve()) require a repo object, but
we're about to add a VFS resolve() method to the repos. In and of
itself, that isn't necessarily a problem, but as an optimization, we
want the VFS resolve() to be able to detect when the repo it's been
given is a RemoteRepo and redirect the call to remote_repo.resolve().
Doing so pushes the one single resolve() call to the remote instead of
executing the resolve() locally with a lot of individual calls to the
remote_repo's other methods.
Adding is_remote() makes that possible without having to 'import repo'
in the VFS (repo already imports vfs).
Perhaps we'll rework it later, but this will do for now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 31 Mar 2018 19:14:43 +0000 (14:14 -0500)]
Move vfs resolve() tests to tresolve.py
Move resolve() tests from tvfs to tresolve, and the common tree_dict()
test code to a new test.vfs module, in preparation for more extensive
resolve() testing.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Jan 2019 22:22:14 +0000 (16:22 -0600)]
test_resolve_loop: ensure exception is actually thrown
Make sure to resolve the correct path, and ensure the call never
returns. Previously when the path was wrong, and it *was* wrong, the
test would appear to succeed even though it wasn't actually testing
the intended ELOOP case.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Oct 2018 23:04:27 +0000 (18:04 -0500)]
metadata: always add/store/retrieve size for links and normal files
This simplifies cases where we need to transmit Metadata
objects (i.e. bup-get's repo.resolve()), and it means that for trees
created using this new v3 format, retrieving the sizes of chunked
files should be notably less expensive, since they'll be directly
available in the directory's .bupm file.
Without that, we have to seek around in the chunked tree to find the
last byte (cf. vfs._normal_or_chunked_file_size).
Only store the sizes for symlinks and regular files (which might be
chunked) until it's clear that other st_sizes are useful.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 27 Feb 2018 07:32:54 +0000 (23:32 -0800)]
Replace lresolve with resolve(..., follow=False)
Although there's the NOFOLLOW precedent, this is really just to narrow
the API before we add it as a repo method, i.e. so we only have to
handle one function instead of two.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 9 Dec 2018 18:40:26 +0000 (12:40 -0600)]
vfs: change /save/latest back to a symlink to the latest save
The current, reworked vfs presents /somesave/latest as if it actually
is the latest commit. Change it back to a symlink to the latest save
to roughly match the previous behavior -- though now it's a link to
the save name, not to the (removed) /.commit/ subtree.
To restore the link, reintroduce the concept of a fake
symlink (i.e. one that has no corresponding blob in the repository).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 3 Dec 2018 18:58:03 +0000 (12:58 -0600)]
Don't return invalid data for offset reads (observed via fuse)
Fix a serious bug in the vfs that could cause it to return invalid
data for chunked file read()s that didn't start at the beginning of
the file. The issue was first observed via fuse, which makes sense
given that it streams a file in chunks that (currently) each come from
independent, increasing seek-offset FileReaders.
The previous dropwhile() invocation in the _tree_chunks generator,
used to skip past chunks that were completely before the offset, was
simple but wrong, and would skip too far. Replace it with
_skip_chunks_before_offset().
Add randomized tests of both simple streaming reads, and seek offset
reads, which catch the problem, cover additional cases, and should
prevent regressions.
Thanks to voldial for reporting the problem and providing an easy way
to reproduce it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Oct 2018 17:35:11 +0000 (12:35 -0500)]
update-doc-branches: add command to update man and html
Create a new command to update the man and html branches, and move the
related code there from the Makefile.
Update the branches based on the current (clean) tree, rather than
consulting the git origin, and rely on ls-files rather than globbing
so that the file lists will always be correct -- we'll immediately
notice deletions, avoid picking up stray files in the directory, etc.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 30 Sep 2018 20:29:42 +0000 (15:29 -0500)]
Makefile: fix find -printf issue on FreeBSD
Apparently the use of -printf was causing the error: "printf: missing
format character" with FreeBSD 11.1-RELEASE. Change the helpers lib
count to rely on -print0 and tr, which should be more portable, but
still be undisturbed by unusual paths.
Thanks to Curtis Dunham for reporting the problem and proposing an
alternate solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 7 Jul 2018 17:20:47 +0000 (12:20 -0500)]
vfs: flatten resolution cache key
This will require more storage if there are a lot of lookups with the
same parent and differing paths, but otherwise, without more
intentional structure sharing among paths, this should be better, and
we can always revisit the arrangement later.
Serializing the parent path segments should also make sure the same
parent (semantically-speaking) contributes the same hash to the key.
Previously, Metadata objects could prevent that, given their trivial,
pointer-based hashes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 2 Sep 2018 17:29:01 +0000 (12:29 -0500)]
Refuse to run by default if python version is not 2
Exit with an error if the python major version isn't 2, since we're
working on support for py3, and we'll probably reach a point before
we're finished where bup doesn't immediately crash with py3, but might
well do very bad things to the repository.
Allow overriding the check by setting
BUP_ALLOW_UNEXPECTED_PYTHON_VERSION=true so people can still test py3
if they like.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 09:57:18 +0000 (19:57 +1000)]
tindex: add 0o to current octal literal
Needed for python3.
This preserves current behaviour, but since I get no test failures if I
just chop of the leading 0 and make it a normal int there's a good
chance there's underlying breakage.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: adjust commit message summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Julien Goodwin [Thu, 30 Aug 2018 08:25:25 +0000 (18:25 +1000)]
Replace remaining print statements with print function
Also add the appropriate __future__ import statement.
Signed-off-by: Julien Goodwin <jgoodwin@studio442.com.au>
[rlb@defaultvalue.org: memtest-cmd.py: remove a few preexisting
trailing spaces that were carried over; adjust commit summary] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 6 Jul 2018 17:01:55 +0000 (12:01 -0500)]
vfs: include unique repo id in resolution cache key
...since resolve() currently requires a full parent path and the root
refs are only applicable to a particular repository.
Use differing integers to identify repositories that may be
independent (with respect to refs, tags, etc.), and use (typically
small) integers rather than the repo path/address so that they'll be
short if we want to embed them directly in cache keys later.
Use realpath() for local repositories in order to detect when the same
repository is reachable by multiple paths. (Something similar could
eventually be done for remotes.)
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log