bup.git
11 days agoindex: fix -H option master
Johannes Berg [Wed, 13 May 2020 20:22:04 +0000 (22:22 +0200)]
index: fix -H option

hexlify(ent) doesn't work, that needs to be ent.sha. Fix it.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 weeks agogit/midx: provide context managers for idx classes
Luca Carlon [Thu, 21 May 2020 20:28:41 +0000 (22:28 +0200)]
git/midx: provide context managers for idx classes

Opening files and then mmap()ing them keeps the files open at the
filesystem level, and then they cannot be fully removed until the
fd is closed when giving up the mapping.

On most filesystems, the file still exists but is no longer visible
ion this case. However, at least on CIFS this results in the file
still being visible in the folder, but it can no longer be opened
again, or such. This leads to a crash in 'bup gc' because it wants
to re-evaluate the idx it just tried to delete.

Teach the PackIdx classes the context manager protocol so we can
easily unmap once they're no longer needed, and use that in bup gc
(for now only there).

For consistency, already add the context manager protocol also to
the midx, even if it's not strictly needed yet since bup gc won't
actually do this to an midx.

Signed-off-by: Luca Carlon <carlon.luca@gmail.com>
[add commit message based on error description, add midx part,
remove shatable to avoid live pointers into unmapped region]
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 weeks agosave: close files immediately
Johannes Berg [Sat, 16 May 2020 07:03:31 +0000 (09:03 +0200)]
save: close files immediately

Use a with statement to close all files immediately after
hashsplitting. There's also no need to have two except
clauses, so unify them to simplify this change.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
2 weeks agohelpers: use float for format_filesize()
Johannes Berg [Sun, 17 May 2020 21:45:21 +0000 (23:45 +0200)]
helpers: use float for format_filesize()

In format_filesize(), we really do want float division,
in order to display the value correctly. For example, if
there's a file with 45200000 bytes, that should be shown
as 43.1 MB, not 43.0. Fix this by using proper float
division here, not int division.

Fixes: a5809723352c ("helpers: use // not / for division")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 weeks agomincore: fix reading information
Johannes Berg [Fri, 1 May 2020 20:16:39 +0000 (22:16 +0200)]
mincore: fix reading information

The _fmincore_chunk_size is typically set to 64MiB, which
makes sense to avoid doing very large mmap() operations
(to save already precious VM on 32-bit systems).

However, since that's in bytes, we cannot divide a size in
pages by it, and expect any useful outcome.

Calculate the number of chunks (chunk_count) properly based
on the size of the file, rather than its number of pages.
Otherwise, chunk_count typically ends up just 1 even for a
very large file (my test file was ~500MiB), and mincore()
is run just once, so we fill the presence information only
for the first 64MiB of the file, even if it was previously
completely in RAM.

Given a large enough test file (and enough RAM to keep it
there), the following should print about the same times
twice:

  cat test > /dev/null ; \
  time cat test > /dev/null ; \
  bup split --noop test ; \
  time cat test >/dev/null

Without the fix, it's evident that the file is evicted from
RAM almost entirely (apart from the first 64MiB) even in
this synthetic case.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 weeks agoFix bup-web error formatting when port unsupplied
Wyatt Alt [Sun, 19 Apr 2020 21:46:52 +0000 (14:46 -0700)]
Fix bup-web error formatting when port unsupplied

Previously supplying the bup dir to bup web by mistake would result in a
python syntax error.

Signed-off-by: Wyatt Alt <wyatt.alt@gmail.com>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 weeks agogit: create_commit_blob: allow timezones to be specified as 0
Johannes Berg [Sun, 19 Jan 2020 20:18:35 +0000 (21:18 +0100)]
git: create_commit_blob: allow timezones to be specified as 0

Checking "if adate_tz" means that if it's 0 (UTC) then we'll
actually use localtime, which is wrong. Do this only when it's
specified as None.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 weeks agoFall back to calloc when __builtin_mul_overflow isn't available
Rob Browning [Sun, 19 Apr 2020 20:47:25 +0000 (15:47 -0500)]
Fall back to calloc when __builtin_mul_overflow isn't available

Check for __builtin_mul_overflow at configure time, and fall back to
calloc when it's not found.

Thanks to Greg Troxel for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agocirrus: move to FreeBSD 12-1
Johannes Berg [Sat, 18 Apr 2020 21:59:25 +0000 (23:59 +0200)]
cirrus: move to FreeBSD 12-1

Since there appears to be trouble with the image we were using.  After
testing a few others, 12-1 appears to be OK.

cf. https://github.com/cirruslabs/cirrus-ci-docs/issues/625

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: elaborate on rationale in commit message]

6 weeks agovfs: use None for unknown uid/gid
Johannes Berg [Sat, 8 Feb 2020 21:54:56 +0000 (22:54 +0100)]
vfs: use None for unknown uid/gid

This means we show '?' instead of 0 for unknown UIDs when
numeric output is requested, as it was before.

This also uncovered a forgotten bytes annotation for the
"unknown" string ('?' should be b'?').

Somehow, this new behaviour (of printing 0 instead of ?)
also got quite enshrined in the test suite, fix that too.

And finally, on python 2, fuse doesn't accept None in the
stat struct (but does on python 3, go figure).

Fixes: f76c37383ddb ("Remove vfs (replaced by vfs2)")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agogit: add a test for not keeping midx files open
Johannes Berg [Sat, 18 Jan 2020 21:36:36 +0000 (22:36 +0100)]
git: add a test for not keeping midx files open

This test creates a few dummy idx files, generates an midx,
queries a PackIdxList for a non-existent object, unlinks the
midx and checks that we still have it open, but that we close
it at PackIdxList::refresh now.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agogit: split out idx file writing to a separate class
Johannes Berg [Fri, 3 Jan 2020 12:53:20 +0000 (13:53 +0100)]
git: split out idx file writing to a separate class

Split the idx file writing into a separate class, to make that
kind of action available separately. This will be useful for the
next patch where we use it to test some idx/midx code.

In the future, it'll also be useful for encrypted repositories
since the idx format there will be useful for local caching to
take advantage of midx and bloom code as is, but the packwriter
will of course not be useful.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agogit: stop using .encode('hex') in MissingObject()
Johannes Berg [Sat, 18 Apr 2020 21:29:22 +0000 (23:29 +0200)]
git: stop using .encode('hex') in MissingObject()

This needs to use hexlify() instead for python3 compatibility.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agofsck: stop using .encode('hex')
Johannes Berg [Sat, 18 Apr 2020 21:28:59 +0000 (23:28 +0200)]
fsck: stop using .encode('hex')

An error path is using this still, use hexlify() instead.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agotest-save-errors: fix shebang for freebsd
Johannes Berg [Mon, 16 Mar 2020 21:09:02 +0000 (22:09 +0100)]
test-save-errors: fix shebang for freebsd

It appears that freebsd doesn't support recursive interpreters,
so you cannot use a shell script directly as one. Instead, to
fix it, invoke it via /usr/bin/env.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
6 weeks agossh: simplify the code
Johannes Berg [Mon, 13 Jan 2020 19:53:26 +0000 (20:53 +0100)]
ssh: simplify the code

There's no point in shipping PATH to the remote server, since
it will be different there. We can also simplify the loopback
check, and we don't really need to munge the PATH there either
if we just use path.exe() in place of a plain 'bup' for it.

While at it, also fix the formatting instruction for the ints
to %d, instead of %s.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agogit: overhaul git version checking
Rob Browning [Tue, 11 Feb 2020 02:47:13 +0000 (20:47 -0600)]
git: overhaul git version checking

Rework the git version tetsting to handle versions like "1.5.2-rc3"
or (apparently) "1.5.2-rc3 (something ...)".  Add tests for parsing of
all the version types in the current git tag history that we need to
support.

Support and document BUP_ASSUME_GIT_VERSION_IS_FINE=1 as an escape
hatch in case the parsing isn't sufficiently comprehensive, or
upstreeam changes their practices in future releases.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 months agocmd/fsck-cmd.py: Append newline after quoted par2 output
Christian Cornelssen [Tue, 18 Feb 2020 15:17:33 +0000 (16:17 +0100)]
cmd/fsck-cmd.py: Append newline after quoted par2 output

With a recent `par2` installed, `bup fsck -v` was printing messages
like this:

  Unexpected par2 error output
  ''Assuming par2 supports parallel processing

when there was no error output and the exit status was zero.  A
previous commit 19f9faeb0055dadb7f76a953d51acec8373c6edb eliminated
the spurious reporting; now make sure we print a newline after the
par2 output whenever there actually is an error.

Signed-off-by: Christian Cornelssen <ccorn@1tein.de>
[rlb@defaultvalue.org: add information from the pr to commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agocmd/fsck-cmd.py: Do not warn about empty par2 output
Christian Cornelssen [Tue, 18 Feb 2020 14:43:51 +0000 (15:43 +0100)]
cmd/fsck-cmd.py: Do not warn about empty par2 output

With a recent `par2` installed, `bup fsck -v` outputs messages like

  Unexpected par2 error output
  ''Assuming par2 supports parallel processing

when there was no error output and the exit status was zero.  Stop
printing the warning in that case.

Signed-off-by: Christian Cornelssen <ccorn@1tein.de>
[rlb@defaultvalue.org: add information from the pr to commit message]
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 months agoindex: make --fake-valid match the man page
Johannes Berg [Mon, 17 Feb 2020 20:02:11 +0000 (21:02 +0100)]
index: make --fake-valid match the man page

The index command currently clobbers the hash of a file when
marking it as valid, but the man page states:

    --fake-valid
        mark specified paths as up-to-date even if they aren't.
        This can be useful for testing, or to avoid unnecessarily
        backing up files that you know are boring.

The latter part ("avoid unnecessarily backing up [...]") cannot be
implemented with --fake-valid as is, because of the clobbering of
the hash: the fake invented hash will not exist in the repository,
and thus save checks and saves the file.

Fix this by clobbering the hash only if it's the invalid EMPTY_SHA.

Add a test for this to test-save-smaller, just because that's where
we discovered it.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agosave: add test for --smaller, fix DESIGN document
Johannes Berg [Mon, 17 Feb 2020 19:03:22 +0000 (20:03 +0100)]
save: add test for --smaller, fix DESIGN document

Add a test for --smaller, in particular showing that the actual --smaller
behaviour doesn't match what's described in the DESIGN file, which says:

    Another interesting trick is that you can skip backing up files even if
    IX_HASHVALID *isn't* set, as long as you have that file's sha1 in the
    repository.  What that means is you've chosen not to backup the latest
    version of that file; instead, your new backup set just contains the
    most-recently-known valid version of that file.  This is a good trick if you
    want to do frequent backups of smallish files and infrequent backups of
    large ones (as in 'bup save --smaller').  Each of your backups will be
    "complete," in that they contain all the small files and the large ones, but
    intermediate ones will just contain out-of-date copies of the large files.

This ("Each of your backups will be 'complete,' [...]") would seem to indicate
all files should be present, but in fact neither new nor old files are actually
saved by 'bup save --smaller'.

To avoid confusion, also update the DESIGN documentation here.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agodocumentation: quote literal * in man pages
Johannes Berg [Sun, 16 Feb 2020 21:13:03 +0000 (22:13 +0100)]
documentation: quote literal * in man pages

For "1024*1024" etc. we need to quote the * so that it
doesn't get used as bold markup, fix that.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agobuild: fix C-side dependencies
Johannes Berg [Wed, 5 Feb 2020 19:36:46 +0000 (20:36 +0100)]
build: fix C-side dependencies

We need to depend on bupsplit.h, otherwise changes there
don't force a rebuild.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agoget: convert opt.source to bytes
Johannes Berg [Tue, 4 Feb 2020 20:27:23 +0000 (21:27 +0100)]
get: convert opt.source to bytes

I noticed this while playing with something else that
didn't just pass the repo_dir to git, but instead used
it with some os.path.join() calls that complain about
mixed unicode/bytes.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agocompat: directly assign bytes_from_uint = chr
Johannes Berg [Fri, 31 Jan 2020 21:00:56 +0000 (22:00 +0100)]
compat: directly assign bytes_from_uint = chr

This is significantly faster than the indirection and
seems to reduce the runtime of the test suite by about
3% on my machine with python 2 (76.121s -> 73.725s,
but I tried only once which is clearly not enough.)

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agovfs: remove dead cache_get_revlist_item()
Johannes Berg [Wed, 29 Jan 2020 20:32:39 +0000 (21:32 +0100)]
vfs: remove dead cache_get_revlist_item()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agoBaseConn: let _read/_readline raise NotImplementedError
Johannes Berg [Fri, 13 Dec 2019 20:55:53 +0000 (21:55 +0100)]
BaseConn: let _read/_readline raise NotImplementedError

This way, it's easier to understand the code, since these
functions aren't referenced without existing in BaseConn.

Also change has_input() to raise NotImplementedError instead
of trying to instantiate NotImplemented - the latter is just
a singleton to return from the rich comparison methods.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agogit/client/server: remove rev_list() count support
Johannes Berg [Tue, 28 Jan 2020 23:21:01 +0000 (00:21 +0100)]
git/client/server: remove rev_list() count support

This is obviously not used, as passing count!=None would
crash the client method (client.py doesn't import Integral).
Rather than fixing that, just remove support for it entirely.

While at it, also clean up a duplicate rev_list_invocation()
call in the server.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agocatpipe: remove useless StopIteration catching
Johannes Berg [Tue, 28 Jan 2020 23:16:54 +0000 (00:16 +0100)]
catpipe: remove useless StopIteration catching

StopIteration cannot escape this loop construct, remove it.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agosave: remove pointless metalist check
Johannes Berg [Tue, 28 Jan 2020 19:41:10 +0000 (20:41 +0100)]
save: remove pointless metalist check

The metalist can never be empty, since at every level we add
at least the directory's own metadata; it may not have actual
metadata, but there's an entry all the time.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 months agosave/vfs: update comments wrt. tree/bupm ordering
Johannes Berg [Sat, 25 Jan 2020 23:17:38 +0000 (00:17 +0100)]
save/vfs: update comments wrt. tree/bupm ordering

After looking into this and thinking about it, the comments here
are a bit misleading - save states the entries must be in a given
order without a rationale, and vfs states that the order is wrong
but gives an explanation that's not quite right.

Update both comments to make this clearer, and to document that
there's no inherent reason, just happened to pick something when
the save code was written, which turned out to be not the best.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
3 months agotests: add test for save encountering duplicates
Johannes Berg [Sat, 25 Jan 2020 22:13:17 +0000 (23:13 +0100)]
tests: add test for save encountering duplicates

Add a test for save encountering duplicates in the index, both
for a file and a directory, which was fixed in the previous patch.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 months agosave: don't confuse metadata on duplicate files
Johannes Berg [Sat, 25 Jan 2020 22:41:56 +0000 (23:41 +0100)]
save: don't confuse metadata on duplicate files

If there are duplicate files, save removes the duplicates but
doesn't remove the duplicates from metadata, so the metadata
list gets messed up wrt. the file list. Fix this.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agosave: minor code cleanups
Johannes Berg [Sat, 25 Jan 2020 20:34:52 +0000 (21:34 +0100)]
save: minor code cleanups

Move some code around to avoid double ifs and nested ifs
where elif chains can be used instead.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 months agotest-save-errors: add tests for inaccessible metadata
Johannes Berg [Sat, 25 Jan 2020 21:21:24 +0000 (22:21 +0100)]
test-save-errors: add tests for inaccessible metadata

Add test cases for when metadata for a file and directory cannot
be read (causing an IOError instead).

Note that this test fails if the previous two patches aren't
applied.

Note also that if such an error ever happened without the patch
to save, the repository is essentially corrupt and there no way
to figure out _which_ file didn't get metadata.

Note that this uses some python trickery to force an IOError in
metadata reading.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: name test in commit summary line]
Tested-by: Rob Browning <rlb@defaultvalue.org>
3 months agovfs: deal with bupm containing no metadata for a file
Johannes Berg [Sat, 25 Jan 2020 21:20:36 +0000 (22:20 +0100)]
vfs: deal with bupm containing no metadata for a file

If a file was for some reason saved without metadata, deal with
this and create an empty/unknown metadata for it instead.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agosave: add empty metadata if reading fails
Johannes Berg [Sat, 25 Jan 2020 21:18:53 +0000 (22:18 +0100)]
save: add empty metadata if reading fails

If we cannot read a file's metadata, we must add an entry in the
repository nonetheless because the bupm contents must match the
tree contents.

Add a dummy (empty) metadata entry in this case, as is already
done when reading a directory's metadata.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agotests: metadata: fix failures on systems with selinux enabled
Johannes Berg [Thu, 2 Jan 2020 20:09:57 +0000 (21:09 +0100)]
tests: metadata: fix failures on systems with selinux enabled

We get a number of failures if selinux is enabled:

! tmetadata.py:298  ['user.foo', 'security.selinux'] == ['user.foo']     FAILED
! tmetadata.py:302  ['user.foo', 'security.selinux'] == ['user.foo']     FAILED
! tmetadata.py:306  ['user.foo', 'security.selinux'] == ['user.foo']     FAILED

Filter out security.selinux to address that.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agogit: fix PackIdxList keeping deleted files open
Johannes Berg [Sat, 18 Jan 2020 21:26:16 +0000 (22:26 +0100)]
git: fix PackIdxList keeping deleted files open

When an midx is deleted underneath bup, usually by itself running
'bup midx --auto', then PackIdxList may keep them open. This can
cause bup to run out of disk space easily since these files can
be fairly big, and can be recreated multiple times in a backup
run.

To fix this, remove any open PackMidx instances from the list and
close them explicitly.

Out of an abundance of caution, also explicitly close the bloom
instance if we have one - the same issue should apply here even if
I couldn't observe it, since the GC isn't guaranteed to clean up
the object immediately.

I remember debugging this issue years ago without coming to any
good conclusion, and it's been mentioned on the mailing list a few
times as well, e.g.
https://groups.google.com/d/msg/bup-list/AqIyv9n9WPE/-Wl2JVh5AQAJ

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
3 months agoperf-glance: make compatible with python==python3
Johannes Berg [Sat, 8 Feb 2020 20:10:58 +0000 (21:10 +0100)]
perf-glance: make compatible with python==python3

If python is python3, then print is a function. Fix the code
to make it a function since python2 doesn't really care (in
this particular case where we just have a single argument).

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-web: mark as unresolved with respect to python 3
Rob Browning [Sat, 4 Jan 2020 19:31:48 +0000 (13:31 -0600)]
test-web: mark as unresolved with respect to python 3

Since cmd-web hasn't been adjusted for python 3 yet.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-restore-map-owner: accommodate python 3 and test there
Rob Browning [Sat, 4 Jan 2020 18:41:34 +0000 (12:41 -0600)]
test-restore-map-owner: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 months agoEnable test-index-check-device test-xdev for python 3
Rob Browning [Fri, 3 Jan 2020 22:20:33 +0000 (16:20 -0600)]
Enable test-index-check-device test-xdev for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoid-other-than: accommodate python 3
Rob Browning [Fri, 3 Jan 2020 21:24:12 +0000 (15:24 -0600)]
id-other-than: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agopwdgrp: check for bytes with isinstance everywhere
Rob Browning [Sun, 19 Jan 2020 19:35:34 +0000 (13:35 -0600)]
pwdgrp: check for bytes with isinstance everywhere

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agopwdgrp: pass strings to python for python 3
Rob Browning [Fri, 3 Jan 2020 21:23:54 +0000 (15:23 -0600)]
pwdgrp: pass strings to python for python 3

Python 3's getpwnam and getgrnam functions only accept unicode
strings, so decode the bytes we have as iso-8859-1, which is what they
should be, given bup-python's LC_CTYPE override.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agohashsplit: replace join_bytes with cat_bytes
Rob Browning [Fri, 3 Jan 2020 20:27:09 +0000 (14:27 -0600)]
hashsplit: replace join_bytes with cat_bytes

Add a C cat_bytes that can concatenate two bytes objects with offsets
and extents.  This allows us to have the same implementation for
python 2 and 3, to drop another use of buffer(), and may be handy in
the future, particularly given the expense of getting a buffer offset
in python 3 (i.e. memoryview() adds about ~200 bytes).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoget: adjust for python 3 and test there
Rob Browning [Fri, 3 Jan 2020 02:23:09 +0000 (20:23 -0600)]
get: adjust for python 3 and test there

Thanks to Johannes Berg for pointing out some overlooked conversions
in a previous version of the patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agofuse: adjust for python 3 and test there
Rob Browning [Thu, 2 Jan 2020 21:30:28 +0000 (15:30 -0600)]
fuse: adjust for python 3 and test there

The python 3 version could have issues until the fuse module supports
binary data more completely (e.g. bytes paths), or until we switch to
some other foundation, but it may be OK even so (with some
inefficiency) given our bup-python iso-8859-1 hack.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust memtest and random_sha for python 3 and enable test.sh
Rob Browning [Thu, 2 Jan 2020 19:04:21 +0000 (13:04 -0600)]
Adjust memtest and random_sha for python 3 and enable test.sh

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoftp: accommodate python 3 and test there
Rob Browning [Thu, 2 Jan 2020 18:28:09 +0000 (12:28 -0600)]
ftp: accommodate python 3 and test there

For now, completely rely on the bup-python LC_CTYPE=iso-8859-1 setting
since at a minimum, bup.shquote can't handle bytes.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoprune-older: accommodate python 3 and test there
Rob Browning [Thu, 2 Jan 2020 17:30:39 +0000 (11:30 -0600)]
prune-older: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-on: enable for python 3
Rob Browning [Thu, 2 Jan 2020 02:45:57 +0000 (20:45 -0600)]
test-on: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-rdiff-backup: enable for python 3
Rob Browning [Thu, 2 Jan 2020 02:27:00 +0000 (20:27 -0600)]
test-rdiff-backup: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-main: enable for python 3
Rob Browning [Thu, 2 Jan 2020 02:23:43 +0000 (20:23 -0600)]
test-main: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoimport-duplicity-cmd: adjust for python 3 and enable test
Rob Browning [Thu, 2 Jan 2020 01:59:34 +0000 (19:59 -0600)]
import-duplicity-cmd: adjust for python 3 and enable test

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agohelpers: add close_fds to exo
Rob Browning [Sun, 12 Jan 2020 17:54:26 +0000 (11:54 -0600)]
helpers: add close_fds to exo

Add close_fds to exo() for import-duplicity, and while we're
there, define readpipe in terms of exo(), since it's just a
simplification.

Default close_fds to True, since it doesn't look like we need to
preserve the open fds in our other calls.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agohelpers.exo: only report non-empty stderr
Rob Browning [Sun, 12 Jan 2020 18:03:23 +0000 (12:03 -0600)]
helpers.exo: only report non-empty stderr

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-gc: enable for python 3
Rob Browning [Thu, 2 Jan 2020 01:39:44 +0000 (19:39 -0600)]
test-gc: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust rm-cmd and bup.rm for python 3 and enable test-rm
Rob Browning [Thu, 2 Jan 2020 01:22:13 +0000 (19:22 -0600)]
Adjust rm-cmd and bup.rm for python 3 and enable test-rm

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust resolve-parent for python 3 and enable test-save-*
Rob Browning [Wed, 1 Jan 2020 21:04:30 +0000 (15:04 -0600)]
Adjust resolve-parent for python 3 and enable test-save-*

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoEnable test-rm-between-index-and-save for python 3
Rob Browning [Wed, 1 Jan 2020 20:55:30 +0000 (14:55 -0600)]
Enable test-rm-between-index-and-save for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoTest test-restore-single-file test-save-creates-no-unrefs w/python 3
Rob Browning [Wed, 1 Jan 2020 20:51:53 +0000 (14:51 -0600)]
Test test-restore-single-file test-save-creates-no-unrefs w/python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoEnable test-index test-restore-single-file for python 3
Rob Browning [Wed, 1 Jan 2020 20:50:47 +0000 (14:50 -0600)]
Enable test-index test-restore-single-file for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust on-cmd on--server-cmd mux-cmd DemuxConn for python 3
Rob Browning [Wed, 1 Jan 2020 20:44:25 +0000 (14:44 -0600)]
Adjust on-cmd on--server-cmd mux-cmd DemuxConn for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-sparse-files: accommodate python 3 and test there
Rob Browning [Wed, 1 Jan 2020 20:33:35 +0000 (14:33 -0600)]
test-sparse-files: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agodata-size: accommodate python 3
Rob Browning [Wed, 1 Jan 2020 20:31:35 +0000 (14:31 -0600)]
data-size: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agosparse-test-data: accommodate python 3
Rob Browning [Wed, 1 Jan 2020 20:27:58 +0000 (14:27 -0600)]
sparse-test-data: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust restore-cmd for python 3 and then enable test-meta
Rob Browning [Wed, 1 Jan 2020 20:08:07 +0000 (14:08 -0600)]
Adjust restore-cmd for python 3 and then enable test-meta

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-meta: accommodate python 3
Rob Browning [Sat, 4 Jan 2020 18:06:13 +0000 (12:06 -0600)]
test-meta: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-packsizelimit: test with python 3
Rob Browning [Wed, 1 Jan 2020 20:00:21 +0000 (14:00 -0600)]
test-packsizelimit: test with python 3

Tested-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust split-cmd join-cmd margin-cmd for python 3; test-split-join
Rob Browning [Wed, 1 Jan 2020 19:52:29 +0000 (13:52 -0600)]
Adjust split-cmd join-cmd margin-cmd for python 3; test-split-join

Run test-split-join for python 3 after finishing the relevant
adjustments.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agounknown-owner: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
unknown-owner: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agohardlink-sets: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
hardlink-sets: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agolist-idx-cmd: adjust for python 3 and enable test-list-idx
Rob Browning [Wed, 1 Jan 2020 19:16:43 +0000 (13:16 -0600)]
list-idx-cmd: adjust for python 3 and enable test-list-idx

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-cat-file test-command-without-init-fails: test with python 3
Rob Browning [Wed, 1 Jan 2020 19:09:13 +0000 (13:09 -0600)]
test-cat-file test-command-without-init-fails: test with python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust --exclude* parsing and drecurse-cmd for python 3 and test
Rob Browning [Wed, 1 Jan 2020 19:03:46 +0000 (13:03 -0600)]
Adjust --exclude* parsing and drecurse-cmd for python 3 and test

Handle --exclude and --exclude-from values as bytes, adjust
drecurse-command for python 3, and include test-drecurse.sh
in the python 3 test set.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agocat-file-cmd: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
cat-file-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agons-timestamp-resolutions: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
ns-timestamp-resolutions: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agosubtree-hash: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
subtree-hash: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agodamage-cmd: adjust for python 3 and enable test-fsck
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
damage-cmd: adjust for python 3 and enable test-fsck

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust columnate for python 3; enable test-ls test-ls-remote
Rob Browning [Tue, 31 Dec 2019 23:44:39 +0000 (17:44 -0600)]
Adjust columnate for python 3; enable test-ls test-ls-remote

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoTest test-argv test-compression test-index-clear test-tz w/python 3
Rob Browning [Tue, 31 Dec 2019 23:28:45 +0000 (17:28 -0600)]
Test test-argv test-compression test-index-clear test-tz w/python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotresolve: accommodate python 3 and test there
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
tresolve: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotag-cmd: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 23:02:50 +0000 (17:02 -0600)]
tag-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoAdjust server and client to accommodate python 3
Rob Browning [Tue, 31 Dec 2019 21:37:02 +0000 (15:37 -0600)]
Adjust server and client to accommodate python 3

Make the typical bytes-related adjustments to bup-server, bup.client,
and bup.ssh and enable the tclient tests with python 3.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotvfs: accommodate python 3 and test there
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
tvfs: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoindex: fix Entry comparison methods
Johannes Berg [Tue, 28 Jan 2020 23:20:09 +0000 (00:20 +0100)]
index: fix Entry comparison methods

Some of these methods are evidently not used today,
but they should have proper arguments anyway.

Fixes: 3ff7455dd474 ("Don't use cmp()")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
4 months agoAdjust buptest logcmd to handle bytes
Rob Browning [Tue, 31 Dec 2019 19:54:25 +0000 (13:54 -0600)]
Adjust buptest logcmd to handle bytes

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agoINTEGRAL_ASSIGNMENT_FITS: actually provide return value for clang
Rob Browning [Tue, 31 Dec 2019 18:19:39 +0000 (12:19 -0600)]
INTEGRAL_ASSIGNMENT_FITS: actually provide return value for clang

Apparently clang does need the pragmas, so either I tested it
incorrectly before, or my local clang is different.  It looks like
clang doesn't ignore the pragmas as far as the expression result value
is concerned, so explicitly put the value at the end.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agowrite_sparsely bitmatch set_linux_file_attr: accommodate python 3
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
write_sparsely bitmatch set_linux_file_attr: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agotests: git: test larger blob encode/decode
Johannes Berg [Thu, 30 Jan 2020 20:15:19 +0000 (21:15 +0100)]
tests: git: test larger blob encode/decode

This exercises more code paths, especially in decode.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agogit: remove unused _git_capture()
Johannes Berg [Wed, 29 Jan 2020 20:45:10 +0000 (21:45 +0100)]
git: remove unused _git_capture()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agotest-web: check that static/styles.css can be retrieved
Johannes Berg [Wed, 29 Jan 2020 20:40:31 +0000 (21:40 +0100)]
test-web: check that static/styles.css can be retrieved

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agooptions: remove unused _atoi()
Johannes Berg [Wed, 29 Jan 2020 20:35:57 +0000 (21:35 +0100)]
options: remove unused _atoi()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agotests: vint: test EOFError after first byte
Johannes Berg [Wed, 29 Jan 2020 19:09:49 +0000 (20:09 +0100)]
tests: vint: test EOFError after first byte

Since the first byte is handled separately for the sign bit,
validate that we also get an EOFError if there are a few
bytes but the last one also has the 0x80 bit set.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agovint: remove unnecessary condition
Johannes Berg [Wed, 29 Jan 2020 19:09:27 +0000 (20:09 +0100)]
vint: remove unnecessary condition

"if c:" can never be false, since we checked before.
Remove the extra condition.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
4 months agohelpers: remove unused count()
Johannes Berg [Tue, 28 Jan 2020 23:25:49 +0000 (00:25 +0100)]
helpers: remove unused count()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agoclient: import socket
Johannes Berg [Tue, 28 Jan 2020 23:23:18 +0000 (00:23 +0100)]
client: import socket

Fixes: 7ce8041f0345 ("Teach bup about URLs and non-ssh remotes")

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
4 months agoclient: import atoi
Johannes Berg [Tue, 28 Jan 2020 23:22:56 +0000 (00:22 +0100)]
client: import atoi

Fixes: 22d01e1a8077 ("If you specified the port number on the command line, convert it to an int.")

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>