bup.git
8 months agotest-restore-map-owner: accommodate python 3 and test there
Rob Browning [Sat, 4 Jan 2020 18:41:34 +0000 (12:41 -0600)]
test-restore-map-owner: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
8 months agoEnable test-index-check-device test-xdev for python 3
Rob Browning [Fri, 3 Jan 2020 22:20:33 +0000 (16:20 -0600)]
Enable test-index-check-device test-xdev for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoid-other-than: accommodate python 3
Rob Browning [Fri, 3 Jan 2020 21:24:12 +0000 (15:24 -0600)]
id-other-than: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agopwdgrp: check for bytes with isinstance everywhere
Rob Browning [Sun, 19 Jan 2020 19:35:34 +0000 (13:35 -0600)]
pwdgrp: check for bytes with isinstance everywhere

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agopwdgrp: pass strings to python for python 3
Rob Browning [Fri, 3 Jan 2020 21:23:54 +0000 (15:23 -0600)]
pwdgrp: pass strings to python for python 3

Python 3's getpwnam and getgrnam functions only accept unicode
strings, so decode the bytes we have as iso-8859-1, which is what they
should be, given bup-python's LC_CTYPE override.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agohashsplit: replace join_bytes with cat_bytes
Rob Browning [Fri, 3 Jan 2020 20:27:09 +0000 (14:27 -0600)]
hashsplit: replace join_bytes with cat_bytes

Add a C cat_bytes that can concatenate two bytes objects with offsets
and extents.  This allows us to have the same implementation for
python 2 and 3, to drop another use of buffer(), and may be handy in
the future, particularly given the expense of getting a buffer offset
in python 3 (i.e. memoryview() adds about ~200 bytes).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoget: adjust for python 3 and test there
Rob Browning [Fri, 3 Jan 2020 02:23:09 +0000 (20:23 -0600)]
get: adjust for python 3 and test there

Thanks to Johannes Berg for pointing out some overlooked conversions
in a previous version of the patch.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agofuse: adjust for python 3 and test there
Rob Browning [Thu, 2 Jan 2020 21:30:28 +0000 (15:30 -0600)]
fuse: adjust for python 3 and test there

The python 3 version could have issues until the fuse module supports
binary data more completely (e.g. bytes paths), or until we switch to
some other foundation, but it may be OK even so (with some
inefficiency) given our bup-python iso-8859-1 hack.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust memtest and random_sha for python 3 and enable test.sh
Rob Browning [Thu, 2 Jan 2020 19:04:21 +0000 (13:04 -0600)]
Adjust memtest and random_sha for python 3 and enable test.sh

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoftp: accommodate python 3 and test there
Rob Browning [Thu, 2 Jan 2020 18:28:09 +0000 (12:28 -0600)]
ftp: accommodate python 3 and test there

For now, completely rely on the bup-python LC_CTYPE=iso-8859-1 setting
since at a minimum, bup.shquote can't handle bytes.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoprune-older: accommodate python 3 and test there
Rob Browning [Thu, 2 Jan 2020 17:30:39 +0000 (11:30 -0600)]
prune-older: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-on: enable for python 3
Rob Browning [Thu, 2 Jan 2020 02:45:57 +0000 (20:45 -0600)]
test-on: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-rdiff-backup: enable for python 3
Rob Browning [Thu, 2 Jan 2020 02:27:00 +0000 (20:27 -0600)]
test-rdiff-backup: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-main: enable for python 3
Rob Browning [Thu, 2 Jan 2020 02:23:43 +0000 (20:23 -0600)]
test-main: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoimport-duplicity-cmd: adjust for python 3 and enable test
Rob Browning [Thu, 2 Jan 2020 01:59:34 +0000 (19:59 -0600)]
import-duplicity-cmd: adjust for python 3 and enable test

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agohelpers: add close_fds to exo
Rob Browning [Sun, 12 Jan 2020 17:54:26 +0000 (11:54 -0600)]
helpers: add close_fds to exo

Add close_fds to exo() for import-duplicity, and while we're
there, define readpipe in terms of exo(), since it's just a
simplification.

Default close_fds to True, since it doesn't look like we need to
preserve the open fds in our other calls.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agohelpers.exo: only report non-empty stderr
Rob Browning [Sun, 12 Jan 2020 18:03:23 +0000 (12:03 -0600)]
helpers.exo: only report non-empty stderr

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-gc: enable for python 3
Rob Browning [Thu, 2 Jan 2020 01:39:44 +0000 (19:39 -0600)]
test-gc: enable for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust rm-cmd and bup.rm for python 3 and enable test-rm
Rob Browning [Thu, 2 Jan 2020 01:22:13 +0000 (19:22 -0600)]
Adjust rm-cmd and bup.rm for python 3 and enable test-rm

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust resolve-parent for python 3 and enable test-save-*
Rob Browning [Wed, 1 Jan 2020 21:04:30 +0000 (15:04 -0600)]
Adjust resolve-parent for python 3 and enable test-save-*

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoEnable test-rm-between-index-and-save for python 3
Rob Browning [Wed, 1 Jan 2020 20:55:30 +0000 (14:55 -0600)]
Enable test-rm-between-index-and-save for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoTest test-restore-single-file test-save-creates-no-unrefs w/python 3
Rob Browning [Wed, 1 Jan 2020 20:51:53 +0000 (14:51 -0600)]
Test test-restore-single-file test-save-creates-no-unrefs w/python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoEnable test-index test-restore-single-file for python 3
Rob Browning [Wed, 1 Jan 2020 20:50:47 +0000 (14:50 -0600)]
Enable test-index test-restore-single-file for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust on-cmd on--server-cmd mux-cmd DemuxConn for python 3
Rob Browning [Wed, 1 Jan 2020 20:44:25 +0000 (14:44 -0600)]
Adjust on-cmd on--server-cmd mux-cmd DemuxConn for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-sparse-files: accommodate python 3 and test there
Rob Browning [Wed, 1 Jan 2020 20:33:35 +0000 (14:33 -0600)]
test-sparse-files: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agodata-size: accommodate python 3
Rob Browning [Wed, 1 Jan 2020 20:31:35 +0000 (14:31 -0600)]
data-size: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agosparse-test-data: accommodate python 3
Rob Browning [Wed, 1 Jan 2020 20:27:58 +0000 (14:27 -0600)]
sparse-test-data: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust restore-cmd for python 3 and then enable test-meta
Rob Browning [Wed, 1 Jan 2020 20:08:07 +0000 (14:08 -0600)]
Adjust restore-cmd for python 3 and then enable test-meta

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-meta: accommodate python 3
Rob Browning [Sat, 4 Jan 2020 18:06:13 +0000 (12:06 -0600)]
test-meta: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-packsizelimit: test with python 3
Rob Browning [Wed, 1 Jan 2020 20:00:21 +0000 (14:00 -0600)]
test-packsizelimit: test with python 3

Tested-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust split-cmd join-cmd margin-cmd for python 3; test-split-join
Rob Browning [Wed, 1 Jan 2020 19:52:29 +0000 (13:52 -0600)]
Adjust split-cmd join-cmd margin-cmd for python 3; test-split-join

Run test-split-join for python 3 after finishing the relevant
adjustments.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agounknown-owner: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
unknown-owner: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agohardlink-sets: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
hardlink-sets: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agolist-idx-cmd: adjust for python 3 and enable test-list-idx
Rob Browning [Wed, 1 Jan 2020 19:16:43 +0000 (13:16 -0600)]
list-idx-cmd: adjust for python 3 and enable test-list-idx

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-cat-file test-command-without-init-fails: test with python 3
Rob Browning [Wed, 1 Jan 2020 19:09:13 +0000 (13:09 -0600)]
test-cat-file test-command-without-init-fails: test with python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust --exclude* parsing and drecurse-cmd for python 3 and test
Rob Browning [Wed, 1 Jan 2020 19:03:46 +0000 (13:03 -0600)]
Adjust --exclude* parsing and drecurse-cmd for python 3 and test

Handle --exclude and --exclude-from values as bytes, adjust
drecurse-command for python 3, and include test-drecurse.sh
in the python 3 test set.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agocat-file-cmd: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
cat-file-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agons-timestamp-resolutions: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
ns-timestamp-resolutions: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agosubtree-hash: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
subtree-hash: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agodamage-cmd: adjust for python 3 and enable test-fsck
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
damage-cmd: adjust for python 3 and enable test-fsck

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust columnate for python 3; enable test-ls test-ls-remote
Rob Browning [Tue, 31 Dec 2019 23:44:39 +0000 (17:44 -0600)]
Adjust columnate for python 3; enable test-ls test-ls-remote

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoTest test-argv test-compression test-index-clear test-tz w/python 3
Rob Browning [Tue, 31 Dec 2019 23:28:45 +0000 (17:28 -0600)]
Test test-argv test-compression test-index-clear test-tz w/python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotresolve: accommodate python 3 and test there
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
tresolve: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotag-cmd: accommodate python 3
Rob Browning [Tue, 31 Dec 2019 23:02:50 +0000 (17:02 -0600)]
tag-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoAdjust server and client to accommodate python 3
Rob Browning [Tue, 31 Dec 2019 21:37:02 +0000 (15:37 -0600)]
Adjust server and client to accommodate python 3

Make the typical bytes-related adjustments to bup-server, bup.client,
and bup.ssh and enable the tclient tests with python 3.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotvfs: accommodate python 3 and test there
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
tvfs: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoindex: fix Entry comparison methods
Johannes Berg [Tue, 28 Jan 2020 23:20:09 +0000 (00:20 +0100)]
index: fix Entry comparison methods

Some of these methods are evidently not used today,
but they should have proper arguments anyway.

Fixes: 3ff7455dd474 ("Don't use cmp()")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
8 months agoAdjust buptest logcmd to handle bytes
Rob Browning [Tue, 31 Dec 2019 19:54:25 +0000 (13:54 -0600)]
Adjust buptest logcmd to handle bytes

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoINTEGRAL_ASSIGNMENT_FITS: actually provide return value for clang
Rob Browning [Tue, 31 Dec 2019 18:19:39 +0000 (12:19 -0600)]
INTEGRAL_ASSIGNMENT_FITS: actually provide return value for clang

Apparently clang does need the pragmas, so either I tested it
incorrectly before, or my local clang is different.  It looks like
clang doesn't ignore the pragmas as far as the expression result value
is concerned, so explicitly put the value at the end.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agowrite_sparsely bitmatch set_linux_file_attr: accommodate python 3
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
write_sparsely bitmatch set_linux_file_attr: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotests: git: test larger blob encode/decode
Johannes Berg [Thu, 30 Jan 2020 20:15:19 +0000 (21:15 +0100)]
tests: git: test larger blob encode/decode

This exercises more code paths, especially in decode.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agogit: remove unused _git_capture()
Johannes Berg [Wed, 29 Jan 2020 20:45:10 +0000 (21:45 +0100)]
git: remove unused _git_capture()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agotest-web: check that static/styles.css can be retrieved
Johannes Berg [Wed, 29 Jan 2020 20:40:31 +0000 (21:40 +0100)]
test-web: check that static/styles.css can be retrieved

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agooptions: remove unused _atoi()
Johannes Berg [Wed, 29 Jan 2020 20:35:57 +0000 (21:35 +0100)]
options: remove unused _atoi()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agotests: vint: test EOFError after first byte
Johannes Berg [Wed, 29 Jan 2020 19:09:49 +0000 (20:09 +0100)]
tests: vint: test EOFError after first byte

Since the first byte is handled separately for the sign bit,
validate that we also get an EOFError if there are a few
bytes but the last one also has the 0x80 bit set.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agovint: remove unnecessary condition
Johannes Berg [Wed, 29 Jan 2020 19:09:27 +0000 (20:09 +0100)]
vint: remove unnecessary condition

"if c:" can never be false, since we checked before.
Remove the extra condition.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agohelpers: remove unused count()
Johannes Berg [Tue, 28 Jan 2020 23:25:49 +0000 (00:25 +0100)]
helpers: remove unused count()

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agoclient: import socket
Johannes Berg [Tue, 28 Jan 2020 23:23:18 +0000 (00:23 +0100)]
client: import socket

Fixes: 7ce8041f0345 ("Teach bup about URLs and non-ssh remotes")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agoclient: import atoi
Johannes Berg [Tue, 28 Jan 2020 23:22:56 +0000 (00:22 +0100)]
client: import atoi

Fixes: 22d01e1a8077 ("If you specified the port number on the command line, convert it to an int.")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agoclient: import DemuxConn
Johannes Berg [Tue, 28 Jan 2020 23:22:33 +0000 (00:22 +0100)]
client: import DemuxConn

Fixes: fb3bd84cfd24 ("Add DemuxConn and `bup mux` for client-server")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agovfs: fix finish_extract()
Johannes Berg [Tue, 28 Jan 2020 23:19:33 +0000 (00:19 +0100)]
vfs: fix finish_extract()

The 'dir' variable doesn't exist here, must be 'meta' instead.

Fixes: 0962d3904735 ("Add initial support for metadata archives.")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agosave: remove unused eatslash() function
Johannes Berg [Tue, 28 Jan 2020 16:38:44 +0000 (17:38 +0100)]
save: remove unused eatslash() function

Code coverage reporting pointed this out, and indeed it's unused.
Remove it.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
8 months agoRework shstr to handle bytes and strings; add squote and bquote
Rob Browning [Tue, 31 Dec 2019 19:45:00 +0000 (13:45 -0600)]
Rework shstr to handle bytes and strings; add squote and bquote

These could be smarter, i.e '1 could become "'"1 rather than ''"'"'1',
but we can always improve it later.  And add at last some tests.
Don't rely on compat.quote for strings so that we know we'll have the
same behavior for bytes and strings.

Thanks to Johannes Berg for pointing out an incorrect variable name in
a previous revision.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agotmetadata: accommodate python 3 and test there
Rob Browning [Tue, 31 Dec 2019 17:47:47 +0000 (11:47 -0600)]
tmetadata: accommodate python 3 and test there

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agovfs: accommodate python 3
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
vfs: accommodate python 3

Thanks to Johannes Berg for pointing out a stray str() in one of the
write_ioerror guards in an earlier revision.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoset_linux_file_attr: accommodate python 3
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
set_linux_file_attr: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agoclient: fix index-cache location when there's no path
Johannes Berg [Wed, 22 Jan 2020 08:25:40 +0000 (09:25 +0100)]
client: fix index-cache location when there's no path

Putting my current tree into production, I noticed that the
index-cache was completely re-downloaded (taking a long time)
due to a change in storage location, which was broken in the
commit 85edc0f1c133 ("bup.client: accommodate python 3").

The "self.dir or b'None'" was in commit 85edc0f1c133
("bup.client: accommodate python 3") was clearly well-intended,
but also had the effect of transforming the empty string (which
evaluates to False) to b'None' instead, which is wrong since in
'bup on' cases there's no dir, but parse_remote() comes up with
an empty string instead of None.

Fix that and add a test that checks that the index location
without a dir is actually preserved as such.

Fixes: 85edc0f1c133 ("bup.client: accommodate python 3")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
8 months agopy2raise: add missing 'import sys'
Johannes Berg [Thu, 23 Jan 2020 22:54:01 +0000 (23:54 +0100)]
py2raise: add missing 'import sys'

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
9 months agogit: accommodate python 3 and enable tests
Rob Browning [Mon, 30 Dec 2019 19:38:24 +0000 (13:38 -0600)]
git: accommodate python 3 and enable tests

We finally have enough of the pieces in place.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agomerge_into: accommodate python 3
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
merge_into: accommodate python 3

Switch to Py_buffers to accommodate python 3, use malloc/calloc to
avoid potentially involving the GIL, and check allocation failures.

Thanks to Johannes Berg for pointing out a potential overflow on the C
side -- fixed by adjusting checked_malloc() to accept the same
arguments as checked_calloc().

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agofirstword extract_bits: accommodate python 3
Rob Browning [Sun, 29 Dec 2019 22:22:40 +0000 (16:22 -0600)]
firstword extract_bits: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agobup.midx: accommodate python 3
Rob Browning [Sat, 28 Dec 2019 01:46:02 +0000 (19:46 -0600)]
bup.midx: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoopen_noatime bup_get_linux_file_attr: accommodate python 3
Rob Browning [Sun, 29 Dec 2019 22:22:40 +0000 (16:22 -0600)]
open_noatime bup_get_linux_file_attr: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agowrite_idx: switch to buffer to accommodate python 3
Rob Browning [Sun, 29 Dec 2019 22:17:11 +0000 (16:17 -0600)]
write_idx: switch to buffer to accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agohelpers: accommodate python 3 and enable tests
Rob Browning [Sat, 28 Dec 2019 01:46:02 +0000 (19:46 -0600)]
helpers: accommodate python 3 and enable tests

Update bup.helpers and thelpers to handle python 3, and enable
thelpers for python 3.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoClient.read_ref: replace encode with hexlify
Rob Browning [Mon, 20 Jan 2020 20:30:46 +0000 (14:30 -0600)]
Client.read_ref: replace encode with hexlify

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agocirrus: run at least one long-check
Rob Browning [Mon, 20 Jan 2020 20:54:38 +0000 (14:54 -0600)]
cirrus: run at least one long-check

Since the cirrus root tests already take a good while longer than the
non-root tests, and we don't currently have any root-only long tests,
run it from the non-root debian task.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
9 months agothelpers: call tzset() after changing TZ
Rob Browning [Sun, 29 Dec 2019 20:46:16 +0000 (14:46 -0600)]
thelpers: call tzset() after changing TZ

  Note: Although in many cases, changing the TZ environment variable
  may affect the output of functions like localtime() without calling
  tzset(), this behavior should not be relied on.

  -- https://docs.python.org/3/library/time.html#time.tzset

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agopwdgrp: actually use our grp wrappers
Rob Browning [Sun, 29 Dec 2019 19:44:51 +0000 (13:44 -0600)]
pwdgrp: actually use our grp wrappers

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
9 months agovint: check for bytes after read
Rob Browning [Sun, 29 Dec 2019 05:08:50 +0000 (23:08 -0600)]
vint: check for bytes after read

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agobup.client: accommodate python 3
Rob Browning [Sat, 28 Dec 2019 01:46:02 +0000 (19:46 -0600)]
bup.client: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoAdd compat.reraise to handle python 3 syntax breakage
Rob Browning [Sat, 28 Dec 2019 20:39:44 +0000 (14:39 -0600)]
Add compat.reraise to handle python 3 syntax breakage

Add a exception reraise function to compat that will allow us to
rewrite invocations like this:

    except Exception as e:
        raise ClientError, e, sys.exc_info()[2]

as this:

    except Exception as e:
        reraise(ClientError(e))

since python 3 now provides a with_traceback() method that we can (and
must) use instead.

Put the python 2 specific formulation (shown above) in a separate
py2raise module that we can conditionally import because python 3
decided to make the python 2 code produce a syntax error.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agometadata: adjust our posix1e calls for python 3
Rob Browning [Sat, 4 Jan 2020 18:52:55 +0000 (12:52 -0600)]
metadata: adjust our posix1e calls for python 3

Accommodate at pylibacl's argument requirements (at least 0.5.4).  It
looks like it allows bytes for the ACL() file argument, but not for
filedef:

  $ cmd/bup-python
  Python 3.7.5 (default, Oct 27 2019, 15:43:29)
  [GCC 9.2.1 20191022] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import posix1e
  >>> posix1e.ACL(file=b'README.md')
  <posix1e.ACL object at 0x7fa7bd5cee70>
  >>> posix1e.ACL(file='README.md')
  <posix1e.ACL object at 0x7fa7bd5a8bb0>
  >>> posix1e.ACL(filedef=b'.')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: argument 5 must be str, not bytes

And it expects a string for the to_any_text() prefix argument, but
rquires bytes for the sparator:

  >>> posix1e.ACL(file='README.md').to_any_text(prefix=b'', separator='')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: argument 1 must be str, not bytes

  >>> posix1e.ACL(file='README.md').to_any_text(prefix='', separator='')
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: argument 2 must be a byte string of length 1, not str

  >>> posix1e.ACL(file='README.md').to_any_text(prefix='', separator=b'x')
  b'user::rw-xgroup::r--xother::---'

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
9 months agoAdjust metadata handling for python 3
Rob Browning [Sat, 28 Dec 2019 19:54:03 +0000 (13:54 -0600)]
Adjust metadata handling for python 3

Adapt bup.metadata for python 3 and the other bits directly affected:
bup-ftp, bup-ls, bup-meta, bup-xstat, and bup.ls.

Rename metadata detailed_str() and summary_str() to detailed_bytes()
and summary_bytes() since they aren't (and absolutely should not be)
localized.  They produce output that should be suitable for
programmatic use, i.e. "bup ls | grep ...".  Not sure we'll keep those
names, but they'll do for now.

Also rename fstime_to_sec_str() to fstime_to_sec_bytes() since that's
the only way we ever use it.

Make a minimal change to bup-ftp for now -- just enough to handle the
changed ls.within_repo arguments.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agobup-save bup.drecurse bup.hlinkdb bup.path: accommodate python 3
Rob Browning [Sat, 28 Dec 2019 18:49:13 +0000 (12:49 -0600)]
bup-save bup.drecurse bup.hlinkdb bup.path: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoindex-cmd: accommodate python 3
Rob Browning [Sat, 28 Dec 2019 01:46:02 +0000 (19:46 -0600)]
index-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agobup.gc: accommodate python 3
Rob Browning [Fri, 27 Dec 2019 19:39:11 +0000 (13:39 -0600)]
bup.gc: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoget-cmd: accommodate python 3
Rob Browning [Fri, 27 Dec 2019 19:39:11 +0000 (13:39 -0600)]
get-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agomidx-cmd: accommodate python 3
Rob Browning [Fri, 27 Dec 2019 19:39:11 +0000 (13:39 -0600)]
midx-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agofsck-cmd: accommodate python 3
Rob Browning [Fri, 27 Dec 2019 19:39:11 +0000 (13:39 -0600)]
fsck-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agobloom-cmd: accommodate python 3
Rob Browning [Fri, 27 Dec 2019 19:39:11 +0000 (13:39 -0600)]
bloom-cmd: accommodate python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agomidx: shun buffers
Rob Browning [Fri, 27 Dec 2019 19:24:56 +0000 (13:24 -0600)]
midx: shun buffers

Rework PackMidx to avoid buffers which are reasonably heavyweight, and
will be even larger (as memoryviews) in python 3, requiring ~200
bytes.  Instead, just use direct offsets into the underlying mmap --
slicing an mmap currently just produces bytes.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoindex: remove vestigial buffer and sys imports
Rob Browning [Fri, 27 Dec 2019 19:03:11 +0000 (13:03 -0600)]
index: remove vestigial buffer and sys imports

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
9 months agogit: shun buffers in packidxes
Rob Browning [Fri, 27 Dec 2019 19:02:42 +0000 (13:02 -0600)]
git: shun buffers in packidxes

Rework PackIdx to avoid buffers which are reasonably heavyweight, and
will be even larger (as memoryviews) in python 3 (require ~200 bytes).
Instead, just use direct offsets into the underlying mmap -- slicing
an mmap currently just produces bytes.

Store the fanout table as a homogeneous array rather than a list or
tuple with individually allocated integers.

Instead of looking up hashes one at a time, traverse the index during
gc via its iterator.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agohashsplit: adapt for python 3
Rob Browning [Thu, 26 Dec 2019 23:54:58 +0000 (17:54 -0600)]
hashsplit: adapt for python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agocmd/bup: adapt for python 3
Rob Browning [Thu, 26 Dec 2019 18:50:47 +0000 (12:50 -0600)]
cmd/bup: adapt for python 3

Make all the changes necessary for cmd/bup to work with both python 2
and 3.  Given the current state, the majority of the changes handle
untangling the python 3 unicode/data conflation.

Convert to b'x' literals where needed (e.g. for path or path-derived
values), use argv_bytes to convert at least the command line values
that must not be interpreted as locale strings, and switch to the
bytes-only compat.environ.

More broadly speaking, aside from the changes we abosolutely have to
make, the general intent is for us to handle locale-specific
conversions carefully and explicitly when appropriate, not
transparently.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoAdd "do nothing" path_msg to centralize path conversions
Rob Browning [Thu, 26 Dec 2019 18:27:07 +0000 (12:27 -0600)]
Add "do nothing" path_msg to centralize path conversions

Add path_msg(p) and use it in cmd/bup (as a start).  The intent is to
centralize the encoding of all path values that are to be included in
strings intended for "display" (e.g. stderr, which appears to have to
be a text stream in python 3).

For now, the function will do nothing -- i.e. given the currently
enforced iso-8859-1 encoding we'll just continue to produce the
original path bytes on stderr, but we may well want to make this
configurable at some point (perhaps git's quotePath algorithm might
provide a likely option), and if nothing else, using path_msg()
everywhere makes it much easier to identify and adapt the relevant
code, whatever we decide.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agobup: use correct bup executable in on--server
Johannes Berg [Wed, 1 Jan 2020 17:50:07 +0000 (18:50 +0100)]
bup: use correct bup executable in on--server

Currently, on--server loses the correct bup command in one situation:
if you run 'bup on remote' with a remote forced ssh command that's
different from the installed bup, on--server will still execute the
installed version of bup for all sub-commands.

Fix that by using bup.path.exe() in on--server for bup execution.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agoMakefile: Use explict bup_python to invoke a python script
Greg Troxel [Sun, 12 Jan 2020 19:05:19 +0000 (14:05 -0500)]
Makefile: Use explict bup_python to invoke a python script

Otherwise, the fallback to "python" fails (or perhaps uses the wrong
python).

Signed-off-by: Greg Troxel <gdt@lexort.com>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>
9 months agometadata: accept only fixed python-xattr in python3
Johannes Berg [Sun, 12 Jan 2020 23:08:25 +0000 (00:08 +0100)]
metadata: accept only fixed python-xattr in python3

This is currently broken on python3, it returns junk when
we pass bytes because it uses string and %s internally.
I made a fix for it, so we can detect here if it's fixed
(in which case the NS_USER constant is bytes, not string),
load the module only if it is indeed fixed.

We can do this test always since in python2 bytes == str
and thus isinstance('user', bytes) == True.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>