Johannes Berg [Fri, 31 Jan 2020 21:00:56 +0000 (22:00 +0100)]
compat: directly assign bytes_from_uint = chr
This is significantly faster than the indirection and
seems to reduce the runtime of the test suite by about
3% on my machine with python 2 (76.121s -> 73.725s,
but I tried only once which is clearly not enough.)
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 13 Dec 2019 20:55:53 +0000 (21:55 +0100)]
BaseConn: let _read/_readline raise NotImplementedError
This way, it's easier to understand the code, since these
functions aren't referenced without existing in BaseConn.
Also change has_input() to raise NotImplementedError instead
of trying to instantiate NotImplemented - the latter is just
a singleton to return from the rich comparison methods.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:21:01 +0000 (00:21 +0100)]
git/client/server: remove rev_list() count support
This is obviously not used, as passing count!=None would
crash the client method (client.py doesn't import Integral).
Rather than fixing that, just remove support for it entirely.
While at it, also clean up a duplicate rev_list_invocation()
call in the server.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 19:41:10 +0000 (20:41 +0100)]
save: remove pointless metalist check
The metalist can never be empty, since at every level we add
at least the directory's own metadata; it may not have actual
metadata, but there's an entry all the time.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 23:17:38 +0000 (00:17 +0100)]
save/vfs: update comments wrt. tree/bupm ordering
After looking into this and thinking about it, the comments here
are a bit misleading - save states the entries must be in a given
order without a rationale, and vfs states that the order is wrong
but gives an explanation that's not quite right.
Update both comments to make this clearer, and to document that
there's no inherent reason, just happened to pick something when
the save code was written, which turned out to be not the best.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Johannes Berg [Sat, 25 Jan 2020 22:13:17 +0000 (23:13 +0100)]
tests: add test for save encountering duplicates
Add a test for save encountering duplicates in the index, both
for a file and a directory, which was fixed in the previous patch.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 22:41:56 +0000 (23:41 +0100)]
save: don't confuse metadata on duplicate files
If there are duplicate files, save removes the duplicates but
doesn't remove the duplicates from metadata, so the metadata
list gets messed up wrt. the file list. Fix this.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 20:34:52 +0000 (21:34 +0100)]
save: minor code cleanups
Move some code around to avoid double ifs and nested ifs
where elif chains can be used instead.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 25 Jan 2020 21:21:24 +0000 (22:21 +0100)]
test-save-errors: add tests for inaccessible metadata
Add test cases for when metadata for a file and directory cannot
be read (causing an IOError instead).
Note that this test fails if the previous two patches aren't
applied.
Note also that if such an error ever happened without the patch
to save, the repository is essentially corrupt and there no way
to figure out _which_ file didn't get metadata.
Note that this uses some python trickery to force an IOError in
metadata reading.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: name test in commit summary line] Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 18 Jan 2020 21:26:16 +0000 (22:26 +0100)]
git: fix PackIdxList keeping deleted files open
When an midx is deleted underneath bup, usually by itself running
'bup midx --auto', then PackIdxList may keep them open. This can
cause bup to run out of disk space easily since these files can
be fairly big, and can be recreated multiple times in a backup
run.
To fix this, remove any open PackMidx instances from the list and
close them explicitly.
Out of an abundance of caution, also explicitly close the bloom
instance if we have one - the same issue should apply here even if
I couldn't observe it, since the GC isn't guaranteed to clean up
the object immediately.
I remember debugging this issue years ago without coming to any
good conclusion, and it's been mentioned on the mailing list a few
times as well, e.g.
https://groups.google.com/d/msg/bup-list/AqIyv9n9WPE/-Wl2JVh5AQAJ
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 8 Feb 2020 20:10:58 +0000 (21:10 +0100)]
perf-glance: make compatible with python==python3
If python is python3, then print is a function. Fix the code
to make it a function since python2 doesn't really care (in
this particular case where we just have a single argument).
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 3 Jan 2020 21:23:54 +0000 (15:23 -0600)]
pwdgrp: pass strings to python for python 3
Python 3's getpwnam and getgrnam functions only accept unicode
strings, so decode the bytes we have as iso-8859-1, which is what they
should be, given bup-python's LC_CTYPE override.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 3 Jan 2020 20:27:09 +0000 (14:27 -0600)]
hashsplit: replace join_bytes with cat_bytes
Add a C cat_bytes that can concatenate two bytes objects with offsets
and extents. This allows us to have the same implementation for
python 2 and 3, to drop another use of buffer(), and may be handy in
the future, particularly given the expense of getting a buffer offset
in python 3 (i.e. memoryview() adds about ~200 bytes).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 2 Jan 2020 21:30:28 +0000 (15:30 -0600)]
fuse: adjust for python 3 and test there
The python 3 version could have issues until the fuse module supports
binary data more completely (e.g. bytes paths), or until we switch to
some other foundation, but it may be OK even so (with some
inefficiency) given our bup-python iso-8859-1 hack.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 31 Dec 2019 18:19:39 +0000 (12:19 -0600)]
INTEGRAL_ASSIGNMENT_FITS: actually provide return value for clang
Apparently clang does need the pragmas, so either I tested it
incorrectly before, or my local clang is different. It looks like
clang doesn't ignore the pragmas as far as the expression result value
is concerned, so explicitly put the value at the end.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 29 Jan 2020 19:09:49 +0000 (20:09 +0100)]
tests: vint: test EOFError after first byte
Since the first byte is handled separately for the sign bit,
validate that we also get an EOFError if there are a few
bytes but the last one also has the 0x80 bit set.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 29 Jan 2020 19:09:27 +0000 (20:09 +0100)]
vint: remove unnecessary condition
"if c:" can never be false, since we checked before.
Remove the extra condition.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:23:18 +0000 (00:23 +0100)]
client: import socket
Fixes: 7ce8041f0345 ("Teach bup about URLs and non-ssh remotes") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:56 +0000 (00:22 +0100)]
client: import atoi
Fixes: 22d01e1a8077 ("If you specified the port number on the command line, convert it to an int.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:33 +0000 (00:22 +0100)]
client: import DemuxConn
Fixes: fb3bd84cfd24 ("Add DemuxConn and `bup mux` for client-server") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:19:33 +0000 (00:19 +0100)]
vfs: fix finish_extract()
The 'dir' variable doesn't exist here, must be 'meta' instead.
Fixes: 0962d3904735 ("Add initial support for metadata archives.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 31 Dec 2019 19:45:00 +0000 (13:45 -0600)]
Rework shstr to handle bytes and strings; add squote and bquote
These could be smarter, i.e '1 could become "'"1 rather than ''"'"'1',
but we can always improve it later. And add at last some tests.
Don't rely on compat.quote for strings so that we know we'll have the
same behavior for bytes and strings.
Thanks to Johannes Berg for pointing out an incorrect variable name in
a previous revision.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 22 Jan 2020 08:25:40 +0000 (09:25 +0100)]
client: fix index-cache location when there's no path
Putting my current tree into production, I noticed that the
index-cache was completely re-downloaded (taking a long time)
due to a change in storage location, which was broken in the
commit 85edc0f1c133 ("bup.client: accommodate python 3").
The "self.dir or b'None'" was in commit 85edc0f1c133
("bup.client: accommodate python 3") was clearly well-intended,
but also had the effect of transforming the empty string (which
evaluates to False) to b'None' instead, which is wrong since in
'bup on' cases there's no dir, but parse_remote() comes up with
an empty string instead of None.
Fix that and add a test that checks that the index location
without a dir is actually preserved as such.
Fixes: 85edc0f1c133 ("bup.client: accommodate python 3") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
merge_into: accommodate python 3
Switch to Py_buffers to accommodate python 3, use malloc/calloc to
avoid potentially involving the GIL, and check allocation failures.
Thanks to Johannes Berg for pointing out a potential overflow on the C
side -- fixed by adjusting checked_malloc() to accept the same
arguments as checked_calloc().
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 20 Jan 2020 20:54:38 +0000 (14:54 -0600)]
cirrus: run at least one long-check
Since the cirrus root tests already take a good while longer than the
non-root tests, and we don't currently have any root-only long tests,
run it from the non-root debian task.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 29 Dec 2019 20:46:16 +0000 (14:46 -0600)]
thelpers: call tzset() after changing TZ
Note: Although in many cases, changing the TZ environment variable
may affect the output of functions like localtime() without calling
tzset(), this behavior should not be relied on.
Rob Browning [Sat, 28 Dec 2019 20:39:44 +0000 (14:39 -0600)]
Add compat.reraise to handle python 3 syntax breakage
Add a exception reraise function to compat that will allow us to
rewrite invocations like this:
except Exception as e:
raise ClientError, e, sys.exc_info()[2]
as this:
except Exception as e:
reraise(ClientError(e))
since python 3 now provides a with_traceback() method that we can (and
must) use instead.
Put the python 2 specific formulation (shown above) in a separate
py2raise module that we can conditionally import because python 3
decided to make the python 2 code produce a syntax error.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Jan 2020 18:52:55 +0000 (12:52 -0600)]
metadata: adjust our posix1e calls for python 3
Accommodate at pylibacl's argument requirements (at least 0.5.4). It
looks like it allows bytes for the ACL() file argument, but not for
filedef:
$ cmd/bup-python
Python 3.7.5 (default, Oct 27 2019, 15:43:29)
[GCC 9.2.1 20191022] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import posix1e
>>> posix1e.ACL(file=b'README.md')
<posix1e.ACL object at 0x7fa7bd5cee70>
>>> posix1e.ACL(file='README.md')
<posix1e.ACL object at 0x7fa7bd5a8bb0>
>>> posix1e.ACL(filedef=b'.')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 5 must be str, not bytes
And it expects a string for the to_any_text() prefix argument, but
rquires bytes for the sparator:
>>> posix1e.ACL(file='README.md').to_any_text(prefix=b'', separator='')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 1 must be str, not bytes
>>> posix1e.ACL(file='README.md').to_any_text(prefix='', separator='')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 2 must be a byte string of length 1, not str
projects / bup.git / log