Rob Browning [Tue, 31 Dec 2019 18:19:39 +0000 (12:19 -0600)]
INTEGRAL_ASSIGNMENT_FITS: actually provide return value for clang
Apparently clang does need the pragmas, so either I tested it
incorrectly before, or my local clang is different. It looks like
clang doesn't ignore the pragmas as far as the expression result value
is concerned, so explicitly put the value at the end.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 29 Jan 2020 19:09:49 +0000 (20:09 +0100)]
tests: vint: test EOFError after first byte
Since the first byte is handled separately for the sign bit,
validate that we also get an EOFError if there are a few
bytes but the last one also has the 0x80 bit set.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 29 Jan 2020 19:09:27 +0000 (20:09 +0100)]
vint: remove unnecessary condition
"if c:" can never be false, since we checked before.
Remove the extra condition.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:23:18 +0000 (00:23 +0100)]
client: import socket
Fixes: 7ce8041f0345 ("Teach bup about URLs and non-ssh remotes") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:56 +0000 (00:22 +0100)]
client: import atoi
Fixes: 22d01e1a8077 ("If you specified the port number on the command line, convert it to an int.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:22:33 +0000 (00:22 +0100)]
client: import DemuxConn
Fixes: fb3bd84cfd24 ("Add DemuxConn and `bup mux` for client-server") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jan 2020 23:19:33 +0000 (00:19 +0100)]
vfs: fix finish_extract()
The 'dir' variable doesn't exist here, must be 'meta' instead.
Fixes: 0962d3904735 ("Add initial support for metadata archives.") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 31 Dec 2019 19:45:00 +0000 (13:45 -0600)]
Rework shstr to handle bytes and strings; add squote and bquote
These could be smarter, i.e '1 could become "'"1 rather than ''"'"'1',
but we can always improve it later. And add at last some tests.
Don't rely on compat.quote for strings so that we know we'll have the
same behavior for bytes and strings.
Thanks to Johannes Berg for pointing out an incorrect variable name in
a previous revision.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 22 Jan 2020 08:25:40 +0000 (09:25 +0100)]
client: fix index-cache location when there's no path
Putting my current tree into production, I noticed that the
index-cache was completely re-downloaded (taking a long time)
due to a change in storage location, which was broken in the
commit 85edc0f1c133 ("bup.client: accommodate python 3").
The "self.dir or b'None'" was in commit 85edc0f1c133
("bup.client: accommodate python 3") was clearly well-intended,
but also had the effect of transforming the empty string (which
evaluates to False) to b'None' instead, which is wrong since in
'bup on' cases there's no dir, but parse_remote() comes up with
an empty string instead of None.
Fix that and add a test that checks that the index location
without a dir is actually preserved as such.
Fixes: 85edc0f1c133 ("bup.client: accommodate python 3") Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 30 Dec 2019 19:22:20 +0000 (13:22 -0600)]
merge_into: accommodate python 3
Switch to Py_buffers to accommodate python 3, use malloc/calloc to
avoid potentially involving the GIL, and check allocation failures.
Thanks to Johannes Berg for pointing out a potential overflow on the C
side -- fixed by adjusting checked_malloc() to accept the same
arguments as checked_calloc().
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 20 Jan 2020 20:54:38 +0000 (14:54 -0600)]
cirrus: run at least one long-check
Since the cirrus root tests already take a good while longer than the
non-root tests, and we don't currently have any root-only long tests,
run it from the non-root debian task.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 29 Dec 2019 20:46:16 +0000 (14:46 -0600)]
thelpers: call tzset() after changing TZ
Note: Although in many cases, changing the TZ environment variable
may affect the output of functions like localtime() without calling
tzset(), this behavior should not be relied on.
Rob Browning [Sat, 28 Dec 2019 20:39:44 +0000 (14:39 -0600)]
Add compat.reraise to handle python 3 syntax breakage
Add a exception reraise function to compat that will allow us to
rewrite invocations like this:
except Exception as e:
raise ClientError, e, sys.exc_info()[2]
as this:
except Exception as e:
reraise(ClientError(e))
since python 3 now provides a with_traceback() method that we can (and
must) use instead.
Put the python 2 specific formulation (shown above) in a separate
py2raise module that we can conditionally import because python 3
decided to make the python 2 code produce a syntax error.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 4 Jan 2020 18:52:55 +0000 (12:52 -0600)]
metadata: adjust our posix1e calls for python 3
Accommodate at pylibacl's argument requirements (at least 0.5.4). It
looks like it allows bytes for the ACL() file argument, but not for
filedef:
$ cmd/bup-python
Python 3.7.5 (default, Oct 27 2019, 15:43:29)
[GCC 9.2.1 20191022] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import posix1e
>>> posix1e.ACL(file=b'README.md')
<posix1e.ACL object at 0x7fa7bd5cee70>
>>> posix1e.ACL(file='README.md')
<posix1e.ACL object at 0x7fa7bd5a8bb0>
>>> posix1e.ACL(filedef=b'.')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 5 must be str, not bytes
And it expects a string for the to_any_text() prefix argument, but
rquires bytes for the sparator:
>>> posix1e.ACL(file='README.md').to_any_text(prefix=b'', separator='')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 1 must be str, not bytes
>>> posix1e.ACL(file='README.md').to_any_text(prefix='', separator='')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: argument 2 must be a byte string of length 1, not str
Rob Browning [Sat, 28 Dec 2019 19:54:03 +0000 (13:54 -0600)]
Adjust metadata handling for python 3
Adapt bup.metadata for python 3 and the other bits directly affected:
bup-ftp, bup-ls, bup-meta, bup-xstat, and bup.ls.
Rename metadata detailed_str() and summary_str() to detailed_bytes()
and summary_bytes() since they aren't (and absolutely should not be)
localized. They produce output that should be suitable for
programmatic use, i.e. "bup ls | grep ...". Not sure we'll keep those
names, but they'll do for now.
Also rename fstime_to_sec_str() to fstime_to_sec_bytes() since that's
the only way we ever use it.
Make a minimal change to bup-ftp for now -- just enough to handle the
changed ls.within_repo arguments.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 27 Dec 2019 19:24:56 +0000 (13:24 -0600)]
midx: shun buffers
Rework PackMidx to avoid buffers which are reasonably heavyweight, and
will be even larger (as memoryviews) in python 3, requiring ~200
bytes. Instead, just use direct offsets into the underlying mmap --
slicing an mmap currently just produces bytes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 27 Dec 2019 19:02:42 +0000 (13:02 -0600)]
git: shun buffers in packidxes
Rework PackIdx to avoid buffers which are reasonably heavyweight, and
will be even larger (as memoryviews) in python 3 (require ~200 bytes).
Instead, just use direct offsets into the underlying mmap -- slicing
an mmap currently just produces bytes.
Store the fanout table as a homogeneous array rather than a list or
tuple with individually allocated integers.
Instead of looking up hashes one at a time, traverse the index during
gc via its iterator.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 26 Dec 2019 18:50:47 +0000 (12:50 -0600)]
cmd/bup: adapt for python 3
Make all the changes necessary for cmd/bup to work with both python 2
and 3. Given the current state, the majority of the changes handle
untangling the python 3 unicode/data conflation.
Convert to b'x' literals where needed (e.g. for path or path-derived
values), use argv_bytes to convert at least the command line values
that must not be interpreted as locale strings, and switch to the
bytes-only compat.environ.
More broadly speaking, aside from the changes we abosolutely have to
make, the general intent is for us to handle locale-specific
conversions carefully and explicitly when appropriate, not
transparently.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 26 Dec 2019 18:27:07 +0000 (12:27 -0600)]
Add "do nothing" path_msg to centralize path conversions
Add path_msg(p) and use it in cmd/bup (as a start). The intent is to
centralize the encoding of all path values that are to be included in
strings intended for "display" (e.g. stderr, which appears to have to
be a text stream in python 3).
For now, the function will do nothing -- i.e. given the currently
enforced iso-8859-1 encoding we'll just continue to produce the
original path bytes on stderr, but we may well want to make this
configurable at some point (perhaps git's quotePath algorithm might
provide a likely option), and if nothing else, using path_msg()
everywhere makes it much easier to identify and adapt the relevant
code, whatever we decide.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 1 Jan 2020 17:50:07 +0000 (18:50 +0100)]
bup: use correct bup executable in on--server
Currently, on--server loses the correct bup command in one situation:
if you run 'bup on remote' with a remote forced ssh command that's
different from the installed bup, on--server will still execute the
installed version of bup for all sub-commands.
Fix that by using bup.path.exe() in on--server for bup execution.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sun, 12 Jan 2020 23:08:25 +0000 (00:08 +0100)]
metadata: accept only fixed python-xattr in python3
This is currently broken on python3, it returns junk when
we pass bytes because it uses string and %s internally.
I made a fix for it, so we can detect here if it's fixed
(in which case the NS_USER constant is bytes, not string),
load the module only if it is indeed fixed.
We can do this test always since in python2 bytes == str
and thus isinstance('user', bytes) == True.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 12 Jan 2020 21:33:15 +0000 (15:33 -0600)]
bloom: fix logic controlling bloom regeneration
Add missing MAX_BLOOM_BITS index in the logic in bup bloom that
determines whether or not we should regenerate the filter. We never
noticed because:
$ python2
>>> 0 < {1 : 2}
True
$ python3
>>> 0 < {1 : 2}
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: '<' not supported between instances of 'int' and 'dict'
Also regnerate if the -k value differs from the existing filter's k.
Thanks to Johannes Berg for pointing out some nontrivial problems in
an earlier version.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 18 Dec 2019 19:08:26 +0000 (20:08 +0100)]
git: remove global variable ignore_midx
This can easily be kept as state in the git.PackIdxList()
class instead, so do that.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 18 Dec 2019 21:43:58 +0000 (22:43 +0100)]
get: remove extra src_repo
We already have a src_repo from the with statement, no need to
instantiate another one (that won't even be closed properly).
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
The Python documentation [0] indicates that an 'O' passed to
Py_BuildValue will have its refcount incremented. Since some elements of
the tuple created in stat_struct_to_py are pre-converted in C to
PyObjects, they already have a refcount of 1 - use 'N' to avoid
incrementing it and ensure Python can deallocate them correctly.
Rob Browning [Thu, 26 Dec 2019 00:18:32 +0000 (18:18 -0600)]
Only change LC_CTYPE for bup-python itself, not sh or subprocesses
With the previous approach, bash would complain on systems that don't
provide the ISO-8859-1 locale (python always supports it), and those
complaints would break some tests.
In any case, since the original hack was only intended to affect
bup-python, try to make that the case. Specify the LC_CTYPE only to
python via env (so bash won't see it), and then undo the environment
changes at the top of cmd/bup so they won't affect unrelated
subprocesses.
Thanks to Johannes Berg for reporting the problem and helping devise
the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 26 Dec 2019 00:08:13 +0000 (18:08 -0600)]
Fix the handling of the configure MAKE and PYTHON vars
Instead of trying to negotiate the maze of twisty quoting
passages (bash vs sh, how do you get it to work correctly from a
makefile target, etc.), drop config.vars.sh and just store each
variable value in a separate config/config.var/NAME file, and read
that file whenever we need the value, so that we should be able to
handle more or less arbitrary values, e.g.:
make PYTHON='srsly?
why?'
Along the same lines, instead of trying to figure out, in any given
case, how to quote values for sed (through various layers), add
dev/replace, which just does literal replacements of values provided
on the command line.
Also along the same lines, add dev/shquote that quotes values properly
for POSIX sh, and use it. Previously we produced bash %q quotations,
which can result in $'' bashisms, and then handed them to /bin/sh.
These changes (particularly the switch to the literal @bup_python@
replacement) also lay the groundwork for forthcoming adjustments to
cmd/bup.
Thanks to Johannes Berg for pointing out a couple of bugs, including
that I'd forgotten to use fsdecode after adding it.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 23 Dec 2019 19:56:12 +0000 (13:56 -0600)]
mux: handle python 3's change to default close_fds=True
Change mux to explicitly specify close_fds=False to deal with the fact
that python apparently changed the default in 3.2, and it absolutely
does need to keep the fds open.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 2 Jan 2020 21:34:52 +0000 (22:34 +0100)]
tests: fuse: use stat instead of relying on "ls -l" output
On my Fedora 31 system, this currently fails because the test
expects the file mode bits to be printed as "-rw-r--r--" but
I get "-rw-r--r--." instead, perhaps due to selinux.
Fix this to use stat --format instead to extract the bits we
want to check here (user, group, mode, timestamp).
While at it, I noticed that this test has some confusion with
timezones and only worked because 'ls -l' doesn't show the
time in this case since the file is so old. Fix that by making
_everything_ here use UTC, instead of just some parts.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Johannes Berg [Mon, 16 Dec 2019 09:12:24 +0000 (10:12 +0100)]
metadata: use python-xattr pyxattr compat code
Newer versions of python-xattr have pyxattr compatibility code
(since my pull request https://github.com/xattr/xattr/pull/38).
Modify bup to allow using it; prefer pyxattr since it's faster,
but sometimes due to other dependencies python-xattr might be
present already (and since it conflicts with pyxattr, it's not
possible to install both.)
Also add a .cirrus.yml task to test this on Debian.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Johannes Berg [Fri, 13 Dec 2019 20:37:16 +0000 (21:37 +0100)]
man pages: fix some formatting
I noticed one place where -- was used instead of \--, leading to a
emdash instead of two hyphens, but then I looked and realized this
was not just a single place; fix all of those.
While at it, I saw some --keep options being referenced without
backticks, fix that too.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Rob Browning [Fri, 18 Oct 2019 04:22:59 +0000 (23:22 -0500)]
merge_iter: don't compare generators with < via heapq
Previously we were implicitly comparing (entry, reader_generator) with
"<" in the merge_iter heapq. In python 2, the generators would just
compare as pointers, which was fine, because we only cared about the
ordering of the initial entries.
However, in Python 3 comparing generators provokes a TypeError, so
create a trivial helper class that can carry the two items, but only
compare the entries via the < operator.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 14 Oct 2019 23:48:24 +0000 (18:48 -0500)]
Make adjustments to fix tbloom for python 3
Make the changes needed to get the tbloom tests passing with python 3.
Change the C helper argument conversions to types that work with each
version (there didn't appear to be a good, shared common format string
type). Switch to a Py_buffer in some places because python 3 didn't
appear to have a conversion that would convert all the binary
inputs (buffer/memoryview/...) to a plain C byte array the way python
2 can.
Change all the other typical things, i.e. byte paths, '' to b'', and
// for integral division, etc.
With this, tbloom should pass with both python major versions (python
3 version tested was python 3.7.5).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 3 Nov 2019 21:10:41 +0000 (15:10 -0600)]
Require LC_CTYPE to be iso-8859-1 for Python 3
Add a guard to compat (for now) that will crash if the LC_CYTPE isn't
ISO-8859-1 (aka Latin-1) when running under python3. We definitely
need this for handling argv safely (with at least python 3.7), and
since we're planning to require this arrangement anyway, let's start
now.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 10 Sep 2019 06:56:02 +0000 (01:56 -0500)]
Prevent Python 3 from interfering with argv bytes
Python 3 insists on treating all arguments as Unicode, and and if the
bytes don't fit, it shoehorns them in anyway[1]. We need the raw,
original bytes in many cases (paths being the obvious example), and
Python claims they can be extracted via fsdecode.
But experimentation with 3.7 has demonstrated that while this is
necessary, it is not sufficient to handle all possible binary
arguments in at least a UTF-8 locale. The interpreter may crash at
startup with some (randomly generated) argument values:
Fatal Python error: _PyMainInterpreterConfig_Read: memory allocation failed
ValueError: character U+134bd2 is not in range [U+0000; U+10ffff]
Current thread 0x00007f2f0e1d8740 (most recent call first):
Traceback (most recent call last):
File "t/test-argv", line 28, in <module>
out = check_output(cmd)
File "/usr/lib/python3.7/subprocess.py", line 395, in check_output
**kwargs).stdout
File "/usr/lib/python3.7/subprocess.py", line 487, in run
output=stdout, stderr=stderr)
To fix that, always set the encoding to ISO-8859-1 before launching
Python, which should hopefully (given that ISO-8859-1 is a single-byte
"pass through" encoding), prevent Python from interfering with the
arguments.
Add t/test-argv to perform randomized testing for clean argv
pass-through. At the moment, with Python 3.7.3, if I disable the code
before the python exec in cmd/bup-python, this test eventually
provokes the crash above (though not on every run).
[1] https://www.python.org/dev/peps/pep-0383/
Thanks to Aaron M. Ucko for pointing out LC_ALL had been overlooked in
an earlier version of this patch, and would have undone the
adjustments.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 22:46:38 +0000 (17:46 -0500)]
Simplify bup startup process
Now that we've moved bup to cmd/, we adjust bup-python at install
time, and we can rely on that to find the module and cmd dirs,
simplify the startup process. Let bup-python handle all the
PYTHONPATH settings, etc., and drop BUP_MAIN_EXE, relying instead on a
path.exe() that finds the binary via path.__file__.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 21:47:16 +0000 (16:47 -0500)]
Move bup to cmd/ and symlink ./bup to cmd/bup
Move main.py to cmd/bup and make ./bup a permanent symlink to it.
Make the installed bup executable a relative symlink to the
LIBDIR/cmd/bup file. Always use the location of the bup executable as
a way to locate the other subcommands and the lib and resource
directories, either when running from the source tree or from an
install tree.
This work finishes the switch to have all the commands to rely on the
bup-python wrapper so that we can establish some norms we'll need to
support Python 3. It should also allow us to simplify the main.py
startup process.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Oct 2019 22:23:42 +0000 (17:23 -0500)]
Install bup-python and rely on it everywhere
Add a cmd/python-cmd.sh template and use it to generate
cmd/bup-python. Install bup-python alongside all the other commands
and stop rewriting the command #! headers at install time, so that
they'll use it.
We're setting up all of the commands to rely on bup-python so that
we'll have a common place to establish some norms that we'll want in
order to support Python 3.
Though in fact, you can't actually run the python-based subcommands
independently because they don't have the right PYTHONPATH, etc.
That's something we'll also fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 20 Oct 2019 17:49:14 +0000 (12:49 -0500)]
Move pwd grp functions to pwdgrp module; require binary fields
Move the pwd and grp related functions in helpers to a new more
focused pwdgrp module and adjust them (for py3) to require and provide
binary (bytes) values for the appropriate fields: name, passwd, etc.
These fields are arbitrary bytes with no encoding information on
platforms like Linux and *BSD, so must be handled that way.
Create our own Passwd and Group classes to carry the data, and make
them __slots__ based since they're quite a lot smaller, even than
namedtuples:
https://justmachinelearning.com/2019/07/29/python-consumes-a-lot-of-memory-or-how-to-reduce-the-size-of-objects/
See the DESIGN section on Python 3 strings for additional information.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 20 Oct 2019 17:46:37 +0000 (12:46 -0500)]
username/userfullname: use struct names
Use the pwd names, not indexes. That's clearer and it'll support
upcoming py3 compatibility work. While we're at it, start making it
clear these are binary (bytes).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 30 Nov 2019 18:09:04 +0000 (12:09 -0600)]
_helpers: fix -Wsign-compare warnings
Python 3 adds -Wsign-compare to the compiler arguments. Address the
issues that this causes gcc to warn about. Disable the warnings for
the INTEGRAL_ASSIGNMENT_FITS macro since it's intentionally doing
something that the compiler can't tell is OK (assuming it is), add
some simple tests, and change the types and code in other places to
avoid the suspect comparisons. Currently clang doesn't produce the
warnings.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 29 Nov 2019 19:16:10 +0000 (13:16 -0600)]
Makefile: append python 2 tests to common lists
Instead of maintaining two separate lists for python 2 and python 3
tests, just have one list of tests that work in both versions, and
then selectively append the tests that only work in python 2 when
appropriate.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 28 Nov 2019 08:01:02 +0000 (02:01 -0600)]
Remove inefficient (or will be) uses of buffer
On amd64 sys.getsizeof(b'') is a bit under 40 bytes across python 2/3,
while a buffer(b'') adds 64 bytes in python 2, and the memoryview(b'')
that replaces it in python 3 adds 200. So just copy the bytes unless
the added overhead is worth it.
And while we're here, fix a few more python 3 str/bytes compatability
issues, and remove some redundant offset arithemetic by leaning on
range() a bit more. (Not that it likely matters, but aside from being
simpler, this is apparently more efficient too, because it moves more
of the work to C).
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 29 Nov 2019 19:15:30 +0000 (13:15 -0600)]
hashsplit: join buffers more efficiently
Drop buffer_concat in favor of join_bytes, since we can use b''.join()
to join the two put() chunks in python 3, which should be notably more
efficient than copying the memoryview bytes before joining them.
In python2, also use join when we just have bytes, otherwise
accommodate the fact that you can't join bytes and buffers there, and
you can't append bytes to a buffer. The case we currently care about
should still end up being handled the same as befoere (in the first
clause), i.e. buf + bytes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 29 Nov 2019 20:58:36 +0000 (14:58 -0600)]
fsck: report par2 parallel detection status
When attempting to detect whether or not par2 supports parallel
processing, don't suppress its stdout for now so that we'll see
anything unexpected (should be empty). When verbose, log what we
think we've detected along with any unexpected par2 stderr output.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>