Johannes Berg [Thu, 28 May 2020 20:05:45 +0000 (22:05 +0200)]
tests: partially convert to pytest
Convert all the python-based library unit tests to pytest,
and add a pytest.ini to only look in the test/ directory.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: run pytest outside wvtest check wrapper so the
output is visible; restore some of buptest.py and wvtest.py so the
other tests can still run; rename pytest shim to wvpytest to
distinguish from ./wvtest.py.; add ./pytest to centralize common
options; switch .wvtest import to wvtest now that lib/test is a proper
module dir; narrow pytest.ini test dirs to test/int and test/ext; move
sort-z to test/bin; fix some dependencies in dev/prep-for-*]
Rob Browning [Sun, 1 Nov 2020 20:03:41 +0000 (14:03 -0600)]
Add pytest context manager to save/restore test environment
Without this, test_git.py could fail because test_ftp had left its
GIT_DIR lying around, and other tests could fail because it changed
and didn't restore the working directory. Make sure to call tzset()
after any TZ changes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 27 May 2020 17:58:38 +0000 (19:58 +0200)]
tests: move all tests to test/
Instead of having some testing code interleaved with bup's
real code in lib/bup/test/ and lib/bup/t/ move everything
related to testing into test/, which is also renamed from
just t/.
This aligns with pytest for future changes, and, almost more
importantly, cleans up the lib/ directory to only have the
operational code.
The latter is necessary to properly 'libify' bup in order to
be able to use tox properly to test with different versions
of python in venv later.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: also move utilities like compare-trees to dev/;
make test/lib a test-specific module tree and have wvtest.py include
it; use test/ext/ and test/int/ (external and internal) as the test
dir names.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 25 Nov 2020 20:37:01 +0000 (14:37 -0600)]
cirrus: allow failures on macos with python 2
It looks like macos has changed something that makes it very difficult
to detect and use the *utime* related functions. Apparently Emacs
just decided to live without it, and Python had to go to substantial
lengths to sort it out. So until/unless we have a reasonable fix,
just mark that combination as expected to fail.
Python 3 should work fine since we rely on it to sort out the problem
for us.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 25 Nov 2020 20:33:57 +0000 (14:33 -0600)]
cirrus: make macos image match reality (catalina)
Apparently it's actually choosing catalina anyway (see the labels
during the runs), and that appears to be all that's supported now at
https://cirrus-ci.org/guide/macOS/
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 24 Nov 2020 00:21:00 +0000 (18:21 -0600)]
Depend on python 3 for utime when using 3
Since python 3.3+ now claims to implement what we need (follow
symlinks, ns resolution), rely on it for utime when possible in
preference to our own C code in _helpers.
This should fix new failures on macos due to changes to the
availability of utimensat, etc.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
test.sh: set LC_ALL=C for git fsck to avoid localized output
Observed on Ubuntu 20.04 with German as the primary language. The test
clearly parses output in English and since adapting the parsing to the
language of the test environment is not what the test tests,
specifying the language is the way to go.
The original problem can be reproduced via
LC_ALL=de_DE.utf8 ./wvtest run t/test.sh
if your system has de_DE.utf8 avalilable.
Signed-off-by: Karl-Philipp Richter <krichter@posteo.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message; set LC_ALL=C just for
fsck rather than setting LANGUAGE globally] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 19 Nov 2020 20:56:31 +0000 (21:56 +0100)]
test-meta: use grep -a on xstat/meta
xattr data may contain arbitrary bytes, including NUL.
If we use grep on such data without -a, then it just
prints "Binary file (standard input) matches" and we
haven't tested anything.
Use grep -a for the appropriate invocations to avoid
this issue, we only care about ascii lines here.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 20 Nov 2020 07:29:02 +0000 (01:29 -0600)]
write_idx: fix handling of signed/unsigned values
Coerce the incoming Py_ssize_t count directly to the required uint32_t
while checking for overflow. The underlying problem was identified
when we hit two sign-compare warnings on (32-bit) platforms where
ssize_t is 32 bits.
Explicitly add -Wsign-comapare to CFLAGS, just to be sure it's
included.
Reported-by: Dominique Leuenberger Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Sep 2020 22:15:46 +0000 (17:15 -0500)]
gc: always clean up the packwriter; abort on errors
Make sure to always close or abort the packwriter. Abort on errors
under the assumption that any packfile in progress should only contain
blobs duplicated from the packfiles that were being garbage collected
to produce it, and those packfiles should never be removed until we've
finished writing and closing the one they're being swept into.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
floor() returns a float, even though it just rounded down,
so we need to convert to an int() to avoid a deprecation
warning here.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 27 Aug 2020 21:31:57 +0000 (23:31 +0200)]
bup: pass TTY width through to sub-command and remote
As we rewrite the sub-command's stderr, it cannot determine
the width. As a result, e.g. "bup ls" will always columnate
only to 70 columns, instead of using the entire width. Fix
this by passing the real width through in an environment
variable, just like we already pass through the fact that
it's a TTY.
Also pass it through on the ssh command line so that if you
run 'bup on ... ls' it actually behaves right.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 27 Aug 2020 20:25:16 +0000 (22:25 +0200)]
io: fix path_msg on python2
We get the binary representation here on both python 2 and 3,
but on 2 we really don't need to do any decoding since we must
print it as 'str'. Decoding also fails if the input is valid
UTF-8, which seems strange, but even if it worked we actually
don't want unicode objects here to print, we'd have to encode
them again to print.
Reported-by: Henninger Henningstone <henningstone@gmx.de> Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
features: show version number of the Python interpreter
This can be handy for debiugging, especially if the interpreter used
(like /usr/bin/python3) is a symlink to the actual Python interpreter
(/usr/bin/python3.7) and/or doesn't include the "micro" version at all.
Signed-off-by: Alexander Barton <alex@barton.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 13 Aug 2020 13:20:33 +0000 (15:20 +0200)]
tests: tclient: fix buffering behaviour in the test
If the objects we write are small enough to be at least
partially buffered instead of written out to the server
connection, the test will hang in the loop as the data
hasn't made it to the server yet, which therefore can't
make a suggestion yet.
In normal non-testing scenarios this isn't an issue as
the connection is flushed before every new read(), and
thus if the client actually needs to wait for something
from the server, it'll have flushed before. Here, we're
directly poking at the internals by using has_input(),
and nothing causes a flush, causing the above scenario.
Fix this by some more ugly poking at the internals and
flush the connection before we go into the loop.
While at it, also avoid hanging there forever and break
out of the loop if one second passed without receiving
anything from the server - that should be long enough
for it to hash the two objects and respond. Also, add a
time.sleep() there to avoid busy spinning which takes
CPU cycles from the server that needs to be hashing the
object.
Reported-by: Robert Edmonds <edmonds@debian.org> Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Johannes Berg [Tue, 4 Aug 2020 14:46:16 +0000 (16:46 +0200)]
save: make the py3 save commit message args match py2
We shouldn't use sys.argv, it's missing the actual arguments
now. Also, encode it but drop the "b" on the bytes (if present,
i.e. python3) to make this look the same as on python2.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: make formatting decision via py_maj; adjust
commit message] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 5 Aug 2020 19:17:13 +0000 (21:17 +0200)]
version: fix --date argument for py3
This has an issue with str vs. bytes, fix it.
Reported-by: Mark J Hewitt <mjh@idnet.com> Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 5 Aug 2020 07:54:55 +0000 (09:54 +0200)]
help: fix for python3
Fix "bup subcommand --help" to properly compare and start
the man viewer.
Reported-by: Robert Edmonds <edmonds@debian.org> Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Jul 2020 21:28:46 +0000 (23:28 +0200)]
ftp: fix ls arguments for python3
During the conversion to python 3, ftp ls could no longer
get option arguments, because we're now passing *bytes* to
the option parser that still expects *str*.
Fix this by fsdecode()'ing the bytes into str, which then
causes the option parsing to work properly, and the result
will be fsencode()'d again via argv_bytes() for usage.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Mon, 27 Jul 2020 21:15:16 +0000 (23:15 +0200)]
hlinkdb: always load as bytes on python3
We really need this to be bytes, so always load as such.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: changed %s to %d] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Eliminate redundant check of index start against ctime
When (the first version of) this check was added 10 years ago in b4b4ef116880, it was presumably to ensure that "index; save; touch;
index" in the same second would flag a file as needing saving.
Since then, tmax has been added in the indexing process to solve the
problem by capping timestamps stored in the index. This capping means
that a file with ctime in the same second as an indexing start will
be picked up on both the indexes in the example above - there's no need
to special case the check.
Signed-off-by: Aidan Hobson Sayers <aidanhs@cantab.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 17 Jul 2020 19:42:16 +0000 (21:42 +0200)]
web: don't re-resolve item in listing
We already have an item, we just need its metadata. There's no
need to re-resolve it. Somehow, resolving it again is also very
slow for large directories (perhaps re-reading metadata again
and again?), and this significantly speeds up things.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 19 Jul 2020 19:28:36 +0000 (14:28 -0500)]
Add bup-features to report status; drop ACL warnings
Add a new "bup features" command that reports information about the
status and capabilities of the current installation.
Given that, and the new summary at the end of the ./configure output,
drop the platform specific warnings, which were a bit odd anyway,
since they wouldn't warn you on say Linux if we found libacl, but it
didn't actually have all the features we required.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 5 Jul 2020 19:40:40 +0000 (14:40 -0500)]
Avoid varying git archive content for ref; rework versioning
Don't include the log --decorate names in the archive via export-subst
because they can change after the fact when new tags or branches are
added for a given hash -- for example, when we created the 0.30.x
branch after tagging 0.30.1. Archives retrieved before the branch was
created would have a different set of NAMES in _release.py.
Move _release to source_info and add an optional checkout_info module.
source_info contains the (no longer variable) export-subst commit hash
and date values, and checkout_info contains the same data for a git
checkout. Automatically update checkout_info whenever we're at the
top level of a git source tree, but don't include it in the archives.
Record the base version in version.py explicitly as either a release
version like 0.31, for an actual release (which must be committted
before tagging the release), or a development version like
0.31~ (indicating a version that's always less than 0.31).
Rework bup version to report the lib/version for an actual release, or
that version suffixed with the commit hash when running a non-release,
and add a "+" when uncommitted modifications are detected. For
example:
Drop the --tag argument from bup version since the tags are variable,
and you can always ask git to describe the hash via
git describe --always HASH
Add dev/refresh, similar to moreutils sponge, to handle file creation
safely, something we may want to deploy more widely (e.g. instead of
the $$/PID based tempfiles in the Makefile).
Thanks to Greg Troxel for reporting the problem.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 30 Jun 2020 23:50:17 +0000 (18:50 -0500)]
Drop pwd/gid reentrant function variants for now
From the POSIX pages, it sounded like _SC_GETPW_R_SIZE_MAX, etc. were
upper limits, but other docs, and observed behavior (reported ERANGE
failures) indicate that it's more likely they're just suggestions. As
a result, we'd either need larger hard-coded buffers, or a
reallocate/retry loop if we kept them.
Instead, since we're not at all thread-safe and aren't likely to be
anytime soon, the dynamic allocations are just less efficient and more
complex, so back off to the non-reentrant flavors for now.
Thanks to kd7spq for reporting the problem and Johannes Berg for
tracking down the cause and help settling on this solution.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 27 Jun 2020 16:47:31 +0000 (11:47 -0500)]
drecurse: use portable S_ISDIR() instead of deriving S_IFMT
In some simple testing, there didn't appear to be any notable,
consistent performance difference, and the S_IFMT(0xffffffff) call
fails with an OverflowError on at least macos.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 21 Jun 2020 17:46:50 +0000 (12:46 -0500)]
configure: test for functional readline more carefully
Apparently on (cirrus) macos, just testing for the ability to compile
readline.h isn't sufficient because configure ends up selecting the
built-in readline (which is insufficient) rather than the one we
specifically installed via brew. More specifically, the built-in
readline has the wrong prototype for rl_completion_entry_function
which causes this error:
_helpers.c:2096:38: error: incompatible function pointer types assigning to 'Function *' (aka 'int (*)(const char *, int)') from 'char *(const char *, int)' [-Werror,-Wincompatible-function-pointer-types]
rl_completion_entry_function = on_completion_entry;
^ ~~~~~~~~~~~~~~~~~~~~
So change the test for an acceptable readline to check that
specifically, which is a better test for all the platforms.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
On some platforms -Wstrict-prototype is now the default, and readline
includes prototypes like "int foo()" rather than "int foo(void)", so
for now, just suppress those warnings for the readline includes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 21 Jun 2020 16:29:23 +0000 (11:29 -0500)]
configure-sampledata: only create random paths if asked
Stop creating randomized paths in t/sampledata/ by default. I'd
originally just added this to allow some quick testing, and while it
now appears to be fine on Linux/ext4, it's too aggressive to be the
default, so hide it behind a BUP_TEST_RANDOMIZED_SAMPLEDATA_PATHS
environment variable.
Among other things, make-random-paths just crashes on cirrus macos,
and cirrus freebsd was having (different) trouble. It might also have
been macos where test-import-duplicity.sh failed on compare-trees
mismatches. Not sure whether that was an issue with bup, rsync, or
duplicity.
We'll want to restore broader randomized path testing, but likely via
a less blunt instrument, since placing the paths in t/sampledata
affects any test that relies on it, and existing testing constructs
like
WVPASSEQ ... $(... | wc -l)
are completely incompatible.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 20 Jun 2020 16:55:53 +0000 (11:55 -0500)]
Use pkg-config opportunistically
Use pkg-config's --cflags and --libs when they're available for
libreadline or libacl, but don't require pkg-config. When it's not
found, just check for the libraries with a test compile.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 19 Jun 2020 05:19:07 +0000 (07:19 +0200)]
grp_struct_to_py(): fix error handling
Both getgrgid_r() and getgrnam_r() *return* an error number
on failures, and don't store it to errno. Thus, rc will not
be less than zero, and we need to set errno before we can
create a python error from it.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Fri, 1 May 2020 22:52:53 +0000 (00:52 +0200)]
DESIGN: document the actual hashsplit algorithm
The hashsplit algorithm, when used for the fanout, has a quirk
that appears to be due to an implementation bug.
In order for splitting to occur, the lowest 13 (BUP_BLOBBITS)
bits of the csum need to be 1. Then, per DESIGN, the next bits
that are 1 are used for the fanout. However, the implementation
doesn't actually work this way. What actually happens is that
the lower 13 bits need to be ones:
........1'1111'1111'1111
Then, the DESIGN document states that the next bits that are 1
should be used for the fanout:
....'111_'____'____'____
However, the implementation actually ignores the next bit ('x')
....'11x_'____'____'____
and it doesn't matter whether that's set to 0 or 1, the fanout
will be based on the next higher bits (marked '.' and '1').
Fix this in the DESIGN documentation rather than changing the
algorithm as the latter would cause a save of an identical file
to completely rewrite the tree objects that make up the file,
due to different fanout behaviour.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Johannes Berg [Tue, 26 May 2020 22:29:23 +0000 (00:29 +0200)]
metadata: fix test failure with xattrs
The test_apply_to_path_restricted_access() is broken because
the expected string doesn't take into account that it's now
a u'' string (on python2) due to path_msg(), but this doesn't
appear on any non-selinux systems because the original file
never has any xattr, so the test passes, just not on my Fedora
system.
Change the expected message and remove the quote entirely as
it's different between python 2 and 3, and also try to set an
xattr in the test - if that fails, just continue as it used
to be without reading/setting it.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 30 May 2020 21:55:46 +0000 (23:55 +0200)]
metadata: port ACL support to C
Use some own C code instead of the posix1e python bindings, as those
don't have correct 'bytes' support (at least right now), which means
that we cannot use them with arbitrary file, user and group names.
Our own wrappers just use 'bytes' throughout.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: adjust to rely on pkg-config] Reviewed-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 30 May 2020 02:50:25 +0000 (21:50 -0500)]
Split src tree python use to config/bin/python and dev/bup-python
Replace cmd/bup-python with config/bin/python, which is just a symlink
to the configured python, and dev/bup-python, which is what "bup
python" used to be. Adjust all the code to use config/bin/python when
we can (i.e. when we don't need bup modules), and dev/bup-python
otherwise. Drop "bup python", since we don't need it anymore.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Now that the last subcommand (web) has been ported to Python 3, we
at least some randomized binary test coverage, and we think we've
addressed all the Python 3 issues we know of, remove the
BUP_ALLOW_UNEXPECTED_PYTHON_VERSION guardrail.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 25 May 2020 19:46:52 +0000 (14:46 -0500)]
Stop forcing LC_CTYPE=ISO-8859-1
Now that we've made adjustments to work around all the Python 3
problems with non-Unicode data (argv, env vars, readline, acls, users,
groups, hostname, etc.), and added randomized binary path and argv
testing, stop overriding the LC_CTYPE since that should no longer be
necessary.
Thanks to Johannes Berg for nudging me to consider whether we might
now be in a position to do this (with a bit more work), and for quite
a bit of help getting all the precursors in place once we thought it
was feasible.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 7 Jun 2020 14:05:26 +0000 (09:05 -0500)]
Bypass Python 3 glibc argv problems by routing args through env
Until/unless https://sourceware.org/bugzilla/show_bug.cgi?id=26034 is
resolved by Python or GNU libc, sidestep the problem, which can crash
Python 3 during initialization, with a trivial sh wrapper that diverts
the command line arguments into BUP_ARGV_{0,1,2,...} environment
variables, since those can be safely retrieved.
Add compat.argvb and compat.argv and populate them at startup with the
BUP_ARGV_* values. Adjust all the relevant commands to rely on those
vars instead of sys.argv.
Although the preamble say "rewritten during install", that's not in
place yet, but will be soon (when we drop LC_CTYPE and rework
bup-python).
Thanks to Johannes Berg for suggesting this, and help figuring out the
details.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 28 Apr 2020 21:35:33 +0000 (23:35 +0200)]
hashsplit: avoid cat_bytes() if possible
If our current buffer is empty, there's no need to cat_bytes()
it with the new buffer, we can just replace the empty one with
the new one. This saves the memcpy() in many cases. Especially
if the whole file was read in one chunk (by bumping up the read
size) this saves a lot of time.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sun, 17 May 2020 19:43:45 +0000 (21:43 +0200)]
tests: web: also add some invalid UTF-8
The '¡excitement!' really tests only valid UTF-8 since it
comes from the original bash file, add another test that
explicitly creates a byte sequence that is invalid utf-8
and ensures that this is preserved properly as well.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 7 Jun 2020 20:32:10 +0000 (15:32 -0500)]
Wrap readline oursleves to avoid py3's interference
We don't want Python to "help" us by guessing and insisting on the
encoding before we can even look at the incoming data, so wrap
readline ourselves, with a bytes-oriented (and more direct) API. This
will allows us to preserve the status quo for now (and maintain parity
between Python 2 and 3) when using Python 3 as we remove our LC_CTYPE
override.
At least on Linux, readline --cflags currently defines _DEFAULT_SOURCE
and defines _XOPEN_SOURCE to 600, and the latter conflicts with a
setting of 700 via Python.h in some installations, so for now, just
defer to Python as long as it doesn't choose an older version.
Thanks to Johannes Berg for fixes for allocation issues, etc. in an
earler version, and help figuring out the #define arrangement.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 31 May 2020 19:10:11 +0000 (14:10 -0500)]
distutils: handle CFLAGS and LDFLAGS directly
Otherwise it places LDFLAGS in the middle of the link arguments,
before lib/bup/*.o which means we can't add lib
dependencies (e.g. -lreadline). Pass the libs directly and specify them
via the appropriate extra_* arguments.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 30 May 2020 19:10:02 +0000 (21:10 +0200)]
bup: add own gethostname() wrapper
This is necessary because python3 insists that hostnames should
be utf-8, which is a rather questionable assumption.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: don't define HOST_NAME_MAX if it's not already] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Thu, 28 May 2020 05:44:16 +0000 (00:44 -0500)]
pwdgrp: add C helpers to get user/group bytes directly
Thanks to Johannes Berg for fixing some bugs in an earlier revision.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Sat, 30 May 2020 19:41:10 +0000 (21:41 +0200)]
metadata: don't modify ACL list when writing
ACLs should be stored as a two-entry list on files, and four-entry
list for directories. Unfortunately, when writing, we expand the
two-entry list for files to four, because the metadata format is
always with four entries.
However, on reading, we trim the last two empty entries, so that
we can end up in an inconsistent situation: On a metadata entry
for a file that has been written already, it will still have four
entries, and that won't compare correctly etc.
This isn't an issue today because we only ever do the compare
in restore, where we didn't load from disk but from the meta-
data in the repository, which always starts out four entries.
Still, fix the inconsistency and don't erroneously extend the
list when writing.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Mon, 25 May 2020 19:55:46 +0000 (14:55 -0500)]
Move cmd to lib/ and reverse symlink
This prepares for removal of the bup-python wrapper. Given this
change we'll be able to have the same sys.path in the source tree and
install tree, and so won't have to go back to mangling that during
installs.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 13 May 2020 20:22:04 +0000 (22:22 +0200)]
index: fix -H option
hexlify(ent) doesn't work, that needs to be ent.sha. Fix it.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log