Rob Browning [Wed, 2 Dec 2020 01:43:13 +0000 (19:43 -0600)]
bup: add support for "internal" subcommands
Look for the existing something-cmd.py commands as both
lib/cmd/bup-something and lib/bup/cmd/something.py, and don't load the
latter as a module unless specifically allowed, so that we can create
separate rename commits as we convert each relevant external command
to an internal command. This should make it easier to see what
actually changed in the git history, e.g. with --ignore-space-change.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 6 Jan 2021 01:43:50 +0000 (19:43 -0600)]
prep-for-macos-build: don't run brew update
The update was causing tests to fail, and the previous attempt to
address the issue e244aa8ec2ea4b754d6242ee81dd0d3c6dace543 didn't.
Thanks to Johannes Berg proposing the fix.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 31 Dec 2020 22:07:10 +0000 (23:07 +0100)]
index: add newline to 'corrupt index' message
This otherwise prints the traceback afterwards starting
on the same line, which looks odd.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 31 Dec 2020 21:29:38 +0000 (22:29 +0100)]
index: remove pointless git.repo() invocation
Since the only way we can get to clear_index() is with
indexfile having previously been assigned
indexfile = git.repo(b'bupindex')
since opt.clear and opt.indexfile are explicitly mutually
exclusive, there's no point in calling git.repo() again,
it has no effect.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
bup-web: handle 4 item getsockname() result for IPv6 sockets
So bup-web will create a new network socket to listen on for HTTP
connections and it may use either IPv4 or IPv6 (or both) addresses,
depending on the configuration of the system. This is done using the
tornado.netutil.bind_sockets().
Currently, bup-web does not restrict the address family to either
AF_INET/AF_INET6 and according to the documentation of
[tornado](https://www.tornadoweb.org/en/stable/netutil.html?highlight=netutil#tornado.netutil.bind_sockets)
it will use both address families and listen for connections on IPv4 and
IPv6 addresses.
> Family may be set to either socket.AF_INET or socket.AF_INET6 to
> restrict to IPv4 or IPv6 addresses, otherwise both will be used if
> available.
However, when your system has only IPv6 enabled, the print() statement
may fail with:
Traceback (most recent call last):
File "/opt/bup/lib/bup/cmd/bup-web", line 310, in <module>
print('Serving HTTP on %s:%d...' % sockets[0].getsockname())
TypeError: not all arguments converted during string formatting
For systems that have IPv4 as well as IPv6 enabled, it may or may not
fail, depending on whether the first returned socket by getsockname() is
for AF_INET6 or AF_INET4
The reason is, getsockname() for a AF_INET6 connections returns a tuple with
4 items: `(host, port, flowinfo, scope_id)`, while for AF_INET4
addresses, it contains only `(host, port)`.
Since we are only interested in the first 2 items, make sure to return
only the first 2 items, which are `host` and `port` for IPv4 (AF_INET)
and IPv6 (AF_INET6).
Signed-off-by: Christian Brabandt <cb@256bit.org> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit summary]
Johannes Berg [Tue, 29 Dec 2020 21:26:40 +0000 (22:26 +0100)]
restore: fix error message for inaccessible items
This actually shows up during tests, but it's hidden as it's
an error where we _expect_ bup to fail (and don't check why).
Fix the message in this case.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 22 Dec 2020 17:17:45 +0000 (11:17 -0600)]
Replace atoi with explicit checks; rm unused atof
Replace atoi with explicit checks, making the intention a bit clearer
at each call site, and avoiding trouble we've had in the past with the
default to 0 for any unrecognized input.
This does make us stricter about the values (and the environment),
i.e. BUP_TTY_WIDTH=random-noise will now cause a crash.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Mon, 21 Dec 2020 20:45:55 +0000 (21:45 +0100)]
ssh: don't pass through bad BUP_TTY_WIDTH
If there's no BUP_TTY_WIDTH variable already, then environ.get()
will result in None, which cannot be converted to an integer so
we get 0. Then we pass 0 to the other side in the BUP_TTY_WIDTH
environment variable.
Unfortunately, 0 is a really bad default, since textwrap cannot
work with a width of 0, causing tracebacks such as the one Mark
J Hewitt reported:
https://groups.google.com/g/bup-list/c/rfIw3STN2EY/m/C5Y3UrMIAgAJ
Fix this by not passing it through if there's no good value.
Reported-by: Mark J Hewitt <mjh@idnet.com> Reported-by: Alexander Barton <alex@barton.de> Fixes: cd3647c81a25 ("bup: pass TTY width through to sub-command and remote") Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 20 Dec 2020 01:36:09 +0000 (19:36 -0600)]
Prefer python 3, and mention intent to drop python 2 support
For now, set the minimum supported 3 version at 3.7 since I just
tested that on a debian/buster system, and update the installation
instructions after seeing what was required to run the tests from
scratch there. Continue to search for 3.6 and 2.6 for now, since no
one's complained, and we can always lower the requirement later if it
seems appropriate.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Tue, 4 Aug 2020 14:49:36 +0000 (16:49 +0200)]
test-web: redirect stdio to make test work standalone
Python (at least 3) behaves differently if somehow
stdio is connected to the terminal, and external
SIGTERM then doesn't cause the same behaviour as
actually pressing Ctrl-C. Redirect the output to
a file to make the test run standalone.
Print the file if any errors occur so that if there's
anything interesting in there, we can see it.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 28 May 2020 21:14:53 +0000 (23:14 +0200)]
Migrate the remaining (standalone) test/ext tests to pytest
Enable the test/ext/conftest.py collector to allow pytest to handle
all of the test/ext standalone tests too.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: show the pytest invocations; switch to use
test/ext/conftest.py collector; use the release mark in check and
distcheck; augment commit message; augment the information in
HACKING] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 8 Nov 2020 18:13:55 +0000 (12:13 -0600)]
Add pytest collector for all test/cmd/test-* files
Add a collector for all the test/cmd tests that runs them as
individual subprocesses. Add support for a pytest "release" mark and
apply it to the test-release-archive test. Disable the collector
until the upcoming changes that rely on it are in place.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sun, 15 Nov 2020 17:44:27 +0000 (11:44 -0600)]
test-prune-older: accommodate unified_diff break in python 3.5
Apparently python 3.5 changed unified_diff to crash when given bytes
arguments, and added a new diff_bytes function. Use the latter when
the version is >= 3.5.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 28 May 2020 20:05:45 +0000 (22:05 +0200)]
tests: partially convert to pytest
Convert all the python-based library unit tests to pytest,
and add a pytest.ini to only look in the test/ directory.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: run pytest outside wvtest check wrapper so the
output is visible; restore some of buptest.py and wvtest.py so the
other tests can still run; rename pytest shim to wvpytest to
distinguish from ./wvtest.py.; add ./pytest to centralize common
options; switch .wvtest import to wvtest now that lib/test is a proper
module dir; narrow pytest.ini test dirs to test/int and test/ext; move
sort-z to test/bin; fix some dependencies in dev/prep-for-*]
Rob Browning [Sun, 1 Nov 2020 20:03:41 +0000 (14:03 -0600)]
Add pytest context manager to save/restore test environment
Without this, test_git.py could fail because test_ftp had left its
GIT_DIR lying around, and other tests could fail because it changed
and didn't restore the working directory. Make sure to call tzset()
after any TZ changes.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Wed, 27 May 2020 17:58:38 +0000 (19:58 +0200)]
tests: move all tests to test/
Instead of having some testing code interleaved with bup's
real code in lib/bup/test/ and lib/bup/t/ move everything
related to testing into test/, which is also renamed from
just t/.
This aligns with pytest for future changes, and, almost more
importantly, cleans up the lib/ directory to only have the
operational code.
The latter is necessary to properly 'libify' bup in order to
be able to use tox properly to test with different versions
of python in venv later.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: also move utilities like compare-trees to dev/;
make test/lib a test-specific module tree and have wvtest.py include
it; use test/ext/ and test/int/ (external and internal) as the test
dir names.]
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 25 Nov 2020 20:37:01 +0000 (14:37 -0600)]
cirrus: allow failures on macos with python 2
It looks like macos has changed something that makes it very difficult
to detect and use the *utime* related functions. Apparently Emacs
just decided to live without it, and Python had to go to substantial
lengths to sort it out. So until/unless we have a reasonable fix,
just mark that combination as expected to fail.
Python 3 should work fine since we rely on it to sort out the problem
for us.
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Wed, 25 Nov 2020 20:33:57 +0000 (14:33 -0600)]
cirrus: make macos image match reality (catalina)
Apparently it's actually choosing catalina anyway (see the labels
during the runs), and that appears to be all that's supported now at
https://cirrus-ci.org/guide/macOS/
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Tue, 24 Nov 2020 00:21:00 +0000 (18:21 -0600)]
Depend on python 3 for utime when using 3
Since python 3.3+ now claims to implement what we need (follow
symlinks, ns resolution), rely on it for utime when possible in
preference to our own C code in _helpers.
This should fix new failures on macos due to changes to the
availability of utimensat, etc.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
test.sh: set LC_ALL=C for git fsck to avoid localized output
Observed on Ubuntu 20.04 with German as the primary language. The test
clearly parses output in English and since adapting the parsing to the
language of the test environment is not what the test tests,
specifying the language is the way to go.
The original problem can be reproduced via
LC_ALL=de_DE.utf8 ./wvtest run t/test.sh
if your system has de_DE.utf8 avalilable.
Signed-off-by: Karl-Philipp Richter <krichter@posteo.de> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message; set LC_ALL=C just for
fsck rather than setting LANGUAGE globally] Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 19 Nov 2020 20:56:31 +0000 (21:56 +0100)]
test-meta: use grep -a on xstat/meta
xattr data may contain arbitrary bytes, including NUL.
If we use grep on such data without -a, then it just
prints "Binary file (standard input) matches" and we
haven't tested anything.
Use grep -a for the appropriate invocations to avoid
this issue, we only care about ascii lines here.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Fri, 20 Nov 2020 07:29:02 +0000 (01:29 -0600)]
write_idx: fix handling of signed/unsigned values
Coerce the incoming Py_ssize_t count directly to the required uint32_t
while checking for overflow. The underlying problem was identified
when we hit two sign-compare warnings on (32-bit) platforms where
ssize_t is 32 bits.
Explicitly add -Wsign-comapare to CFLAGS, just to be sure it's
included.
Reported-by: Dominique Leuenberger Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Rob Browning [Sat, 12 Sep 2020 22:15:46 +0000 (17:15 -0500)]
gc: always clean up the packwriter; abort on errors
Make sure to always close or abort the packwriter. Abort on errors
under the assumption that any packfile in progress should only contain
blobs duplicated from the packfiles that were being garbage collected
to produce it, and those packfiles should never be removed until we've
finished writing and closing the one they're being swept into.
Signed-off-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
floor() returns a float, even though it just rounded down,
so we need to convert to an int() to avoid a deprecation
warning here.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 27 Aug 2020 21:31:57 +0000 (23:31 +0200)]
bup: pass TTY width through to sub-command and remote
As we rewrite the sub-command's stderr, it cannot determine
the width. As a result, e.g. "bup ls" will always columnate
only to 70 columns, instead of using the entire width. Fix
this by passing the real width through in an environment
variable, just like we already pass through the fact that
it's a TTY.
Also pass it through on the ssh command line so that if you
run 'bup on ... ls' it actually behaves right.
Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
Johannes Berg [Thu, 27 Aug 2020 20:25:16 +0000 (22:25 +0200)]
io: fix path_msg on python2
We get the binary representation here on both python 2 and 3,
but on 2 we really don't need to do any decoding since we must
print it as 'str'. Decoding also fails if the input is valid
UTF-8, which seems strange, but even if it worked we actually
don't want unicode objects here to print, we'd have to encode
them again to print.
Reported-by: Henninger Henningstone <henningstone@gmx.de> Signed-off-by: Johannes Berg <johannes@sipsolutions.net> Reviewed-by: Rob Browning <rlb@defaultvalue.org> Tested-by: Rob Browning <rlb@defaultvalue.org>
projects / bup.git / log