Drop support for python 2

Don't make broad, disruptive changes for now, but disallow python 2 in
./configure, adjust the docs, and remove code where it's unlikely to
cause conflicts with other pending changes.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable super-init-not-called

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

tag: use git.update_ref()

Even tag refs can be packed by git, so writing to a file
might cause issues. Call git update-ref for tags as well.

In order to do that properly (and allow 'bup tag' to keep
its ability to modify tags), change git.update_ref() to
take a 'force' argument that allows updating a ref even
if it already exists with an old value we don't know.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Fix all __del__ complaints; fail tests on AssertionErrors

Fail the tests if we detect any AssertionErrors

This exposed various problems in the recent contextmanagement work,
e.g. the assertions in the __del__ methods had been failing.  Of
course this means that from now on, asserts should only be used to
detect fatal problems.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message]
Tested-by: Rob Browning <rlb@defaultvalue.org>

Detect failures to explicitly close mmaps in py3 too

...since they're expensive, and the wrapper has detected issues a
number of times.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Client: make close durable; clean up partitial initializations

Always call close after any __init__ errors, and make sure close
always tries to clean up everything that's been initialized so far.
This fixes a __del__ complaint exposed when using the mmap wrapper in
py3.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

fmincore: fix mmap leak

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Add compat.mmap.close to set _bup_closed and always inititalize it

Add a compat.mmap.close method that properly maintains the _bup_closed
value, and make sure it always exists.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Client.__init__: clean up at the end, even if check_ok() fails

By the time we reach the check_ok() at the end, there are resources
that should be released if that call fails.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message]
Tested-by: Rob Browning <rlb@defaultvalue.org>

PackWriter._end: always try to release objcache and parentfd

Move the self.file guard inside the contextmanager.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: split from larger commit; adjust commit message]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Context manage connections (via BaseConn)

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: split from larger commit; adjust commit message]
[rlb@defaultvalue.org: add pending_raise to exit handler]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

PackWriter_Remote.close: call parent close

We weren't explicitly closing the PackWriter, and __del__ no longer
does either.  Make PackWriter a "new style" object so that the super()
call will also work with python 2.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: split from larger commit; adjust commit message]
[rlb@defaultvalue.org: convert PackWriter to "new style" object]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

NoOpContextManager: fix init/close/__del__ logic

Initialize self.closed, etc.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: split from larger commit; adjust commit message]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

MetaStoreReader: protect __del__ against init failures

The previous code could crash in __del__ if init's open() threw.

Thanks to Johannes Berg for reporting the issue.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

tree_item_with_meta, test_long_index: add missing context managers

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
[rlb@defaultvalue.org: split from larger commit; adjust commit message]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

index.Reader.filter: throw when parent's missing (don't assert)

...since we're planning to make the tests fail on assertion failures.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: adjust commit message]
Tested-by: Rob Browning <rlb@defaultvalue.org>

test_index: don't create an extra writer

This was erroneously left over when changing to
context management, and my (new?) version of pytest
started warning on it.

Fixes: f26a6a9ec9d7 ("index.Writer.__del__: replace with context management")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

git: PackIdxList: close extra packs when refreshing

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: defer list/sort until after closes]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

test-fuse: improve/broaden applicability checks

The 'fuse' group doesn't exist on (most?) modern distros
anymore, so checking for it and membership results in not
being able to run the tests.

Additionally, on some systems (e.g. Fedora) the wrong fuse
module is installed (there are at least two competing ones
in python with the name 'fuse'), so run 'bup fuse -h' at
the beginning of the test just to do the import checks.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: rebase onto WVSKIP addition]
[rlb@defaultvalue.org: adjust commit summary]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Teach pytest to handle WVSKIP and use it where we can

Have pytest scan the wvtest output for skip messages, and for now,
treat the whole test as skipped if we find any.  This means that
for now WVSKIP can only be used as an up-front guard, but that's all
we need in most cases.

This should help make it more likely that we'll notice when tests are
unexpectedly skipped.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

wvtest.sh: add a simplistic WVSKIP

Upstream wvtest actually defines the syntax, so add a trivial WVSKIP
that just prints "! ... skip ok".

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable syntax-error

This could be useful to catch issues early, and only the
py2raise module has errors now, so ignore it.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

random: use helpers.istty1

The helpers.istty1 variable can be used in place of the
same conditions open-coded here.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

main: treat BUP_FORCE_TTY as bitmap

We treat BUP_FORCE_TTY from the environment as a bitmap
all the time, so do the same in main. This requires an
update to one of the tests using it, but otherwise it's
supposed to be internal API, i.e. not documented in the
man page or such.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

main: fix output string truncation

There's a bug here that causes output to be cut off if it's
if the line was too long, in that case 'width' chars would
be removed at the beginning, rather than actually showing
'width' chars. Fix that.

Reported-by: Nix <nix@esperi.org.uk>
Fixes: 00fb1f1b2a53 ("Clean subprocess output without newliner")
Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>

main: always put BUP_FORCE_TTY into the environment

Nix reports that progress output from midx/bloom while save
is running is no longer printed.

The reason turns out to be somewhat complicated:
 * midx/bloom progress output is printed directly from the C
   code, if stderr is known to be a tty from isatty(2) or the
   environment variable BUP_FORCE_TTY
 * save is a built-in command now, and so are midx/bloom (but
   the latter is less relevant)
 * a process inherits the environment that the parent has,
   unless otherwise specified

So with the _old_ setup (with save not a module but process),
when save is started, main would set both BUP_FORCE_TTY (and
BUP_TTY_WIDTH) in the environment for the subprocess, causing
save to obtain it, and further midx/bloom to inherit it from
save. One of the downsides of this setup is that the size of
the window is now fixed, so resizing doesn't update.

With the _new_ setup, where save is a module, we don't set
BUP_FORCE_TTY or BUP_TTY_WIDTH in the environment since the
redirection and fixing etc. all happens in the main, and the
code is directly accessing stdout/stderr via the filtering.

The problem now is the following:
 1) We create the filter thread, so that stdout/stderr are
    no longer pointing to the real tty fd, so that isatty()
    returns false. This is fine for save, or when we start
    bloom/midx directly, as the _helpers.c istty2 has been
    evaluated already before we redirect the fds.
 2) As described, we don't set the environment variables
    when we run the save code, since it's a module.

However, as a result, when save starts midx/bloom as a new
subprocess, they inherit the redirected fds, so that they're
not writing to the tty themselves, but also don't get the
environment variable BUP_FORCE_TTY since they're started by
save and that was a module.

The solution then is fairly simple: set both BUP_FORCE_TTY
and BUP_TTY_WIDTH in the environment unconditionally. The
latter is necessary so that options._tty_width() works in
the module-based commands as well.

This didn't just affect save -> midx/bloom, but also the ssh
calls that pass BUP_FORCE_TTY/BUP_TTY_WIDTH to the remote
process, so potentially the server's debug output was also
affected. Nix reported that the client debug output was not
shown ("Receiving index from server: %d/%d\r"), but I don't
have an explanation for that related to this commit, as that
code is actually running in the process of the save, thus
should be shown depending on helpers.istty2, which was
always correct in the main process.

Also, I'll note that we now always put BUP_FORCE_TTY into
the environment, which we avoided previously for some calls.
However, prior to the change to have most things be modules,
we were also doing that, since everything we called (git and
other tools like par2) would be started by a subprocess, so
this doesn't really seem to pose any danger.

Reported-by: Nix <nix@esperi.org.uk>
Fixes: 5dd0172dddb9 ("bup: filter stdout/stderr via thread/pipe for internal subcommands")
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: remove now unused merge_dict from imports]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

test-fuse: show errors by running fuse in the foreground

When not run in the foreground, errors (e.g. exceptions) don't show up
on stderr.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

GNUmakefile: clean py2 .pyc files

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

git: close PackIdxList on init failures

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: use pending_raise]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

GNUmakefile: clean mounts earlier to avoid later crashes

Otherwise lingering mounts with unexpected/broken content (e.g. fuse)
can cause find, etc. to crash.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

configure/cleanup: support python 3.10

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>

dev/validate-python: require >= 3.7 for python 3 matching README

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bloom.do_bloom: always close filter; fix None vs 0 tests

Running a "split -n" for a larger file caused the assertion in
ShaBloom.__del__ to fail, i.e. the filter hadn't been explicitly
closed.  Fix the logic in do_bloom to be careful about 0 vs None since
ShaBloom has a __len__ method, and add one missing explicit close().

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Check that all context managed objects are properly closed

Ensure all of our context managed objects have a __del__ that asserts
that the instance has been properly closed so that we'll be more
likely to notice related oversights.

This will only work in cases where __del__ is called before shutdown,
but that should normally be the case for cpython.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Fully (and explicitly) close PackIdxLists

And stop checking _mpi_count in __del__ since there are no guarantees
about if/when it will run (and so could run after another has been
opened).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

PackMidx.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

compat: add ExitStack

For now, just add enough py2 support (since it wasn't too hard) to
handle some forthcoming cases.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

CatPipe.__del__: replace with context management

Drop AbortableIter since CatFile is the only user, in favor of just
closing the catpipe (what the AbortableIter was doing) in the get()
exception handler.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

index.Writer.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Replace LocalRepo/RemoteRepo __del__ with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

save: detect missing metastore via ENOENT not EACCES

While we might also eventually want to handle EACCES, ENOENT should be
what we want for the current handler, i.e. "does it exist at all".

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

index.Reader.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

MetaStoreWriter.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

MetaStoreReader.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

HLinkDB.__del__: replace with context management

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

drecurse: replace OsFile (and __del__) with finalized fd

Add composable helpers.finalize(x, finalizer) context management
adapter and use it to replace OsFile.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

ShaBloom.__del__: replace with context management

These changes also just use finally in some cases, instead of the more
complex py2 compatible BaseException/with_pending_raise() catch becase
I'm leaning in favor of just dropping python 2 support soon.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

compat.mmap: add py2 context management and use everywhere

It looks like py2 doesn't support mmap __enter__ and __exit__, so add
them, and switch all our mmap instances to be compat.mmap so that
we'll be able to context manage them.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

git/packwriter: open(..) prohibited in __del__

When an exception occurs, __del__ is invoked by the interpreter, to
perform cleanup. It seems that since Python 3.4, the behaviour has
changed, and also prohibits invocations of open(..) (source:
https://stackoverflow.com/a/29737870). Instead, contextmanager API
should be used (source: https://stackoverflow.com/a/26544629), which
seems to be in place already.

This should fix exception messages such as 'NameError: name 'open'
is not defined'.

Signed-off-by: Bas Stottelaar <basstottelaar@gmail.com>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

git.packwriter: explicitly manage lifetimes

Explicitly close all packwriters, instead of relying on __del__.  Use
"with ..." context management in most cases..  This is in preparation
for removing __del__.

Rework _end to handle *all* cleanup and to ensure that we always
release the relevant resources, whether closing normally, or aborting.
This means that we only have two places to consider with respect to
resource management: __init__ and _end.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Add any pending exception to context in all __exit__ methods

Use pending_raise() in all __exit__ methods to make sure that any
pending exception is included in the context if the body throws.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Remove Client __del__ in favor of context management

And drop EPIPE suppression for now.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

compat.pending_raise: allow/ignore None ex; make rethrow optional

This supports compact __exit__ handlers, e.g.

  def __exit__(self, type, value, ...):
      with pending_raise(value, rethrow=False):
          clean_up()

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

save: separate opt handling and work; isolate resource lifetimes

Rearrange save to disentangle the options parsing, saving, and
resource (e.g. packfile) lifetime management before we replace __del__
with context management.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

split: separate opt handling and work; isolate packfile lifetime

Rearrange split to disentangle the options parsing, splitting, and
resource (packfile) lifetime management before we replace __del__ with
context management.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

split: handle --noop/copy via do nothing packwriter

This allows us to remove the special-casing for those code paths, and
rearrange the structure in preparation for handling the packwriter via
a context manager, in preparation for removing __del__.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

configure: support optional cc arguments

I noticed in some unrelated reports that older compilers were
rejecting -Wno-unsed-command-line-argument, so add
bup-add-cflag-if-supported to configure and use it for that argument.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

bitmatch: avoid signed/unsigned comparison

Thanks to Brian Minton for reporting the problem:

  I'm using gcc version 8.3.0 (Debian 8.3.0-6). The system is x86_64
  with 8 GB ram. When I try to build the source, with make, I get the
  following error:

    ...

    In file included from lib/bup/_helpers.c:69:
    lib/bup/_helpers.c: In function 'bitmatch':
    .../src/bup/src/bup/intprops.h:626:22: error: comparison of
    integer expressions of different signedness: 'long long unsigned int' and
    'Py_ssize_t' {aka 'long int'} [-Werror=sign-compare]
    : (tmax) / (b) < (a)))
    ^
    .../src/bup/src/bup/intprops.h:427:10: note: in expansion of
    macro '_GL_INT_MULTIPLY_RANGE_OVERFLOW'
    && _GL_INT_MULTIPLY_RANGE_OVERFLOW (a, b, 0, (__typeof__ (*(r))) -1)) \
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    .../src/bup/src/bup/intprops.h:655:36: note: in expansion of
    macro 'INT_MULTIPLY_WRAPV'
    #define INT_MULTIPLY_OK(a, b, r) ! INT_MULTIPLY_WRAPV (a, b, r)
    ^~~~~~~~~~~~~~~~~~
    lib/bup/_helpers.c:624:10: note: in expansion of macro 'INT_MULTIPLY_OK'
    if (!INT_MULTIPLY_OK(byte, 8, &result)
    ^~~~~~~~~~~~~~~

And to Mark J Hewitt for reporting the issue on a 32-bit Raspberry Pi
system.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

bup_gethostname: always null terminate

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

web list-directory: replace != with is not None

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

web: do not show size for dirs and symlinks

It's strange to see a lot of zeros for these items, just
don't show them by making the display_size None.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

bitmatch: check for overflow via intprops

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Support signed commits (i.e. the gpgsig header)

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

CatPipe.get: report unexpected cat-file exit status

Thanks to Mark Hewitt for reporting a problem that suggested we should
handle this case more carefully.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

cstr_from_bytes: fix null termination error

Thanks to Johannes Berg for reporting the problem.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Drop builtin_mul_overflow in favor of INT_MULTIPLY_OK

Now that we have intprops.h drop built_mul_overflow (since intpropos.h
should use it when available).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

make: add check-py2 and check-py3 targets

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

ftp: use sys.exc_info(); import traceback unconditionally

Since it appears that works across all the versions we support.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Allow PANDOC path to contain spaces

Thanks to Arthur Ward for reporting the problem and proposing an
alternate solution.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

test-index-check-device: unmount "lazily" to avoid potential crash

Apparently the umount was failing on some systems, and since we're
already using loop, mke2fs, etc. which aren't really portable, and we
use "umount -l" in cleanup-mounts-under, just use it here too for now.

Thanks to Abdel Said for reporting the problem:

  ! t/test-index-check-device.sh:38 mke2fs -F -j -m 0 testfs.img ok
  ! t/test-index-check-device.sh:39 mount -o loop testfs.img /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU ok
  ! t/test-index-check-device.sh:41 chown root:root /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU ok
  ! t/test-index-check-device.sh:42 chmod 0700 /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU ok
  ! t/test-index-check-device.sh:45 date ok
  umount: /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU: device is busy.
  (In some cases useful info about processes that use
  the device is found by lsof(8) or fuser(1))
  ! t/test-index-check-device.sh:46 umount /opt/bup/t/mnt/test-index-check-device.sh-YuNQZtU FAILED

Additional discussion: https://groups.google.com/d/msg/bup-list/GhVonW0XGVE/8kIkJEOwCQAJ

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable inconsistent-return-statements

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: return early for dry run in import_duplicity exo]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable catching-non-exception

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: add undefined-variable

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: add missing %s in validate_vfs_path changes]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable unidiomatic-typecheck

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
[rlb@defaultvalue.org: change isinstance str checks to compat.str_type]
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable consider-using-in

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable useless-return

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: enable return-in-init

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

pylint: check for trailing whitespace

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

metadata: accept EOPNOTSUPP for lchmod()

On some systems (e.g. mine, Fedora 33) python2 has
(started to have?) lchmod(), as glibc has it, but
that always only returns EOPNOTSUPP if you really
try to operate on a symlink. Allow for that and
handle it just like ENOSYS.

Note that the handling of ENOSYS is wrong, since
integers can't be thrown or caught - pylint found
this as well, but since I'm fixing EOPNOTSUPP here
I need to fix that at the same time.

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>

README: fix and simplify cirrus badges

Fix broken links (given the recent task name changes), and stop
reporting per-platform status since each one may have multiple tasks.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

system-info: report freebsd cpu and mem info

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

cirrus: run pylint for python 2 and python 3

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

configure: add --with-pylint=[yes|no|maybe] defaulting to maybe

When set to no, don't run pylint from ./pylint, just exit
successfully.  When set to maybe, use dev/have-pylint to figure out if
pylint is available, and if so, run it, otherwise exit successfully
after describing the situation.  When set to yes, always try to run
pylint.

This may be useful more generally, but in particular, it makes it
possible to run this:

  ./configure --with-pylint=maybe
  make check-both

in situations where pylint is available for say python 3, but not
python 2.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

GNUmakefile: run "bup features" during test

Provide additional information about what's being tested, particularly
in CI.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

test: add pylint and test imports

Add pylint and test for unnecessary imports as a first step, which
requires cleaning them all up.

[rlb@defaultvalue.org: add and use ./pylint]

Signed-off-by: Johannes Berg <johannes@sipsolutions.net>
Reviewed-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

cirrus: adjust for new python-config based builds

Adjust to accommodate the switch from PYTHON to BUP_PYTHON_CONFIG
cf. 4d9dd65141326b0f3ffa1658a0535348940ed017

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

prep-for-freebsd-build: switch to py38

That's what's now in 12.2 (which we use in ci).

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

cirrus: adjust test names (root vs non-root, etc.)

The su tests run as non-root (bup), not root.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

dev/system-info: include df, mount, and git version

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

bup: remove redundant allocation doubling

Thanks to Johannes Berg for pointing out the issue.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

test_git: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

test-save-restore: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

test-misc: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

test_prune_older: be independent of user's git defaultBranch

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

cmd/bup: add some missing safety checks

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Use intprops for INTEGRAL_ASSIGNMENT_FITS INTEGER_TO_PY uadd

Rewrite INTEGRAL_ASSIGNMENT_FITS, INTEGER_TO_PY, and uadd using
intprops,  which also avoids needing all the custom (and historically
fragile) compiler option manipulations.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>

Make intprops available as "bup/intprops.h"

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

Add Gnulib intprops.h

Signed-off-by: Rob Browning <rlb@defaultvalue.org>

config.vars.in: add CC so a configure CC will be the default

Add CC to config.vars.in (included by GNUMakefile) so that the CC
detected by ./configure will actually become the default.  Previously
it would still reflect GNU make's default, even after something like
this:

  CC=/some/where/gcc ./configure

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
Tested-by: Rob Browning <rlb@defaultvalue.org>