btrbk: fix duplicate cmdline option -l in lsbtr

After 7 years and a lot of warnings we can drop this code. Note that
most of those refer to `upgrade_to_v0.23.0.md`, which was removed in
16364e2.

.github/FUNDING.yml

@@ -0,0 +1,3 @@
+# These are supported funding model platforms
+
+custom: ['https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S', 'https://btc.com/19DYtoEepxBmn9ZPspJGZrhCtySKCxPcP1']

.github/workflows/codespell.yml

@@ -0,0 +1,26 @@
+---
+name: Codespell
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+permissions:
+  contents: read
+
+jobs:
+  codespell:
+    name: Check for spelling errors
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Codespell
+        uses: codespell-project/actions-codespell@v2
+        with:
+          check_filenames: true
+          skip: ".git,*.pdf,*.svg"
+          ignore_words_list: uptodate

ChangeLog

@@ -1,3 +1,135 @@
+btrbk-0.32.6
+
+  * Fix backup of unrelated (by parent_uuid) snapshots (close #339).
+  * Remove echo -e for portability (close #506).
+  * Support btrfs send protocol v2 (send_protocol and
+    send_compressed_data config options).
+  * Add bzip3 support.
+  * Convert ssh_filter_btrbk.sh to POSIX sh, and harden it.
+  * Slight change in ssh_filter_btrbk.sh logging output.
+  * Minor bugfixes, stability and documentation improvements.
+
+btrbk-0.32.5
+
+  * Correct handling of zero-size raw info file (close #491).
+
+btrbk-0.32.4
+
+  * Fix regression: wrong deprecation warnings in some cases.
+
+btrbk-0.32.3
+
+  * Fix deletion of many subvolumes at once (close #476).
+  * Allow disabling ssh_identity and ssh_user options.
+  * Minor bugfixes and documentation improvements.
+
+btrbk-0.32.2
+
+  * Fix regression: archive on missing target directories.
+  * Fix action "config print".
+  * Print version and help message to stdout (close #452).
+  * Check results of filesystem usage (close #469).
+  * Add "compat ignore_receive_errors" option (close #383).
+  * Support multiple gpg recipients (close #471).
+  * Fix changelog: remove "compat missing_otime" item in v0.32.1.
+  * Fix changelog: correct "warn_unknown_targets" item in v0.31.3.
+  * Minor bugfixes and framework improvements.
+
+btrbk-0.32.1
+
+  * Fix regression: correctly read mountinfo (close #445).
+  * Fix regression: filter paths for "lsbtr" (action "ls").
+  * Add backend btrfs-progs-doas (close #444).
+  * Allow directory traversal for local command line arguments.
+  * Display all source subvolumes in "usage" action.
+  * Handle errors from btrfs filesystem usage.
+
+btrbk-0.32.0
+
+  * MIGRATION
+    - If timestamp_format is not configured, explicitly set
+      "timestamp_format short" to revert old behavior.
+    - Update ssh_filter_btrbk.sh on remote hosts.
+  * Change default for timestamp_format to "long".
+  * Optional "volume" config section.
+  * Use "volume" section only for grouping and relative paths.
+  * Allow absolute path for "subvolume" and "snapshot_dir"
+    (close #407, #121).
+  * Support subvolume names with UTF-8 characters (close #392, #213,
+    #120). Add "safe_commands" option for paranoid people.
+  * Add "incremental_prefs" configuration option (close #387).
+  * Change incremental prefs policy, giving snapshots created by btrbk
+    higher preference than the global ones resolved by parent-uuid.
+  * Change "incremental_clones" option to boolean.
+  * Drop support of deprecated raw file format (btrbk < 0.26.0).
+  * Drop support of deprecated "-r, --resume-only" cmdline option.
+  * Fix ambiguous naming in --format=raw output of "list" actions.
+  * Accept quoted values in config.
+  * Use single quotes for raw table output.
+
+btrbk-0.31.3
+
+  * Add "warn_unknown_targets" configuration option (close #393).
+  * Add -1,--single-column command-line option for listing actions.
+  * Allow relative path command line argument for all actions.
+  * Add support for zstd adaptive compression (close #397).
+  * Add contrib/tools/btrbk_restore_raw.py (close #401).
+  * Minor bugfixes and framework improvements.
+
+btrbk-0.31.2
+
+  * MIGRATION
+    - Update ssh_filter_btrbk.sh on remote hosts.
+  * ssh_filter_btrbk.sh: Fix security vulnerability.
+    Specially crafted commands may be executed without being properly
+    checked. Applies to remote hosts filtering ssh commands using
+    ssh_filter_btrbk.sh in authorized_keys.
+  * Warn if no subvolume defined in config (close #378).
+
+btrbk-0.31.1
+
+  * Bugfix: duplicate snapshot location check (close #360).
+  * Bugfix: action "list all" (close #367).
+  * btrbk-mail: optionally prefix command output lines.
+
+btrbk-0.31.0
+
+  * Add action "extents [diff]": print accurate disk space usage based
+    on block regions (extent data, FIEMAP ioctl).
+  * Improve action "list" and "stats" table output (close #334).
+  * Add subcommand "list all", make default for "list" action.
+  * Change semantics on output of action "list": show only btrbk
+    snapshots/backups (close #333).
+  * Print human readable units in action "diff" (close #345).
+  * Allow custom table format definitions from command line.
+  * Add Bash completion (close #343).
+  * Add support for zstd compression (close #358).
+  * Bugfix: fs_path fallback for action "origin" and logging.
+  * Bugfix: makefile bin-links race (close #341).
+
+btrbk-0.30.0
+
+  * MIGRATION
+    - In your scripts (e.g. cron jobs, systemd units), replace
+      "/usr/sbin/btrbk" with "/usr/bin/btrbk".
+  * Install to "/usr/bin" instead of "/usr/sbin".
+  * Support IPv6 addresses (close #332).
+  * Add "compat busybox" configuration option (close #327).
+  * Create "lsbtr" symlink (alias for "btrbk ls").
+  * Improve action "ls" (allow url, bugfixes).
+  * Add "backend_local_user" configuration option.
+  * btrbk-mail: add more elaborated email and rsync options.
+  * Minor bugfixes and framework improvements.
+
+btrbk-0.29.1
+
+  * Allow multiple path arguments for action "ls".
+  * Change default output to format=short for action "ls".
+  * ssh_filter_btrbk.sh: exclude "btrfs subvolume show|list" from
+    restrict-path (close #309).
+  * Bugfix: fix filter statement match on wildcards (close #311).
+  * Fix regression: print ssh errors only if needed.
+
 btrbk-0.29.0
 
   * MIGRATION
@@ -116,7 +248,7 @@ btrbk-0.27.0
     - Allow snapshot_dir to be a mountpoint.
     - Search complete target tree for correlated subvolumes.
     - Include snapshots from all mountpoints as candidates (disabled
-      due to uptream bug: github.com/kdave/btrfs-progs/issues/96).
+      due to upstream bug: github.com/kdave/btrfs-progs/issues/96).
     - Read /proc/self/mountinfo instead of /proc/self/mounts.
     - Always read /proc/self/mountinfo.
     - Resolve realpath using readlink(1).
@@ -187,7 +319,7 @@ btrbk-0.25.0
   * Allow trailing comments in btrbk.conf (close #129).
   * Bugfix: rate limiting must be done after compression (close #134).
   * raw_target_encrypt: Always set "gpg --no-random-seed-file":
-    prevents creation of "~/.gnupg/random_seed" with slight perfomance
+    prevents creation of "~/.gnupg/random_seed" with slight performance
     penalty.
 
 btrbk-0.24.0

Makefile

@@ -9,6 +9,7 @@
 #
 
 BIN         = btrbk
+BIN_LINKS   = lsbtr
 CONFIGS     = btrbk.conf.example
 DOCS        = ChangeLog \
               README.md
@@ -16,16 +17,18 @@ SCRIPTS    = ssh_filter_btrbk.sh \
               contrib/cron/btrbk-mail \
               contrib/cron/btrbk-verify \
               contrib/migration/raw_suffix2sidecar \
-             contrib/crypt/kdf_pbkdf2.py
+              contrib/crypt/kdf_pbkdf2.py \
+              contrib/tools/btrbk_restore_raw.py
 
 PN          = btrbk
 PREFIX     ?= /usr
 CONFDIR     = /etc
 CRONDIR     = /etc/cron.daily
-BINDIR     = $(PREFIX)/sbin
+BINDIR      = $(PREFIX)/bin
 DOCDIR      = $(PREFIX)/share/doc/$(PN)
 SCRIPTDIR   = $(PREFIX)/share/$(PN)/scripts
 SYSTEMDDIR  = $(PREFIX)/lib/systemd/system
+BASHCOMPDIR = $(PREFIX)/share/bash-completion/completions
 MAN1DIR     = $(PREFIX)/share/man/man1
 MAN5DIR     = $(PREFIX)/share/man/man5
 
@@ -41,23 +44,38 @@ replace_vars = sed \
 	-e "s|@DOCDIR@|$(DOCDIR)|g" \
 	-e "s|@SCRIPTDIR@|$(SCRIPTDIR)|g" \
 	-e "s|@SYSTEMDDIR@|$(SYSTEMDDIR)|g" \
+	-e "s|@BASHCOMPDIR@|$(BASHCOMPDIR)|g" \
 	-e "s|@MAN1DIR@|$(MAN1DIR)|g" \
 	-e "s|@MAN5DIR@|$(MAN5DIR)|g"
 
 all: man
 
-install: install-bin install-etc install-systemd install-share install-man install-doc
+install: install-bin install-bin-links install-etc install-completion install-systemd install-share install-man install-doc
 
 install-bin:
 	@echo 'installing binary...'
 	install -d -m 755 "$(DESTDIR)$(BINDIR)"
 	install -p -m 755 $(BIN) "$(DESTDIR)$(BINDIR)"
 
+install-bin-links: install-bin
+	@echo 'installing symlinks...'
+	for name in $(BIN_LINKS); do \
+		ln -s -n -f $(BIN) "$(DESTDIR)$(BINDIR)/$$name"; \
+	done
+
 install-etc:
 	@echo 'installing example configs...'
 	install -d -m 755 "$(DESTDIR)$(CONFDIR)/btrbk"
 	install -p -m 644 $(CONFIGS) "$(DESTDIR)$(CONFDIR)/btrbk"
 
+install-completion:
+	@echo 'installing bash completion...'
+	install -d -m 755 "$(DESTDIR)$(BASHCOMPDIR)"
+	install -p -m 644 contrib/bash/completion.bash "$(DESTDIR)$(BASHCOMPDIR)/$(BIN)"
+	for name in $(BIN_LINKS); do \
+		ln -s -n -f $(BIN) "$(DESTDIR)$(BASHCOMPDIR)/$$name"; \
+	done
+
 install-systemd:
 	@echo 'installing systemd service units...'
 	install -d -m 755 "$(DESTDIR)$(SYSTEMDDIR)"

README.md

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs
+Btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs
 specific capabilities to create atomic snapshots and transfer them
 incrementally to your backup locations.
 
@@ -9,24 +9,27 @@ The source and target locations are specified in a config file, which
 allows to easily configure simple scenarios like "laptop with locally
 attached backup disks", as well as more complex ones, e.g. "server
 receiving backups from several hosts via ssh, with different retention
-policy".
+policies".
 
 Key Features:
 
   * Atomic snapshots
   * Incremental backups
-  * Configurable retention policy
+  * Flexible retention policy
   * Backups to multiple destinations
   * Transfer via ssh
-  * Resume of backups (if backup target was not reachable for a while)
+  * Robust recovery from interrupted backups (for removable and mobile
-  * Encrypted backups to non-btrfs destinations
+    devices)
+  * Archive to offline storage
+  * Encrypted backups to non-btrfs storage
   * Wildcard subvolumes (useful for docker and lxc containers)
   * Transaction log
   * Comprehensive list and statistics output
   * Resolve and trace btrfs parent-child and received-from relationships
-  * Display file changes between two backups
+  * List file changes between backups
+  * Calculate accurate disk space usage based on block regions
 
-btrbk is designed to run as a cron job for triggering periodic
+Btrbk is designed to run as a cron job for triggering periodic
 snapshots and backups, as well as from the command line (e.g. for
 instantly creating additional snapshots).
 
@@ -35,8 +38,12 @@ Installation
 ============
 
 Btrbk is a single perl script, and does not require any special
-installation procedures or libraries. Simply download the latest
+installation procedures or libraries. Download the latest [btrbk
-[btrbk source tarball] and run the "btrbk" executable.
+source tarball], or try latest master:
+
+    wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
+    chmod +x btrbk
+    sudo ./btrbk ls /
 
 For more information, read the [installation documentation].
 
@@ -49,12 +56,12 @@ For more information, read the [installation documentation].
   * [btrfs-progs]: Btrfs filesystem utilities >= v4.12
   * [Perl interpreter]: Probably already installed on your system
   * [OpenSSH]: If you want to transfer backups from/to remote locations
-  * [Pipe Viewer]: If you want rate limiting and progress bars
+  * [mbuffer]: If you want rate limiting and progress bars
 
   [btrfs-progs]: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
   [Perl interpreter]: https://www.perl.org
-  [OpenSSH]: https://www.openssh.org
+  [OpenSSH]: https://www.openssh.com
-  [Pipe Viewer]: https://www.ivarch.com/programs/pv.shtml
+  [mbuffer]: https://www.maier-komor.de/mbuffer.html
 
 
 Synopsis
@@ -66,29 +73,113 @@ a full description of the command line options.
   [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
 
 
-Configuration File
+Configuration
-==================
+=============
 
 Before running `btrbk`, you will need to create a configuration
 file. You might want to take a look at `btrbk.conf.example` provided
 with this package. For a detailed description, please consult the
 [btrbk.conf(5)] man-page.
 
-When playing around with config-files, it is highly recommended to
+After a configuration change, it is highly recommended to check it by
-check the output using the `dryrun` command before executing the
+running btrbk with the `-n,--dry-run` option:
-backups:
 
-    btrbk -c /path/to/myconfig -v dryrun
+    # btrbk -c /path/to/myconfig -v -n run
 
 This will read all btrfs information on the source/target filesystems
 and show what actions would be performed (without writing anything to
 the disks).
 
+The examples below assume that the btrfs subvolume containing `home`
+and `rootfs` is mounted at `/mnt/btr_pool`. This is usually the btrfs
+root subvolume, which always has `subvolid=5`.
+
+Mounting `subvolid=5` is *recommended* (mandatory for btrbk < v0.32.0)
+if you want to backup your root filesystem `/`.
+
+/etc/fstab:
+
+    /dev/sda1  /mnt/btr_pool  btrfs  subvolid=5,noatime  0 0
+
+Note that some default btrfs installations (e.g. Ubuntu) use subvolume
+names `@` for rootfs (mounted at `/`) and `@home` for `/home`, as a
+naming convention. If this is the case on your file system, replace
+the `subvolume` declarations in the examples accordingly.
+
   [btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html
 
 
-Example: laptop with usb-disk for backups
+Example: Local Regular Snapshots (time-machine)
------------------------------------------
+-----------------------------------------------
+
+The simplest use case is to only create snapshots of your data. This
+will obviously not protect it against hardware failure, but can be
+useful for:
+
+  * protection against inadvertent changes or deletions
+  * keeping past states of copies from rsync or similar tools
+
+Let's assume you need regular snapshots of your home directory, which
+is located in the subvolume `home` of the volume `/mnt/btr_pool`. The
+snapshots are to be stored in `btrbk_snapshots` (on the same volume).
+
+/etc/btrbk/btrbk.conf:
+
+    timestamp_format        long
+    snapshot_preserve_min   18h
+    snapshot_preserve       48h
+
+    volume /mnt/btr_pool
+      snapshot_dir btrbk_snapshots
+      subvolume home
+
+Notice that the `target` option is not provided, and btrbk will only
+manage snapshots located on the same volume in `snapshot_dir`. Btrbk
+does not create subdirs by default, the snapshot directory must first
+be created manually:
+
+    # mkdir /mnt/btr_pool/btrbk_snapshots
+
+The "volume" section is merely used as a specifier for a base
+directory, and can be skipped if you prefer to configure everything
+using absolute paths. The above configuration can also be written as:
+
+    snapshot_dir /mnt/btr_pool/btrbk_snapshots
+    subvolume    /mnt/btr_pool/home
+
+If you don't want to mount the btrfs root filesystem to
+`/mnt/btr_pool`, you might as well configure it like this:
+
+    snapshot_dir /btrbk_snapshots
+    subvolume    /home
+
+Start a dry run (-n, --dry-run):
+
+    # btrbk run -n
+
+Create the first snapshot:
+
+    # btrbk run
+
+Print schedule (-S, --print-schedule):
+
+    # btrbk run -n -S
+
+If it works as expected, configure a cron job to run btrbk hourly:
+
+/etc/cron.hourly/btrbk:
+
+    #!/bin/sh
+    exec /usr/bin/btrbk -q run
+
+Snapshots will now be created every hour. All snapshots are preserved for at
+least 18 hours (`snapshot_preserve_min`), whether they are created by the cron
+job or manually by calling `sudo btrbk run` on the command line. Additionally,
+48 hourly snapshots are preserved (`snapshot_preserve`).
+
+
+Example: Backups to USB Disk
+----------------------------
 
 In this example, we assume you have a laptop with:
 
@@ -105,7 +196,7 @@ In this example, we assume you have a laptop with:
 Retention policy:
 
   * keep all snapshots for 2 days, no matter how frequently you (or
-    your cron-job) run btrbk
+    your cron job) run btrbk
   * keep daily snapshots for 14 days (very handy if you are on
     the road and the backup disk is not attached)
   * keep monthly backups forever
@@ -117,6 +208,9 @@ Retention policy:
     snapshot_preserve_min   2d
     snapshot_preserve      14d
 
+    # Create snapshots only if the backup disk is attached
+    #snapshot_create ondemand
+
     target_preserve_min    no
     target_preserve        20d 10w *m
 
@@ -132,7 +226,7 @@ Retention policy:
 /etc/cron.daily/btrbk:
 
     #!/bin/sh
-    exec /usr/sbin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run
+    exec /usr/bin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run
 
 
   * This will create snapshots on a daily basis:
@@ -142,21 +236,22 @@ Retention policy:
     * `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD`
     * `/mnt/btr_backup/mylaptop/home.YYYYMMDD`
 
-If you want the snapshots to be created only if the backup disk is
+If you prefer triggering the backups manually, change the cron command
-attached, simply add the following line to the config:
+to run the `snapshot` action instead of `run`. Start the backups
+manually by running:
 
-    snapshot_create ondemand
+    # btrbk resume
 
 For a quick additional snapshot of your home, run:
 
     # btrbk snapshot home
 
 
-Example: host-initiated backup on fileserver
+Example: Host-initiated Backup on Fileserver
 --------------------------------------------
 
-Let's say you have a fileserver at "myserver.mydomain.com" where you
+Let's say you have a fileserver at "myserver.example.org" where you
-want to create backups of your laptop disk, the config would look like
+want to create backups of your laptop disk. The config could look like
 this:
 
     ssh_identity /etc/btrbk/ssh/id_rsa
@@ -164,14 +259,14 @@ this:
     volume /mnt/btr_pool
       subvolume rootfs
         target /mnt/btr_backup/mylaptop
-        target ssh://myserver.mydomain.com/mnt/btr_backup/mylaptop
+        target ssh://myserver.example.org/mnt/btr_backup/mylaptop
 
 In addition to the backups on your local usb-disk mounted at
 `/mnt/btr_backup/mylaptop`, incremental backups would also be pushed
-to `myserver.mydomain.com`.
+to `myserver.example.org`.
 
 
-Example: fileserver-initiated backups from several hosts
+Example: Fileserver-initiated Backups from Several Hosts
 --------------------------------------------------------
 
 If you're a sysadmin and want to trigger backups directly from your
@@ -179,17 +274,17 @@ fileserver, the config would be something like:
 
     ssh_identity /etc/btrbk/ssh/id_rsa
 
-    volume ssh://alpha.mydomain.com/mnt/btr_pool
+    volume ssh://alpha.example.org/mnt/btr_pool
       target /mnt/btr_backup/alpha
       subvolume rootfs
       subvolume home
 
-    volume ssh://beta.mydomain.com/mnt/btr_pool
+    volume ssh://beta.example.org/mnt/btr_pool
       target /mnt/btr_backup/beta
       subvolume rootfs
       subvolume dbdata
 
-This will pull backups from alpha/beta.mydomain.com and locally
+This will pull backups from alpha/beta.example.org and locally
 create:
 
   * `/mnt/btr_backup/alpha/rootfs.YYYYMMDD`
@@ -198,33 +293,7 @@ create:
   * `/mnt/btr_backup/beta/dbdata.YYYYMMDD`
 
 
-Example: local time-machine (hourly snapshots)
+Example: Multiple Btrbk Instances
-----------------------------------------------
-
-If all you want is to create snapshots of your home directory on a
-regular basis:
-
-/etc/btrbk/btrbk.conf:
-
-    timestamp_format        long
-    snapshot_preserve_min   18h
-    snapshot_preserve       48h 20d 6m
-
-    volume /mnt/btr_pool
-      snapshot_dir btrbk_snapshots
-      subvolume home
-
-/etc/cron.hourly/btrbk:
-
-    #!/bin/sh
-    exec /usr/sbin/btrbk -q run
-
-Note that you can run btrbk more than once an hour, e.g. by calling
-`sudo btrbk run` from the command line. With this setup, all those
-extra snapshots will be kept for 18 hours.
-
-
-Example: multiple btrbk instances
 ---------------------------------
 
 Let's say we have a host (at 192.168.0.42) running btrbk with the
@@ -248,7 +317,7 @@ monthlies are received from 192.168.0.42. The source filesystem is
 never altered because of `snapshot_preserve_min all`.
 
 
-Example: virtual machine setup
+Example: Virtual Machine Setup
 ------------------------------
 
 Common virtual machine setups have multiple volume sections with same
@@ -286,19 +355,16 @@ Assuming same filesystem: "ssh://localhost:2201/dev/sda1", "ssh://localhost:2202
 ```
 
 
-Example: backup from non-btrfs source
+Example: Backup from non-btrfs Source
 -------------------------------------
 
-First create a btrfs subvolume on the backup server:
+If you want to make backups from a filesystem other than btrfs
+(e.g. ext4 or reiserfs), you need to create a *synchronization
+subvolume* on the backup disk:
 
     # btrfs subvolume create /mnt/btr_backup/myhost_sync
 
-In your daily cron script, prior to running btrbk, sync your source to
+Configure btrbk to use `myhost_sync` as source subvolume:
-`myhost_sync`, something like:
-
-    rsync -a --inplace --delete -e ssh myhost.mydomain.com:/data/ /mnt/btr_backup/myhost_sync/
-
-Then run btrbk, with myhost_sync configured *without any targets*:
 
     volume /mnt/btr_backup
       subvolume myhost_sync
@@ -307,14 +373,28 @@ Then run btrbk, with myhost_sync configured *without any targets*:
         snapshot_preserve_min   latest
         snapshot_preserve       14d 20w *m
 
-This will produce daily snapshots `/mnt/btr_backup/myhost.20150101`,
+The btrbk package provides the "btrbk-mail" script, which automates
-with retention as defined with the snapshot_preserve option.
+the synchronization using rsync, and can be run as cron job or systemd
+timer unit. For configuration details, see the config section in
+"/contrib/cron/btrbk-mail".
 
-Note that the provided script: "contrib/cron/btrbk-mail" has support
+Alternatively, you can run any synchronization software prior to
-for this!
+running btrbk. Something like:
+
+    #!/bin/sh
+    rsync -az --delete \
+          --inplace --numeric-ids --acls --xattrs \
+          -e 'ssh -i /etc/btrbk/ssh/id_rsa' \
+          myhost.example.org:/data/ \
+          /mnt/btr_backup/myhost_sync/
+
+    exec /usr/bin/btrbk -q run
+
+This will produce snapshots `/mnt/btr_backup/myhost.20150101`, with
+retention as defined with the snapshot_preserve option.
 
 
-Example: encrypted backup to non-btrfs target
+Example: Encrypted Backup to non-btrfs Target
 ---------------------------------------------
 
 If your backup server does not support btrfs, you can send your
@@ -332,7 +412,7 @@ compressed and piped through GnuPG.
     raw_target_compress   xz
     raw_target_encrypt    gpg
     gpg_keyring           /etc/btrbk/gpg/pubring.gpg
-    gpg_recipient         btrbk@mydomain.com
+    gpg_recipient         btrbk@example.org
 
     volume /mnt/btr_pool
       subvolume home
@@ -348,7 +428,7 @@ host. For each backup, two files are created:
   * `/backup/home.YYYYMMDD.btrfs.xz.gpg.info`: sidecar file containing
     metadata used by btrbk.
 
-I you are using raw _incremental_ backups, please make sure you
+If you are using raw _incremental_ backups, please make sure you
 understand the implications (see [btrbk.conf(5)], TARGET TYPES).
 
 
@@ -360,23 +440,111 @@ security precautions you can. In most cases backups are generated
 periodically without user interaction, so it is not possible to
 protect your ssh key with a password. The steps below will give you
 hints on how to secure your ssh server for a backup scenario. Note
-that the `btrbk` executable is not needed on the remote side, but you
+that the btrbk package is not required on the remote side, but you
 will need the `btrfs` executable from the [btrfs-progs] package.
 
 
-### Step 1: Create SSH keypair
+### Create SSH Key Pair
 
 On the client side, create a ssh key dedicated to btrbk, without
 password protection:
 
-    ssh-keygen -t rsa -b 2048 -f /etc/btrbk/ssh/id_rsa -C btrbk@mydomain.com -N ""
+    # ssh-keygen -t rsa -b 4096 -f /etc/btrbk/ssh/id_rsa -C btrbk@example.org -N ""
 
 The content of the public key (/etc/btrbk/ssh/id_rsa.pub) is used for
 authentication in "authorized_keys" on the server side (see [sshd(8)]
 for details).
 
 
-### Step 2 (option): root login restricted by "ssh_filter_btrbk.sh"
+### Allow Root Login
+
+The most straight forward setup is to allow root login on the remote
+host. If this is not an option for you, refer to the more complex
+"Dedicated Btrbk User Login" section below.
+
+/etc/ssh/sshd_config:
+
+    PermitRootLogin prohibit-password
+
+Add your btrbk public key to "/root/.ssh/authorized_keys" on the
+server, and you are good to go.
+
+
+### Restrict Access
+
+Restrict ssh access to a static IP address within your network. On the
+remote host, either add a "Match" block in:
+
+/etc/ssh/sshd_config:
+
+    Match Address 192.168.0.42
+
+Or restrict in authorized_keys:
+
+    from="192.168.0.42" <pubkey>...
+
+Consult the [sshd_config(5)] man-page for a detailed explanation and
+more options.
+
+
+Dedicated Btrbk User Login (optional)
+-------------------------------------
+
+If allowing root login is not an option for you, there are several
+ways to restrict SSH access to a regular user.
+
+
+### Option 1: Use sudo
+
+On the client side, configure btrbk use the sudo backend. This changes
+the ssh calls to btrfs commands to `sudo btrfs <subcommand>
+<options>`.
+
+/etc/btrbk/btrbk.conf:
+
+    backend_remote btrfs-progs-sudo
+
+On the remote host, grant root permissions for the "btrfs" command
+groups (subcommands) in "/etc/sudoers". If you are using
+[ssh_filter_btrbk(1)], also add the `ssh_filter_btrbk.sh --sudo`
+option in "authorized_keys" (see below).
+
+
+### Option 2: Use btrfs-progs-btrbk
+
+Instead of using the all-inclusive `btrfs` command,
+"btrfs-progs-btrbk" allows you to restrict privileges to its
+subcommands using linux capabilities(7) or setuid.
+
+Note that the "btrfs-progs-btrbk" package is not available on all
+linux distributions, you might need to build and install it on your
+own (refer to [btrfs-progs-btrbk] on GitHub for more details).
+
+/etc/btrbk/btrbk.conf:
+
+    backend_remote btrfs-progs-btrbk
+
+Make sure that only the required binaries with elevated privileges can
+be called by the btrbk user. For example, on a server acting as "btrbk
+source", allow only the following binaries for the "btrbk" group:
+
+    # getcap /usr/bin/btrfs-*
+    /usr/bin/btrfs-send cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-delete cap_dac_override,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-list cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-show cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-snapshot cap_dac_override,cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    
+    # ls -l /usr/bin/btrfs-*
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-send
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-delete
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-list
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-show
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-snapshot
+
+
+Restrict Commands with "ssh_filter_btrbk.sh" (optional)
+-------------------------------------------------------
 
 Btrbk comes with a shell script "ssh_filter_btrbk.sh", which restricts
 ssh access to sane calls to the "btrfs" command needed for snapshot
@@ -387,38 +555,14 @@ to run it whenever the key is used for authentication. Example
 "/root/.ssh/authorized_keys":
 
     # example backup source (also allowing deletion of old snapshots)
-    command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete" <pubkey>...
+    command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete",restrict <pubkey>...
 
     # example backup target (also allowing deletion of old snapshots)
-    command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete" <pubkey>...
+    command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete",restrict <pubkey>...
 
     # example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no),
     # restricted to subvolumes within /home or /data
-    command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data" <pubkey>...
+    command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data",restrict <pubkey>...
-
-
-### Step 2 (option): dedicated user login, using different backend
-
-Create a user dedicated to btrbk and add the public key to
-"/home/btrbk/.ssh/authorized_keys". In "btrbk.conf", choose either:
-
- * `backend btrfs-progs-btrbk` to use separated binaries with elevated
-   privileges (suid or fscaps) instead of the "btrfs" command (see
-   [btrfs-progs-btrbk]).
-
- * `backend btrfs-progs-sudo`, configure "/etc/sudoers" and add the
-   `ssh_filter_btrbk.sh --sudo` option.
-
-
-### Further considerations
-
-You might also want to restrict ssh access to a static IP address
-within your network:
-
-    from="192.168.0.42",command=... <pubkey>...
-
-For even more security, set up a chroot environment in
-"/etc/ssh/sshd_config" (see [sshd_config(5)]).
 
 
   [ssh_filter_btrbk(1)]: https://digint.ch/btrbk/doc/ssh_filter_btrbk.1.html
@@ -430,10 +574,10 @@ For even more security, set up a chroot environment in
 Restoring Backups
 =================
 
-btrbk does not provide any mechanism to restore your backups, this has
+Btrbk does not provide any mechanism to restore your backups, this has
-to be done manually. In the examples below, we assume that you have a
+to be done manually. In the instructions below, we assume that you
-btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you want to
+have a btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you
-have restored is at `/mnt/btr_pool/data`.
+want to restore is at `/mnt/btr_pool/data`.
 
 **Important**: don't use `btrfs property set` to make a subvolume
 read-write after restoring. This is a low-level command, and leaves
@@ -442,68 +586,87 @@ subsequent incremental backups. Instead, use `btrfs subvolume
 snapshot` (without `-r` flag) as described below.
 
 
-Example: Restore a Snapshot
+### Step 0: Identify Subvolume
------------------------------
-
-First, pick a snapshot to be restored:
 
+    # list snapshots managed by btrbk
     btrbk list snapshots
 
-From the list, pick the snapshot you want to restore. Let's say it's
+    # alternative: list all subvolumes
+    btrbk ls /
+    btrbk ls -L /
+
+From the list, identify the snapshot you want to restore. Let's say it's
 `/mnt/btr_pool/_btrbk_snap/data.20150101`.
 
-If the broken subvolume is still present, move it away:
 
-    mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
+### Step 1: Restore Backup
-
+(skip this step if you restore from a snapshot)
-Now restore the snapshot:
-
-    btrfs subvolume snapshot /mnt/btr_pool/_btrbk_snap/data.20150101 /mnt/btr_pool/data
-
-That's it; your `data` subvolume is restored. If everything went fine,
-it's time to nuke the broken subvolume:
-
-    btrfs subvolume delete /mnt/btr_pool/data.BROKEN
-
-
-Example: Restore a Backup
--------------------------
-
-First, pick a backup to be restored:
-
-    btrbk list backups
-
-From the list, pick the backup you want to restore. Let's say it's
-`/mnt/btr_backup/data.20150101`.
-
-If the broken subvolume is still present, move it away:
-
-    mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
-
-Now restore the backup:
 
+    # locally mounted backup disk
     btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
+
+    # from / to remote host
+    ssh root@remote btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
+    btrfs send /mnt/btr_backup/data.20150101 | ssh root@remote btrfs receive /mnt/btr_pool/
+
+**Hint**: Try to send-receive backups incrementally if possible. In
+case you still have common snapshot / backup pairs (i.e. both
+"snapshot_subvol" and "target_subvol" are listed above), use `btrfs
+send -p <parent>`.
+
+From this point on, `data.20150101` on both disks can be used as
+parents for subsequent send-receive operations, and a *received_uuid*
+relationship is established (see below).
+
+
+### Step 2: Create read-write Subvolume
+
+    # if still present, move broken subvolume away
+    mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
+
+    # create read-write subvolume
     btrfs subvolume snapshot /mnt/btr_pool/data.20150101 /mnt/btr_pool/data
-    btrfs subvolume delete /mnt/btr_pool/data.20150101
 
-Alternatively, if you're restoring data on a remote host, do something
+Your `data` subvolume is restored, you can carry on with incremental
-like this:
+backups to `/mnt/btr_backup`.
 
-    btrfs send /mnt/btr_backup/data.20150101 | ssh root@my-remote-host.com btrfs receive /mnt/btr_pool/
 
-Hint: If you still have common snapshot / backup pairs (i.e. both
+### Step 3: Cleanup
-"snapshot_subvol" and "target_subvol" are listed) consider sending the
-backup incrementally by specifying a parent subvolume:
-
-    btrfs send -p /mnt/btr_backup/<parent-subvolume> [...]
-
-This allows btrbk to continue using the parent subvolume for
-incremental backups later.
-
-If everything went fine, nuke the broken subvolume:
 
+    # if everything went fine, delete the broken subvolume
     btrfs subvolume delete /mnt/btr_pool/data.BROKEN
 
+Make sure to keep `data.20150101` subvolumes on both disks at least
+until you created a new backup using btrbk, in order to keep the
+incremental chain alive.
+
+
+Btrfs Relationship (technical note)
+-----------------------------------
+
+    btrbk origin -t /mnt/btr_backup/data.20150101
+    btrbk ls -L /mnt/btr_pool /mnt/btr_backup
+
+  * **received_uuid** relationship: *correlated*, *identical*
+    read-only subvolumes, cross-filesystem.
+
+        a.received_uuid = b.received_uuid
+        a.received_uuid = b.uuid
+
+    * Required for subvolumes used as parent (or clone-src) of
+      send-receive operations.
+    * Present on subvolumes created by `btrfs send | btrfs receive`.
+    * `/mnt/btr_pool/data.20150101 === /mnt/btr_backup/data.20150101`
+
+  * **parent_uuid** relationship: "is-snapshot-of"
+
+        a.parent_uuid = b.uuid
+
+    * Present on subvolumes created by `btrfs subvolume snapshot` or
+      `btrfs send -p | btrfs receive`.
+    * Used by btrbk to determine best parent.
+    * `/mnt/btr_pool/data.20150101 <-- /mnt/btr_pool/data`
+
 
 FAQ
 ===
@@ -517,8 +680,8 @@ Donate
 
 So btrbk saved your day?
 
-I will definitively continue developing btrbk for free, but if you
+I will definitively continue to develop btrbk for free. If you want to
-want to support me you can do so:
+support my hard work with a donation, you are welcome to do so!
 
 [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S)
 
@@ -549,7 +712,7 @@ If you would like to contribute or have found bugs:
 
   * Visit the [btrbk project page on GitHub] and use the
     [issues tracker] there.
-  * Talk to us on Freenode in `#btrbk`.
+  * Talk to us on [Libera.Chat] in `#btrbk`.
   * Contact the author via email (the email address can be found in
     the sources).
 
@@ -557,6 +720,7 @@ Any feedback is appreciated!
 
   [btrbk project page on GitHub]: https://github.com/digint/btrbk
   [issues tracker]: https://github.com/digint/btrbk/issues
+  [Libera.Chat]: https://libera.chat
 
 
 License

btrbk.conf.example

@@ -17,6 +17,13 @@
 # Enable transaction log
 transaction_log            /var/log/btrbk.log
 
+# Specify SSH private key for remote connections
+ssh_identity               /etc/btrbk/ssh/id_ed25519
+ssh_user                   root
+
+# Use sudo if btrbk or lsbtr is run by regular user
+backend_local_user         btrfs-progs-sudo
+
 # Enable stream buffer. Adding a buffer between the sending and
 # receiving side is generally a good idea.
 # NOTE: If enabled, make sure to install the "mbuffer" package!
@@ -29,7 +36,7 @@ stream_buffer              256m
 # If you want to set a custom name for the snapshot (and backups),
 # use the "snapshot_name" option within the subvolume section.
 #
-# NOTE: btrbk does not autmatically create this directory, and the
+# NOTE: btrbk does not automatically create this directory, and the
 # snapshot creation will fail if it is not present.
 #
 snapshot_dir               _btrbk_snap
@@ -67,12 +74,6 @@ snapshot_dir               _btrbk_snap
 #archive_preserve_min       no
 #archive_preserve           <NN>h <NN>d <NN>w <NN>m <NN>y
 
-# Specify SSH private key for "ssh://" volumes / targets:
-#ssh_identity               /etc/btrbk/ssh/id_ed25519
-#ssh_user                   root
-#ssh_compression            no
-#ssh_cipher_spec            default
-
 # Enable compression for remote btrfs send/receive operations:
 #stream_compress            no
 #stream_compress_level      default
@@ -82,19 +83,19 @@ snapshot_dir               _btrbk_snap
 # can be run at a time.
 #lockfile                   /var/lock/btrbk.lock
 
-# Don't wait for transaction commit on deletion. Set this to "after"
+# Don't wait for transaction commit on deletion. Enable this to make
-# or "each" to make sure the deletion of subvolumes is committed to
+# sure the deletion of subvolumes is committed to disk when btrbk
-# disk when btrbk terminates.
+# terminates.
 #btrfs_commit_delete no
 
 
 #
-# Volume section: "volume <volume-directory>"
+# Volume section (optional): "volume <volume-directory>"
 #
-#   <volume-directory>  Directory of a btrfs volume (or subvolume)
+#   <volume-directory>  Base path within a btrfs filesystem
-#                       containing the subvolume to be backuped
+#                       containing the subvolumes to be backuped
 #                       (usually the mount-point of a btrfs filesystem
-#                       mounted with subvolid=5 option)
+#                       mounted with subvolid=5 option).
 #
 # Subvolume section: "subvolume <subvolume-name>"
 #
@@ -104,19 +105,19 @@ snapshot_dir               _btrbk_snap
 # Target section: "target <type> <volume-directory>"
 #
 #   <type>              (optional) type, defaults to "send-receive".
-#   <volume-directory>  Directory of a btrfs volume (or subvolume)
+#   <volume-directory>  Directory within a btrfs filesystem
 #                       receiving the backups.
 #
 # NOTE: The parser does not care about indentation, this is only for
-# human readability. The options always apply to the last section
+# human readability. All options apply to the last section
 # encountered, overriding the corresponding option of the upper
-# section. This means that the global options must be set before any
+# section. This means that the global options must be set on top,
-# "volume" section.
+# before any "volume", "subvolume" or "target section.
 #
 
 
 #
-# Example configuration:
+# Example retention policy:
 #
 snapshot_preserve_min   2d
 snapshot_preserve       14d
@@ -124,29 +125,49 @@ snapshot_preserve       14d
 target_preserve_min     no
 target_preserve         20d 10w *m
 
-archive_preserve_min    latest
-archive_preserve        12m 10y
 
+#
+# Simple setup: Backup root and home to external disk
+#
+snapshot_dir /btrbk_snapshots
+target       /mnt/btr_backup
+subvolume    /
+subvolume    /home
+
+
+#
+# Complex setup
+#
+# In order to keep things organized, it is recommended to use "volume"
+# sections and mount the top-level subvolume (subvolid=5):
+#
+#  $ mount -o subvolid=5 /dev/sda1 /mnt/btr_pool
+#
 # Backup to external disk mounted on /mnt/btr_backup
 volume /mnt/btr_pool
-  # no action if external disk is not attached
+  # Create snapshots in /mnt/btr_pool/btrbk_snapshots
-  snapshot_create  ondemand
+  snapshot_dir btrbk_snapshots
 
-  # propagates to all subvolume sections:
+  # Target for all subvolume sections:
-  target /mnt/btr_backup/_btrbk
+  target /mnt/btr_backup
 
-  subvolume  root_gentoo
+  # Some default btrfs installations (e.g. Ubuntu) use "@" for rootfs
+  # (mounted at "/") and "@home" (mounted at "/home"). Note that this
+  # is only a naming convention.
+  #subvolume @
+  subvolume root
+  subvolume home
   subvolume kvm
-    # use different retention policy for kvm backups
+    # Use different retention policy for kvm backups:
     target_preserve 7d 4w
 
 
-# Backup to external disk as well as some remote host
+# Backup data to external disk as well as remote host
 volume /mnt/btr_data
-  subvolume  home
+  subvolume  data
-    # always create snapshot, even if targets are unreachable
+    # Always create snapshot, even if targets are unreachable
     snapshot_create always
-    target /mnt/btr_backup/_btrbk
+    target /mnt/btr_backup
     target ssh://backup.my-remote-host.com/mnt/btr_backup
 
 
@@ -155,7 +176,22 @@ volume ssh://my-remote-host.com/mnt/btr_pool
   subvolume data_0
     snapshot_dir   snapshots/btrbk
     snapshot_name  data_main
-    target /mnt/btr_backup/_btrbk/my-remote-host.com
+    target /mnt/btr_backup/my-remote-host.com
+
+
+# Backup on demand (noauto) to remote host running busybox, login as
+# regular user using ssh-agent with current user name (ssh_user no)
+# and default credentials (ssh_identity no).
+volume /home
+  noauto  yes
+  compat  busybox
+  backend_remote  btrfs-progs-sudo
+  ssh_user      no
+  ssh_identity  no
+
+  target ssh://my-user-host.com/mnt/btr_backup/home
+  subvolume  alice
+  subvolume  bob
 
 
 # Resume backups from remote host which runs its own btrbk instance
@@ -165,4 +201,4 @@ volume ssh://my-remote-host.com/mnt/btr_pool
   snapshot_create        no
   snapshot_preserve_min  all
   subvolume home
-    target /mnt/btr_backup/_btrbk/my-remote-host.com
+    target /mnt/btr_backup/my-remote-host.com

contrib/bash/completion.bash

@@ -0,0 +1,133 @@
+_btrbk_init_cmds()
+{
+  # set $cmds to an array of the commands so far
+  #
+  # for example, for this command:
+  #
+  #     btrbk -v --override warn_unknown_targets=yes list config --long
+  #
+  # then $cmds is:
+  #
+  #     cmds=(list config)
+  #
+  cmds=()
+
+  local i
+  for ((i = 1; i < cword; i++)); do
+    case "${words[i-1]}" in
+      '-c' | '--config' | '--exclude' | '-l' | '--loglevel' | '--format' | '--lockfile' | '--override')
+        continue
+        ;;
+    esac
+    [[ ${words[i]} != -* ]] && cmds+=(${words[i]})
+  done
+
+  return 0
+}
+
+_btrbk()
+{
+  local cur prev words cword split cmds
+  _init_completion -s || return
+  _btrbk_init_cmds || return
+
+  case "$prev" in
+    '-c' | '--config')
+      _filedir
+      return
+      ;;
+    '--exclude')
+      return
+      ;;
+    '-l' | '--loglevel')
+      COMPREPLY=($(compgen -W 'error warn info debug trace' -- "$cur"))
+      return
+      ;;
+    '--format')
+      COMPREPLY=($(compgen -W 'table long raw' -- "$cur"))
+      return
+      ;;
+    '--lockfile')
+      _filedir
+      return
+      ;;
+    '--override')
+      return
+      ;;
+  esac
+
+  $split && return
+
+  if [[ $cur == -* ]]; then
+    COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur"))
+    [[ $COMPREPLY == *= ]] && compopt -o nospace
+  else
+    if [[ ! -v 'cmds[0]' ]]; then
+      COMPREPLY=($(compgen -W 'run dryrun snapshot resume prune archive clean stats list usage origin diff extents ls' -- "$cur"))
+    fi
+  fi
+
+  case "${cmds[0]}" in
+    'archive')
+      # <source>
+      if [[ ! -v 'cmds[1]' ]]; then
+        _filedir -d
+      # <target>
+      elif [[ ! -v 'cmds[2]' ]]; then
+        _filedir -d
+      # [--raw]
+      elif [[ $cur == -* ]]; then
+        COMPREPLY+=($(compgen -W '--raw' -- "$cur"))
+      fi
+      ;;
+    'list')
+      if [[ ! -v 'cmds[1]' ]]; then
+        COMPREPLY=($(compgen -W 'all snapshots backups latest config source volume target' -- "$cur"))
+      fi
+      ;;
+    'origin')
+      # <subvolume>
+      if [[ ! -v 'cmds[1]' ]]; then
+        _filedir -d
+      fi
+      ;;
+    'ls')
+      # <path>|<url>...
+      _filedir -d
+      ;;
+    'extents')
+      # [diff] <path>... [exclusive] <path>...
+      if [[ ! -v 'cmds[1]' ]]; then
+          COMPREPLY+=($(compgen -W 'diff' -- "$cur"))
+      elif [[ ! ${cmds[*]} =~ (^|[[:space:]])"exclusive"($|[[:space:]]) ]]; then
+          COMPREPLY+=($(compgen -W 'exclusive' -- "$cur"))
+      fi
+      _filedir -d
+      ;;
+  esac
+} && complete -F _btrbk btrbk
+
+_lsbtr()
+{
+  local cur prev words cword split
+  _init_completion -s || return
+
+  case "$prev" in
+    '-c' | '--config')
+      _filedir
+      ;;
+    '--override')
+      ;;
+  esac
+  $split && return
+
+  if [[ $cur == -* ]]; then
+    COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur"))
+    [[ $COMPREPLY == *= ]] && compopt -o nospace
+  else
+    # <path>|<url>...
+    _filedir -d
+  fi
+} && complete -F _lsbtr lsbtr
+
+# ex: filetype=bash

contrib/cron/btrbk-mail

@@ -2,154 +2,277 @@
 
 ## Wrapper script running "btrbk" and sending email with results
 
-set -uf
-declare -A rsync_src rsync_dst rsync_log rsync_key rsync_opt
 now=$(date +%Y%m%d)
 
+declare -A rsync_src rsync_dst rsync_log rsync_rsh rsync_opt
+declare -A sync_fs_onchange
 
 #####  start config section  #####
 
 # Email recipients, separated by whitespace:
-mailto=$MAILTO
+mailto=${MAILTO:-root}
-
-# List of mountpoints to be mounted/unmounted (whitespace-separated)
-# mount_targets="/mnt/btr_pool /mnt/backup"
-mount_targets=
-umount_targets=$mount_targets
-
-# btrbk configuration file:
-config="/etc/btrbk/btrbk.conf"
-
-# Uncomment this if you only want to receive error messages:
-#btrbk_opts="-q"
-#skip_empty_mail=yes
 
 # Email subject:
 mail_subject_prefix="btrbk <${HOSTNAME:-localhost}>"
 
-# rsync declarations (repeat complete block for more declarations):
+# Add summary and/or detail (rsync/btrbk command output) to mail body.
-rsync_src[example_data]=user@example.com:/data/
+# If both are not set, a mail is only sent on errors.
-rsync_dst[example_data]=/mnt/backup/example.com/data/
+mail_summary=yes
-rsync_log[example_data]=/mnt/backup/example.com/data-${now}.log
+mail_detail=no
-rsync_key[example_data]=/mnt/backup/ssh_keys/id_rsa
-rsync_opt[example_data]="-az --inplace --delete"
 
-# Enabled rsync declarations (space separated list)
+# List of mountpoints to be mounted/unmounted (whitespace-separated)
+# mount_targets="/mnt/btr_pool /mnt/backup"
+mount_targets=
+
+# rsync declarations (repeat complete block for more declarations):
+rsync_src[example_data]="user@example.com:/data/"
+rsync_dst[example_data]="/mnt/backup/example.com/data/"
+rsync_log[example_data]="/mnt/backup/example.com/data-${now}.log"
+rsync_rsh[example_data]="ssh -i /mnt/backup/ssh_keys/id_rsa"
+rsync_opt[example_data]="-az --delete --inplace --numeric-ids --acls --xattrs"
+# If set, add "rsync_dst" to "sync_fs" (see below) if rsync reports files transferred
+#sync_fs_onchange[example_data]=yes
+
+# Enable all rsync declarations (all indices of rsync_src array)
+#rsync_enable=${!rsync_src[@]}
+# Explicitly enable rsync declarations (whitespace-separated list)
 #rsync_enable="example_data"
 rsync_enable=
 
-# Log level (1=error, 2=warn, 3=info)
+# If set, do not run btrbk if rsync reports no changes.
-loglevel=2
+# If set to "quiet", do not send mail.
+#skip_btrbk_if_unchanged=quiet
 
+# Array of directories to sync(1) prior to running btrbk. This is
+# useful for source subvolumes having "snapshot_create ondemand"
+# configured in btrbk.conf.
+#sync_fs=("/mnt/btr_data" "/mnt/btr_pool")
+
+# btrbk command / options:
+btrbk_command="run"
+btrbk_opts="-c /etc/btrbk/btrbk.conf"
+
+
+### Layout options:
+
+# Prefix command output: useful when using mail clients displaying
+# btrbk summary lines starting with ">>>" as quotations.
+#mail_cmd_block_prefix='\\u200B' # zero-width whitespace
+#mail_cmd_block_prefix=". "
+
+# Newline character
+BR=$'\n'
 
 #####  end config section  #####
 
-mail_body=""
+
+check_options()
+{
+    [[ -n "$btrbk_command" ]] || die "btrbk_command is not set"
+    for key in $rsync_enable; do
+        [[ -n "${rsync_src[$key]}" ]] || die "rsync_src is not set for \"$key\""
+        [[ -n "${rsync_dst[$key]}" ]] || die "rsync_dst is not set for \"$key\""
+        [[ -n "${rsync_opt[$key]}" ]] || die "rsync_opt is not set for \"$key\""
+    done
+}
+
+send_mail()
+{
+    # assemble mail subject
+    local subject="$mail_subject_prefix"
+    [[ -n "$has_errors" ]] && subject+=" ERROR";
+    [[ -n "$status"     ]] && subject+=" - $status";
+    [[ -n "$xstatus"    ]] && subject+=" (${xstatus:2})";
+
+    # assemble mail body
+    local body=
+    if [[ -n "$info" ]] && [[ -n "$has_errors" ]] || [[ "${mail_summary:-no}" = "yes" ]]; then
+        body+="$info"
+    fi
+    if [[ -n "$detail" ]] && [[ -n "$has_errors" ]] || [[ "${mail_detail:-no}" = "yes" ]]; then
+        [[ -n "$body" ]] && body+="${BR}${BR}DETAIL:${BR}"
+        body+="$detail"
+    fi
+
+    # skip sending mail on empty body
+    if [[ -z "$body" ]] && [[ -n "$has_errors" ]]; then
+        body+="FATAL: something went wrong (errors present but empty mail body)${BR}"
+    fi
+    [[ -z "$body" ]] && exit 0
+
+    # send mail
+    echo "$body" | mail -s "$subject" $mailto
+    if [[ $? -ne 0 ]]; then
+        echo "$0: Failed to send btrbk mail to \"$mailto\", dumping mail:${BR}" 1>&2
+        echo "<mail_subject>$subject</mail_subject>${BR}<mail_body>${BR}$body</mail_body>" 1>&2
+    fi
+}
+
+einfo()
+{
+    info+="$1${BR}"
+}
+
+ebegin()
+{
+    ebtext=$1
+    detail+="${BR}### $1${BR}"
+}
+
+eend()
+{
+    if [[ $1 -eq 0 ]]; then
+        eetext=${3-success}
+        detail+="${BR}"
+    else
+        has_errors=1
+        eetext="ERROR (code=$1)"
+        [[ -n "$2" ]] && eetext+=": $2"
+        detail+="${BR}### $eetext${BR}"
+    fi
+    info+="$ebtext: $eetext${BR}"
+    return $1
+}
 
 die()
 {
-    echo "$0 FATAL: $1"      1>&2
+    einfo "FATAL: ${1}, exiting"
-    echo "$0 FATAL: exiting" 1>&2
+    has_errors=1
+    send_mail
     exit 1
 }
-log_error()   { [ $loglevel -ge 1 ] && echo "$0 ERROR: $1"   1>&2 ; }
-log_warning() { [ $loglevel -ge 2 ] && echo "$0 WARNING: $1" 1>&2 ; }
-log_info()    { [ $loglevel -ge 3 ] && echo "$0 INFO: $1"    1>&2 ; }
 
-
+run_cmd()
-#
+{
-# mount all mountpoints listed in $mount_targets
+    cmd_out=$("$@" 2>&1)
-#
+    local ret=$?
-for mountpoint in $mount_targets; do
+    detail+="++ ${@@Q}${BR}"
-    $(findmnt -r -n -t btrfs $mountpoint 1>&2)
+    if [[ -n "${mail_cmd_block_prefix:-}" ]] && [[ -n "$cmd_out" ]]; then
-    if [ $? = 0 ]; then
+        detail+=$(echo -n "$cmd_out" | sed "s/^/${mail_cmd_block_prefix}/")
-	log_warning "btrfs filesystem already mounted: $mountpoint"
+        detail+="${BR}"
     else
-	log_info "mount $mountpoint"
+        detail+=$cmd_out
-	$(mount --target $mountpoint 1>&2)
+    fi
-	[ $? = 0 ] || log_error "mount failed: $mountpoint"
+    return $ret
+}
+
+mount_all()
+{
+    # mount all mountpoints listed in $mount_targets
+    mounted=""
+    for mountpoint in $mount_targets; do
+        ebegin "Mounting $mountpoint"
+        run_cmd findmnt -n $mountpoint
+        if [[ $? -eq 0 ]]; then
+            eend -1 "already mounted"
+        else
+            detail+="${BR}"
+            run_cmd mount --target $mountpoint
+            eend $? && mounted+=" $mountpoint"
         fi
     done
+}
+
+umount_mounted()
+{
+    for mountpoint in $mounted; do
+        ebegin "Unmounting $mountpoint"
+        run_cmd umount $mountpoint
+        eend $?
+    done
+}
+
+
+check_options
+mount_all
 
 
 #
 # run rsync for all $rsync_enable
 #
 for key in $rsync_enable; do
-    log_info "starting rsync: $key"
+    ebegin "Running rsync[$key]"
+    if [[ -d "${rsync_dst[$key]}" ]]; then
+        # There is no proper way to get a proper machine readable
+        # output of "rsync did not touch anything at destination", so
+        # we add "--info=stats2" and parse the output.
+        # NOTE: This also appends the stats to the log file (rsync_log).
+        # Another approach to count the files would be something like:
+        # "rsync --out-format='' | wc -l"
+        run_cmd rsync ${rsync_opt[$key]} \
+                    --info=stats2 \
+                    ${rsync_log[$key]:+--log-file="${rsync_log[$key]}"} \
+                    ${rsync_rsh[$key]:+-e "${rsync_rsh[$key]}"} \
+                    "${rsync_src[$key]}" \
+                    "${rsync_dst[$key]}"
+        exitcode=$?
 
-    [ -n "${rsync_src[$key]}" ] || die "rsync_src is not set for \"$key\""
+        # parse stats2 (count created/deleted/transferred files)
-    [ -n "${rsync_dst[$key]}" ] || die "rsync_dst is not set for \"$key\""
+        REGEXP=$'\n''Number of created files: ([0-9]+)'
-    [ -n "${rsync_log[$key]}" ] || die "rsync_log is not set for \"$key\""
+        REGEXP+='.*'$'\n''Number of deleted files: ([0-9]+)'
-    [ -n "${rsync_key[$key]}" ] || die "rsync_key is not set for \"$key\""
+        REGEXP+='.*'$'\n''Number of regular files transferred: ([0-9]+)'
-    [ -n "${rsync_opt[$key]}" ] || die "rsync_opt is not set for \"$key\""
+        if [[ $cmd_out =~ $REGEXP ]]; then
-
+            rsync_stats="${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}"
-    rsync_header="### rsync ${rsync_opt[$key]} ${rsync_src[$key]} ${rsync_dst[$key]}"
+            rsync_stats_long="${BASH_REMATCH[1]} created, ${BASH_REMATCH[2]} deleted, ${BASH_REMATCH[3]} transferred"
-
+            nfiles=$(( ${BASH_REMATCH[1]} + ${BASH_REMATCH[2]} + ${BASH_REMATCH[3]} ))
-    if [ -d ${rsync_dst[$key]} ]; then
-	echo "$rsync_header" >> ${rsync_log[$key]}
-	ret=$(rsync ${rsync_opt[$key]} --info=STATS --log-file=${rsync_log[$key]} -e "ssh -i ${rsync_key[$key]}" ${rsync_src[$key]} ${rsync_dst[$key]})
-	if [ $? != 0 ]; then
-	    log_error "rsync failed: $key"
-	    ret+="\nERROR: rsync failed with exit code $?\n"
-	fi
-	mail_body+="$rsync_header$ret\n\n"
         else
-	ret="rsync destination directory not found for \"$key\", skipping: ${rsync_dst[$key]}"
+            rsync_stats_long="failed to parse stats, assuming files transferred"
-	mail_body+="$rsync_header\n$ret\n\n"
+            rsync_stats="-1/-1/-1"
-	log_error "$ret"
+            nfiles=-1
+        fi
+
+        eend $exitcode "$rsync_stats_long" "$rsync_stats_long"
+        xstatus+=", rsync[$key]=$rsync_stats"
+
+        if [[ $nfiles -ne 0 ]]; then
+            # NOTE: on error, we assume files are transferred
+            rsync_files_transferred=1
+            [[ -n "${sync_fs_onchange[$key]}" ]] && sync_fs+=("${rsync_dst[$key]}")
+        fi
+    else
+        eend -1 "Destination directory not found, skipping: ${rsync_dst[$key]}"
     fi
 done
 
+# honor skip_btrbk_if_unchanged (only if rsync is enabled and no files were transferred)
+if [[ -n "$rsync_enable" ]] && [[ -n "$skip_btrbk_if_unchanged" ]] && [[ -z "$rsync_files_transferred" ]]; then
+     einfo "No files transferred, exiting"
+     status="No files transferred"
+     umount_mounted
+     if [[ "$skip_btrbk_if_unchanged" != "quiet" ]] || [[ -n "$has_errors" ]]; then
+         send_mail
+     fi
+     exit 0
+fi
+
+
+#
+# sync filesystems in sync_fs
+#
+if [[ ${#sync_fs[@]} -gt 0 ]]; then
+    ebegin "Syncing filesystems at ${sync_fs[@]}"
+    run_cmd sync -f "${sync_fs[@]}"
+    eend $?
+fi
+
 
 #
 # run btrbk
 #
-log_info "running btrbk"
+ebegin "Running btrbk"
-ret=$(btrbk -c "$config" ${btrbk_opts:-} run 2>&1)
+run_cmd btrbk ${btrbk_opts:-} ${btrbk_command}
 exitcode=$?
 case $exitcode in
     0)  status="All backups successful"
         ;;
     3)  status="Another instance of btrbk is running, no backup tasks performed!"
         ;;
-    10) status="ERROR: At least one backup task aborted!"
+    10) status="At least one backup task aborted!"
          ;;
-    *)  status="ERROR: btrbk failed with error code $exitcode"
+    *)  status="btrbk failed with error code $exitcode"
         ;;
 esac
+eend $exitcode "$status"
 
-mail_body+=$ret
+umount_mounted
-
+send_mail
-if [ "${skip_empty_mail:-no}" = "yes" ] && [ -z "$mail_body" ] && [ $exitcode -eq 0 ]; then
-    : # skip email sending if skip_empty_mail=yes
-else
-    # send email
-    echo -e "$mail_body" | mail -s "$mail_subject_prefix - $status" $mailto
-    if [ $? != 0 ]; then
-	log_error "failed to send btrbk mail to \"$mailto\", dumping mail body:"
-	echo -e "$mail_body" 1>&2
-    fi
-fi
-
-
-#
-# sync all mountpoints listed in $umount_targets
-#
-# exit on failure!
-#for mountpoint in $umount_targets; do
-#    log_info "btrfs filesystem sync $mountpoint"
-#    $(btrfs filesystem sync $mountpoint 1>&2)
-#    [ $? = 0 ] || die "btrfs filesystem sync failed: $mountpoint"
-#    sleep 1
-#done
-
-
-#
-# unmount all mountpoints listed in $umount_targets
-#
-for mountpoint in $umount_targets; do
-    log_info "umount $mountpoint"
-    $(umount $mountpoint 1>&2)
-    [ $? = 0 ] || log_error "umount failed: $mountpoint"
-done

contrib/cron/btrbk-verify

@@ -31,7 +31,7 @@
 #   NOTE: Depending on your setup (hardware, btrfs mount options),
 #   btrbk-verify may eat all your CPU power and use high bandwidth!
 #   Consider nice(1), ionice(1).
-
+#
 #   Incomplete resource eater list:
 #    - rsync: checksums, heavy disk I/O
 #    - btrfs: decompression, encryption
@@ -77,7 +77,7 @@ set -u
 set -e
 set -o pipefail
 
-btrbk_version_min='0.28.0'
+btrbk_version_min='0.32.0'
 
 # defaults: ignore subvol dirs and root folder timestamp change
 ignore_nested_subvolume_dir=1
@@ -107,8 +107,8 @@ options:
    --ignore-xattrs       ignore xattrs when verifying subvolumes
    --ssh-identity FILE   override ssh_identity from btrbk.conf(5) with FILE,
                          and clear all other ssh_* options (use with --ssh-user)
-   --ssh-user USER       override ssh_user from btrbk.conf(5) with USER
+   --ssh-user USER       override ssh_user from btrbk.conf(5) with USER, and
-                         (only in conjunction with --ssh-identity)
+                         clear all other ssh_* options(use with --ssh-identity)
    --ssh-agent           start ssh-agent(1) and add identity
 
 commands:
@@ -157,7 +157,6 @@ while [[ "$#" -ge 1 ]]; do
           ;;
       --ssh-identity)
           # use different ssh identity (-i option) for rsync rsh.
-          # if set, ssh_user defaults to root.
           # NOTE: this overrides all btrbk ssh_* options
           ssh_identity="$2"
           shift
@@ -186,6 +185,8 @@ while [[ "$#" -ge 1 ]]; do
     shift
 done
 
+BR=$'\n'
+
 log_line()
 {
     echo "$@" 1>&2
@@ -208,13 +209,13 @@ tlog()
     [[ -n "$dryrun" ]] && [[ "$status" == "starting" ]] && status="dryrun_starting"
     local line="$(date --iso-8601=seconds) verify-rsync ${status} ${target} ${source} - -"
     [[ -n "$comment" ]] && line="$line # $comment";
-    tlog_text+="$line\n"
+    tlog_text+="$line${BR}"
     log_debug "$line"
 }
 tlog_print()
 {
     # tlog goes to stdout
-    echo -e "\nTRANSACTION LOG\n---------------\n${tlog_text:-}"
+    echo "${BR}TRANSACTION LOG${BR}---------------${BR}${tlog_text:-}"
 }
 
 # parse "rsync -i,--itemize-changes" output.
@@ -267,11 +268,14 @@ rsync_rsh()
     local rsh_match="(.*) ([a-z0-9_-]+)@([a-zA-Z0-9.-]+)$"
 
     if [[ -z "$rsh" ]]; then
-        echo
+        return
-    elif [[ -n "$ssh_identity" ]]; then
+    elif [[ -n "$ssh_user" ]] || [[ -n "$ssh_identity" ]]; then
         # override btrbk.conf from command line arguments
         log_debug "Overriding all ssh_* options from btrbk.conf"
-        echo "ssh -q -i $ssh_identity -l $ssh_user"
+        local cmd="ssh -q"
+        [[ -n "$ssh_identity" ]] && cmd="$cmd -i '$ssh_identity'"
+        [[ -n "$ssh_user" ]] && cmd="$cmd -l '$ssh_user'"
+        echo "$cmd"
     elif [[ $rsh =~ $rsh_match ]]; then
         echo "${BASH_REMATCH[1]} -l ${BASH_REMATCH[2]}"
     else
@@ -302,7 +306,7 @@ start_ssh_agent()
 
 eval_btrbk_resolved_line()
 {
-    local line=$1
+    local line=" $1"
     local prefix=$2
     local required_keys=$3
     # reset all variables first
@@ -311,25 +315,13 @@ eval_btrbk_resolved_line()
     done
     for vv in $required_keys; do
         # basic input validation, set prefixed variable (eval)
-        local var_assignment=$(echo "$line" | grep -Eo "${vv}"'="[^"]*"')
+        local match=" ${vv}='([^']*('\\\\''[^']*)*)'"
-        if [[ $? -ne 0 ]] || [[ -z "$var_assignment" ]]; then
+        if [[ $line =~ $match ]] ; then
-            if [[ $vv == "target_type" ]]; then
+            eval "${prefix}${vv}='${BASH_REMATCH[1]}'" || return 1
-                # hack for btrbk <= 0.27.2 bug: missing "target_type" in $table_formats{resolved}
+        else
-                # hardcode send-receive type, this results in rsync failing on raw targets with:
+            log_line "btrbk-verify: ERROR: Missing variable \"${vv}\""
-                # rsync: ERROR: cannot stat destination "xxx.btrfs.xz.gpg/": Not a directory (20)
-                eval "${prefix}target_type=\"send-receive\"" || return 1
-                continue
-            fi
-            if [[ $vv == "target_rsh" ]] && [[ -z "$R_target_host" ]]; then
-                # hack for btrbk <= 0.27.2 bug: missing "target_rsh" in $table_formats{resolved}
-                # note that target_rsh is not needed if target_host is not set.
-                # variable has already been reset above, continue
-                continue
-            fi
-            log_line "btrbk-verify: missing required variable \"${vv}\" in btrbk --format=raw line"
             return 1
         fi
-        eval "${prefix}${var_assignment}" || return 1
     done
 }
 
@@ -339,10 +331,6 @@ exit_trap_action()
     [[ $verbose -gt 0 ]] && tlog_print
 }
 
-# restrictions from rsync_rsh():
-[[ -z "$ssh_identity" ]] && [[ -n "$ssh_user" ]] && print_usage 2
-[[ -n "$ssh_identity" ]] && [[ -z "$ssh_user" ]] && print_usage 2
-
 # start ssh-agent(1)
 [[ -n "$ssh_start_agent" ]] && start_ssh_agent
 
@@ -371,20 +359,22 @@ while read -r btrbk_list_line; do
     log_debug "Evaluating [btrbk list] line: $btrbk_list_line"
     [[ -z "$btrbk_list_line" ]] && continue
     if ! eval_btrbk_resolved_line "$btrbk_list_line" \
-         "R_" "snapshot_path target_path source_host target_host target_type source_rsh target_rsh"
+         "R_" "snapshot_subvolume target_subvolume source_host target_host target_type source_rsh target_rsh"
     then
-        log_line "btrbk-verify: WARNING: Skipping task (parse error). Make sure to have >=btrbk-${btrbk_version_min} installed!"
+        log_line "btrbk-verify: ERROR: Parse error of command output: ${btrbk_cmd[@]}"
-        continue
+        log_line "Make sure to have >=btrbk-${btrbk_version_min} installed!"
+        exitstatus=1
+        break
     fi
 
-    source="${R_snapshot_path}"
+    source="${R_snapshot_subvolume}/"
-    target="${R_target_path}"
+    target="${R_target_subvolume}/"
     [[ -n "$R_source_host" ]] && source="${R_source_host}:${source}"
     [[ -n "$R_target_host" ]] && target="${R_target_host}:${target}"
 
-    if [[ -z "$R_snapshot_path" ]]; then
+    if [[ -z "$R_snapshot_subvolume" ]]; then
         log_line "WARNING: Skipping task (missing snapshot): target=$target"
-    elif [[ -z "$R_target_path" ]]; then
+    elif [[ -z "$R_target_subvolume" ]]; then
         log_line "Skipping task (no target): source=$source"
     elif [[ "$R_target_type" != "send-receive" ]]; then
         log_line "Skipping task (target_type=$R_target_type): source=$source, target=$target"
@@ -396,10 +386,11 @@ while read -r btrbk_list_line; do
         # rsync rsh is either source_rsh or target_rsh or empty
         eff_rsh="$R_source_rsh"
         [[ -z "$eff_rsh" ]] && eff_rsh="$R_target_rsh"
+        eff_rsh=$(rsync_rsh "$eff_rsh")
 
         rsync_cmd=("rsync" "${rsync_args[@]}")
-        [[ -n "$eff_rsh" ]] && rsync_cmd+=(-e "$(rsync_rsh "$eff_rsh")")
+        [[ -n "$eff_rsh" ]] && rsync_cmd+=("-e" "$eff_rsh")
-        rsync_cmd+=("${source}/" "${target}/")
+        rsync_cmd+=("${source}" "${target}")
         log_cmd "${rsync_cmd[@]}"
         [[ -n "$dryrun" ]] && rsync_cmd=("cat" "/dev/null")
 

contrib/crypt/kdf_pbkdf2.py

@@ -54,7 +54,7 @@ salt_hex = "".join(["{:02x}".format(x) for x in salt])
 dk_hex = "".join(["{:02x}".format(x) for x in dk])
 
 print("KEY=" + dk_hex);
-print("algoritm=pbkdf2_hmac");
+print("algorithm=pbkdf2_hmac");
 print("hash_name=" + hash_name);
 print("salt=" + salt_hex);
 print("iterations=" + str(iterations));

contrib/migration/raw_suffix2sidecar

@@ -42,7 +42,7 @@ my $compress_format_alt = 'gz|bz2|xz|lzo|lz4';
 my $file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/;  # note: ubuntu uses '@' in the subvolume layout: <https://help.ubuntu.com/community/btrfs>
 my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/;
 my $timestamp_postfix_match = qr/\.(?<YYYY>[0-9]{4})(?<MM>[0-9]{2})(?<DD>[0-9]{2})(T(?<hh>[0-9]{2})(?<mm>[0-9]{2})((?<ss>[0-9]{2})(?<zz>(Z|[+-][0-9]{4})))?)?(_(?<NN>[0-9]+))?/;  # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]"
-my $raw_postfix_match = qr/--(?<received_uuid>$uuid_match)(\@(?<parent_uuid>$uuid_match))?\.btrfs?(\.(?<compress>($compress_format_alt)))?(\.(?<encrypt>gpg))?(\.(?<split>split_aa))?(\.(?<incomplete>part))?/;  # matches ".btrfs_<received_uuid>[@<parent_uuid>][.gz|bz2|xz][.gpg][.split_aa][.part]"
+my $raw_postfix_match = qr/--(?<received_uuid>$uuid_match)(\@(?<parent_uuid>$uuid_match))?\.btrfs?(\.(?<compress>($compress_format_alt)))?(\.(?<encrypt>gpg))?(\.(?<split>split_aa))?(\.(?<incomplete>part))?/;  # matches ".btrfs_<received_uuid>[@<parent_uuid>][.gz|.bz2|.xz|...][.gpg][.split_aa][.part]"
 
 my $dryrun;
 

contrib/tools/btrbk_restore_raw.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+
+import os
+import logging
+import subprocess
+import argparse
+
+
+logger = logging.getLogger(__name__)
+
+
+class TransformProcess:
+    def run(self, bfile, options, **kw):
+        return subprocess.Popen(self.get_cmd(bfile, options), **kw)
+
+    def get_cmd(self, bfile, options):
+        raise NotImplementedError()
+
+    @classmethod
+    def add_parser_options(cls, parser):
+        pass
+
+
+class TransformOpensslDecrypt(TransformProcess):
+    @staticmethod
+    def get_cmd(bfile, options):
+        return [
+            'openssl', 'enc', '-d', '-' + bfile.info['cipher'], '-K',
+            open(options.openssl_keyfile, 'r').read(), '-iv', bfile.info['iv']
+        ]
+
+    @staticmethod
+    def add_parser_options(parser):
+        parser.add_argument('--openssl-keyfile', help="path to private encryption key file")
+
+
+class TransformDecompress(TransformProcess):
+    def __init__(self, program):
+        self.p = program
+
+    def get_cmd(self, bfile, options):
+        return [self.p, '-d']
+
+
+class TransformBtrfsReceive(TransformProcess):
+    @classmethod
+    def run(cls, bfile, options, **kw):
+        return subprocess.Popen(cls.get_cmd(bfile, options), **kw)
+
+    @staticmethod
+    def get_cmd(bfile, options):
+        return ['btrfs', 'receive', options.restore_dir]
+
+
+TRANSFORMERS = (
+    TransformOpensslDecrypt, TransformDecompress, TransformBtrfsReceive
+)
+
+
+class BtrfsPipeline:
+    def __init__(self, bfile):
+        self.bfile = bfile
+        self.processors = []
+
+    def append(self, transformer):
+        self.processors.append(transformer)
+
+    def run(self, options):
+        processes = []
+        with open(self.bfile.data_file, 'rb') as next_input:
+            for transformer in self.processors:
+                process = transformer.run(
+                    self.bfile, options,
+                    stdin=next_input, stdout=subprocess.PIPE,
+                    stderr=subprocess.PIPE)
+                next_input = process.stdout
+                processes.append(process)
+            btrfs_process = TransformBtrfsReceive.run(
+                self.bfile, options, stdin=next_input,
+                stderr=subprocess.PIPE, stdout=subprocess.DEVNULL)
+            processes.append(btrfs_process)
+            # warning: the code below is pretty ugly and hacky
+            terminated = 0
+            while terminated < len(processes):
+                for p in processes:
+                    if p.returncode is not None:
+                        continue
+                    msg = None
+                    try:
+                        p.wait(timeout=1)
+                    except subprocess.TimeoutExpired as e:
+                        pass
+                    except Exception as e:
+                        msg = e
+                    else:
+                        msg = p.stderr.read().decode('utf-8').strip()
+                    finally:
+                        if p.returncode is not None:
+                            terminated += 1
+                            if p.returncode != 0:
+                                for p_other in processes:
+                                    p_other.terminate()
+                                    terminated += 1
+                                if msg:
+                                    logger.error(f"error running {p.args}: {msg}")
+
+    def get_cmd(self, options):
+        command_pipe = [['cat', self.bfile.data_file]]
+        for transformer in self.processors:
+            command_pipe.append(transformer.get_cmd(self.bfile, options))
+        command_pipe.append(TransformBtrfsReceive.get_cmd(self.bfile, options))
+        return ' | '.join(' '.join(x) for x in command_pipe)
+
+
+class BackupFile:
+    def __init__(self, path):
+        assert path.endswith('.info')
+        self.info_file = path
+        self.info = self._parse_info()
+        self.uuid = self.info['RECEIVED_UUID']
+        self.data_file = os.path.join(os.path.dirname(path), self.info['FILE'])
+        self.parent = self.info.get('RECEIVED_PARENT_UUID')
+        self.is_restored = False
+
+    def _parse_info(self):
+        config = {}
+        with open(self.info_file, 'r') as fh:
+            # skip command option line
+            for line in fh.readlines():
+                if '=' not in line:
+                    continue
+                key, val = line.strip().split('=', maxsplit=1)
+                config[key] = val
+        return config
+
+    def get_transformers(self):
+        if 'encrypt' in self.info:
+            if self.info['encrypt'] == 'gpg':
+                raise NotImplementedError('gpg encryption')
+            elif self.info['encrypt'] == 'openssl_enc':
+                yield TransformOpensslDecrypt()
+            else:
+                raise Exception(f'unknown encryption type: "{self.info["encrypt"]}"')
+        if 'compress' in self.info:
+            yield TransformDecompress(self.info['compress'])
+
+    def restore_file(self, options):
+        assert self.info.get('TYPE') == 'raw'
+        assert not self.info.get('INCOMPLETE')
+        pipeline = BtrfsPipeline(self)
+        for transformer in self.get_transformers():
+            pipeline.append(transformer)
+        if options.dry_run:
+            print(pipeline.get_cmd(options))
+        else:
+            logger.info(f"restoring backup {os.path.basename(self.data_file)}")
+            pipeline.run(options)
+        self.is_restored = True
+
+
+def restore_from_path(backup, options):
+    path = os.path.dirname(backup)
+    info_files = {}
+    backup_file = BackupFile(backup + '.info')
+    restored_files = set()
+    for entry in os.scandir(path):
+        if entry.is_file() and entry.name.endswith('.info'):
+            info = BackupFile(entry.path)
+            info_files[info.uuid] = info
+    restored_files.update(restore_backup(backup_file, info_files, options))
+    logger.info(f"finished; restored {len(restored_files)} backup files")
+
+
+def restore_backup(bfile, parents, options):
+    if bfile.is_restored:
+        return
+    if bfile.parent:
+        parent = parents.get(bfile.parent)
+        if not parent:
+            msg = (f"missing parent {bfile.parent} for"
+                   f"'{os.path.basename(bfile.info_file)}'")
+            if options.ignore_missing:
+                logger.warning(msg)
+            else:
+                raise Exception(msg)
+        else:
+            yield from restore_backup(parent, parents, options)
+    bfile.restore_file(options)
+    yield bfile.uuid
+
+
+def main():
+    parser = argparse.ArgumentParser(description="restore btrbk raw backup")
+    parser.add_argument('backup', help="backup file to restore; for incremental"
+                        " backups the parent files must be in the same directory")
+    parser.add_argument('restore_dir', help="target directory for restored subvolumes"
+                                            " (path argument for \"btrfs receive\")")
+    parser.add_argument('-n', '--dry-run', action='store_true',
+                        help="print commands that would be executed")
+    parser.add_argument('--ignore-missing', action='store_true',
+                        help="do not fail on missing parent snapshots")
+
+    for transformer in TRANSFORMERS:
+        transformer.add_parser_options(parser)
+
+    args = parser.parse_args()
+
+    if args.dry_run:
+        logger.setLevel('ERROR')
+
+    restore_from_path(args.backup, args)
+
+
+if __name__ == '__main__':
+    logger.setLevel('INFO')
+    logging.basicConfig(format='%(asctime)s %(levelname)s - %(message)s')
+    main()

doc/FAQ.md

@@ -28,7 +28,7 @@ How can I setup a debian pre-install hook?
 Create a file `/etc/apt/apt.conf.d/70btrbk`, e.g.:
 
     // create a btrfs snapshot before (un)installing packages
-    Dpkg::Pre-Invoke {"/usr/sbin/btrbk run /mnt/btr_pool/rootfs";};
+    Dpkg::Pre-Invoke {"/usr/bin/btrbk run /mnt/btr_pool/rootfs";};
 
 In order to make sure that the snapshots are always generated and
 nothing is deleted, add the btrbk command line options `--preserve
@@ -110,8 +110,8 @@ have to create a run-time (rw) snapshot before booting into it:
 How do I convert '/' (subvolid=5) into a subvolume?
 ---------------------------------------------------
 
-There's several ways to achieve this, the solution described below is
+There's several ways to achieve this, the solution described below
-that it guarantees not to create new files (extents) on disk.
+guarantees not to create new files (extents) on disk.
 
 ### Step 1: make a snapshot of your root filesystem
 
@@ -191,7 +191,7 @@ What is the most efficient way to clone btrfs storage?
 ------------------------------------------------------
 
 It is very common (and avisable!) to keep backups on a separate
-location. In some situations, is is also required to transport the
+location. In some situations, it is also required to transport the
 data physically, either to the datacenter or to your safe in the
 basement.
 

doc/Makefile

@@ -1,6 +1,6 @@
-DOCS       = FAQ.md \
+DOCS       = FAQ.md
-             upgrade_to_v0.23.0.md
 MAN_MAN1   = btrbk.1 \
+             lsbtr.1 \
              ssh_filter_btrbk.1
 MAN_MAN5   = btrbk.conf.5
 
@@ -17,8 +17,19 @@ ifeq ($(COMPRESS), yes)
 endif
 
 # convert using "asciidoctor": <https://asciidoctor.org>
-ASCIIDOCTOR_MANPAGE = asciidoctor -d manpage -b manpage
+# fallback to "a2x" from asciidoc package: <http://asciidoc.org>
-ASCIIDOCTOR_HTML    = asciidoctor -b html5 -d article
+ifneq (, $(shell command -v asciidoctor 2> /dev/null))
+  ASCIIDOC_MANPAGE = asciidoctor -d manpage -b manpage
+  ASCIIDOC_HTML    = asciidoctor -b html5 -d article
+else ifneq (, $(shell command -v a2x 2> /dev/null))
+  # NOTE: using -L (--no-xmllint), as xmllint is a separate package on many distros.
+  ASCIIDOC_MANPAGE = a2x -L -d manpage -f manpage
+  ASCIIDOC_HTML    = asciidoc -b html -d article
+else
+  ASCIIDOC_ERR = $(error "please install either asciidoc or asciidoctor")
+  ASCIIDOC_MANPAGE = $(ASCIIDOC_ERR)
+  ASCIIDOC_HTML    = $(ASCIIDOC_ERR)
+endif
 
 # reproducible builds: reference date is ":date:" attribute from asciidoc source
 date_attr = $(shell sed -rn 's/:date:\s*//p' $(1))
@@ -49,10 +60,10 @@ clean:
 	gzip -9 -n -c $< > $@
 
 %.1 : %.1.asciidoc
-	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_MANPAGE) -o $@ $<
+	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $<
 
 %.5 : %.5.asciidoc
-	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_MANPAGE) -o $@ $<
+	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $<
 
 %.html : %.asciidoc
-	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_HTML) -o $@ $<
+	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_HTML) -o $@ $<

doc/btrbk.1.asciidoc

@@ -1,7 +1,7 @@
 btrbk(1)
 ========
-:date: 2019-10-27
+:date: 2023-03-25
-:release-version: 0.29.0
+:release-version: 0.32.6
 :man manual: Btrbk Manual
 :man source: Btrbk {release-version}
 
@@ -20,11 +20,12 @@ btrbk [-h|--help] [--version]
       [-c|--config <file>] [-n|--dry-run] [--exclude <filter>]
       [-p|--preserve] [--preserve-snapshots] [--preserve-backups]
       [-v|--verbose] [-q|--quiet] [-l|--loglevel <level>]
-      [-t|--table] [--format <output-format>] [--pretty]
+      [-t|--table] [-L|--long] [-1|--single-column]
+      [--format <output-format>] [--pretty]
       [-S|--print-schedule] [--progress]
       [--lockfile <file>]
       [--override <config_option>=<value>]
-      <command> [<args>]
+      <command> [[--] <filter>...]
 
 
 DESCRIPTION
@@ -44,23 +45,27 @@ btrbk.conf(5) for more details.
 
 === Snapshots and Backups
 
-Snapshots as well as backup subvolumes are created in form:
+Snapshots as well as backup subvolumes are created in the form:
 
     <snapshot-name>.<timestamp>[_N]
 
 Where '<snapshot-name>' is identical to the source subvolume name,
-unless the configuration option 'snapshot_name' is set. The
+unless the configuration option 'snapshot_name' is set. '<timestamp>'
-<timestamp> is either "YYYYMMDD" or "YYYYMMDDThhmm" (dependent of the
+is a timestamp describing the creation time (local time of the host
-'timestamp_format' configuration option), where "YYYY" is the year,
+running btrbk) of the snapshot/backup. The format can be configured
-"MM" is the month, "DD" is the day, "hh" is the hour and "mm" is the
+using the 'timestamp_format' option, refer to btrbk.conf(5) for
-minute of the creation time (local time of the host running btrbk). If
+details. If multiple snapshots/backups are created on the same
-multiple snapshots/backups are created on the same date/time, N will
+date/time, 'N' will be incremented on each snapshot, starting at 1.
-be incremented on each snapshot, starting at 1.
 
-Note that 'snapshot' is a btrfs terminology for a ``read-only
+If a snapshot or backup does not match the naming scheme above
-subvolume'' (showing a parent-uuid, see btrfs-subvolume(8)); 'backup'
+(e.g. if it has been renamed manually), btrbk will leave it untouched.
-is a btrbk terminology for a ``read-only subvolume created with
+
-send/receive'' (showing a received-uuid).
+Note that in btrfs terminology, a 'snapshot' is a ``subvolume with
+a given initial content of the original subvolume'' (showing a
+parent-uuid, see btrfs-subvolume(8)), and they can be read-write
+(default) or read-only. In btrbk terminology, 'snapshot' means
+``read-only btrfs snapshot'', and 'backup' means ``read-only subvolume
+created with send/receive'' (showing a received-uuid).
 
 
 OPTIONS
@@ -83,8 +88,15 @@ OPTIONS
     commands that would be executed.
 
 --exclude <filter>::
-    Exclude configured sections matching '<filter>'. See
+    Exclude configured sections matching '<filter>' (see
-    <<_filter_statements,FILTER STATEMENTS>> below.
+    <<_filter_statements,FILTER STATEMENTS>> below), or any specific
+    snapshot from being backuped or deleted, or any specific backup
+    from being deleted.
++
+Note that excluding specific snapshots from being backuped has impact
+on scheduling: e.g. if the "first snapshot of the day" is excluded,
+the "second snapshot of the day" shifts to "first", creating a backup
+as "first backup of the day".
 
 -p, --preserve::
     Preserve all snapshots and backups. Skips deletion of any
@@ -100,13 +112,13 @@ OPTIONS
     specified in the configuration file.
 
 --wipe::
-    Ignore configured snapshot retention policy, delete all but latest
+    Ignore configured snapshot retention policy, delete all but the latest
     snapshots instead. All snapshots needed for incremental backup
     (latest common) are also preserved. Useful if you are getting low
     on disk space (ENOSPC).
 
 -v, --verbose::
-    Verbose output (shortcut for "--loglevel=info").
+    Increase the logging level, see "--loglevel".
 
 -q, --quiet::
     Quiet operation. If set, btrbk does not print the summary after
@@ -114,17 +126,26 @@ OPTIONS
     commands.
 
 -l, --loglevel <level>::
-    Set the level of verbosity. Accepted levels are warn, info, debug,
+    Set the level of verbosity for the stderr logging. Accepted levels
-    and trace.
+    are: error, warn, info, debug, and trace. Default is info.
 
 -t, --table::
     Print output in table format (shortcut for "--format=table").
 
---format table|long|raw::
+-L, --long::
+    Print output in long table format (shortcut for "--format=long").
+
+-1, --single-column::
+    Print output as single column (not available for all commands).
+
+--format table|long|raw|col:[h:]<columns>::
     Print output in specified format. If set to "raw", prints
-    space-separated key="value" pairs (machine readable). Affects
+    space-separated, quoted key=value pairs (machine readable).
-    output format for *run*, *snapshot*, *resume*, *prune*, *archive*
++
-    and *list* commands. Useful for further exporting/scripting.
+If set to "col:", prints only the <columns> specified (comma-separated
+list). Header lines are omitted if the "h:" modifier is present.
+Columns prefixed with "-" are collapsed if empty. Columns postfixed
+with ":RALIGN" are right-aligned.
 
 --pretty::
     Print table output with lowercase, underlined column headings
@@ -140,9 +161,9 @@ OPTIONS
     command (version >= 20180505) installed on the host running btrbk.
 
 --lockfile <file>::
-    Create lockfile <file> on startup; checks lockfile before running
+    Place an exclusive lock on <file> during program execution, using
-    any btrfs commands (using perl "flock"), and exits if the lock is
+    flock(2). If the lock is held by another process, exit before
-    held by another btrbk instance. Overrides configuration option
+    running any actions. Overrides configuration option
     "lockfile". Ignored on dryrun ('-n', '--dry-run').
 
 --override <config_option>=<value>::
@@ -201,8 +222,9 @@ configuring the retention policy.
     snapshots that are not preserved by their configured retention
     policy will be deleted. Note that the latest snapshot (the one
     created in step 1) as well as the latest snapshot/backup pair are
-    always preserved, regardless of the retention policy.
+    always preserved, regardless of the retention policy. If any
-
+    target is unreachable or has errors, all snapshots are preserved
+    in order not to break the incremental chain.
 
 *dryrun* [filter...]::
     Don't run any btrfs commands that would alter the filesystem, just
@@ -323,17 +345,17 @@ file system.
     form. Optionally filtered by [filter...] arguments (see
     <<_filter_statements,FILTER STATEMENTS>> below).
 +
-Available subcommands:
+Available subcommands (default ``all''):
 +
 --
 ifndef::backend-docbook,backend-manpage[]
 [horizontal]
 endif::backend-docbook,backend-manpage[]
 
-*snapshots*;; List all snapshots (and corresponding backups). Note
+*all*;;       List all snapshots and backups created by btrbk.
-              that all snapshots related to configured subvolumes are
+*snapshots*;; List all snapshots created by btrbk.
-              listed, not only the ones created by btrbk.
+*backups*;;   List all backups (and correlated snapshots) created by
-*backups*;;   List all backups (and corresponding snapshots).
+              btrbk.
 *latest*;;    List most recent common snapshot/backup pair, or most
               recent snapshot if no common found.
 *config*;;    List configured source/snapshot/target relations.
@@ -342,8 +364,8 @@ endif::backend-docbook,backend-manpage[]
 *target*;;    List configured targets.
 --
 +
-Use the '--format' command line option to switch between
+Use the '--format' command line option to switch between different
-different output formats.
+output formats.
 
 
 *usage* [filter...]::
@@ -360,16 +382,54 @@ different output formats.
     formats.
 
 *diff* <from> <to>::
-    Print new files since subvolume <from> for subvolume <to>.
+    List the modified files since generation (transid) of subvolume
+    <from> in subvolume <to>. Columns:
++
+------------
+SIZE   file was modified for a total of SIZE bytes
+COUNT  file was modified in COUNT generations
+FLAGS  "+"  file accessed at offset 0 (at least once)
+       "c"  COMPRESS flag is set (at least once)
+       "i"  INLINE flag is set (at least once)
+------------
 
-*ls* <path>::
+*extents* [diff] <subvolume>... [exclusive <subvolume>...]::
-    List all btrfs subvolumes below <path>. Use
+   Print accurate disk space usage and diff based on extent data
-    '--format=table|long|rel|raw' to switch between different output
+   (FIEMAP ioctl, slow!).
-    formats.
++
+--
+Subvolumes following the 'exclusive' keyword are added to a separate
+set, and additional set-exclusive data is printed at the end of the
+list. This gives a hint of how much data will be freed if deleting all
+subvolumes in the set. Example:
+
+    btrbk extents diff /backup/data.* exclusive /backup/data.2010*
+
+The EXCLUSIVE column shows the set-exclusive data of all other listed
+(!) subvolumes (relative complement of block regions). Provided that
+all related subvolumes (holding references to extents) are also
+listed, this amount of disk space would be freed when deleting the
+subvolume.
+
+The DIFF column shows the data added to the previous subvolume
+(relative complement of block regions).
+
+If called with the '--related' option, btrbk also lists all related
+subvolumes. This is not recommended for backups, as parent-uuid
+relations break for received subvolumes as soon as an intermediate
+subvolume is deleted.
+
+Note that reading all extents is a disk-intensive task, expect long
+execution times and high ram usage. Consider setting 'cache_dir'.
+--
+
+*ls* <path>|<url>...::
+    List all btrfs subvolumes below <path>. Use the '--format' command
+    line option to switch between different output formats. See
+    lsbtr(1).
 
 *config* print|print-all::
-    Prints the parsed configuration file. Use the '--format' command
+    Prints the parsed configuration file.
-    line option to switch between different output formats.
 
 
 FILTER STATEMENTS
@@ -420,6 +480,9 @@ Note that for *run* and *snapshot* commands, a filter matching a
 surrounding 'subvolume' section. If this is not desired, consider
 running *snapshot* and *resume* commands separately.
 
+Filter statements can match multiple times (e.g. on group as well as
+host name). In such a case, all matches are processed.
+
 
 FILES
 -----

doc/btrbk.conf.5.asciidoc

@@ -1,7 +1,7 @@
 btrbk.conf(5)
 =============
-:date: 2019-10-27
+:date: 2023-03-25
-:release-version: 0.29.0
+:release-version: 0.32.6
 :man manual: Btrbk Manual
 :man source: Btrbk {release-version}
 
@@ -36,34 +36,21 @@ global options must be set before any sections are defined.
 Blank lines are ignored. A hash character (#) starts a comment
 extending until end of line.
 
-Whitespace or unicode characters are not allowed for file
-names. Allowed characters are:
-
-    [0-9] [a-z] [A-Z] and "._+-@"
-
-This is for sanity/safety/security reasons, we apologize for the
-inconvenience.
-
 
 SECTIONS
 --------
 
-*volume* <volume-directory>|<url>::
+*volume* <volume-directory>|<url> (optional)::
-    Directory of a btrfs volume containing the source subvolume(s) to
+    Absolute path pointing to a btrfs file system containing the
-    be backed up. '<volume-directory>' must be an absolute path and
+    source subvolume(s) to be backed up. Usually the mount point of a
-    point to a btrfs volume (or subvolume). Usually the mount point of
+    btrfs filesystem mounted with the 'subvolid=5' option.
-    a btrfs filesystem mounted with the 'subvolid=5' option.
 
 *subvolume* <subvolume-name>::
-    Subvolume to be backed up, relative to the '<volume-directory>'
+    Subvolume to be backed up, relative to the '<volume-directory>' of
-    specified in the 'volume' section. Multiple 'subvolume' sections
+    the 'volume' section, or absolute if the 'volume' section is
-    are allowed within 'volume' sections. Accepts wildcard character
+    omitted. Accepts wildcard character "*".
-    "*".
 +
 --
-If set to ".", the subvolume at '<volume-directory>' is used as backup
-source, and the snapshots will be created within the source subvolume
-itself (see 'snapshot_dir' option below), which is not recommended.
 Note that if this subvolume is btrfs root (id=5), it needs to have a
 valid UUID, which is not the case for file systems created with
 btrfs-progs < 4.16.
@@ -88,6 +75,10 @@ below. Accepted formats are:
   ssh://<hostname>[:<port>]/<directory>
   <hostname>:<directory>
 
+Where '<hostname>' is either a host name, an IPv4 address in
+dotted-decimal form, or an IP literal encapsulated within square
+brackets (e.g. "[2001:db8::7]").
+
 If you are connecting to virtual machines, consider configuring
 several 'volume' sections for a '<hostname>', with distinct '<port>'
 numbers for each machine.
@@ -105,7 +96,7 @@ otherwise.
 
 *timestamp_format* short|long|long-iso::
     Timestamp format used as postfix for new snapshot subvolume
-    names. Defaults to ``short''.
+    names. Defaults to ``long''.
 +
 --
 ifndef::backend-docbook,backend-manpage[]
@@ -125,12 +116,13 @@ snapshots are created during a daylight saving time clock
 change).
 +
 Note that using ``long-iso'' has implications on the scheduling, see
-<<_retention_policy,RETENTION POLICY>> (caveats) below.
+<<_reference_time,Reference Time>> below.
 
 *snapshot_dir* <directory>::
     Directory in which the btrfs snapshots are created, relative to
-    '<volume-directory>' of the 'volume' section. Note that btrbk does
+    '<volume-directory>' of the 'volume' section, or absolute if the
-    not autmatically create this directory, and the snapshot creation
+    'volume' section is omitted. Note that btrbk does not
+    automatically create this directory, and the snapshot creation
     will fail if it is not present.
 
 *snapshot_name* <basename>::
@@ -140,15 +132,16 @@ Note that using ``long-iso'' has implications on the scheduling, see
 
 *snapshot_create* always|onchange|ondemand|no::
     If set to ``always'', snapshots are always created. If set to
-    ``onchange'', snapshots are only created if the source subvolume
+    ``onchange'', snapshots are only created if the last snapshot is
-    has changed since the last snapshot (more precisely: if the btrfs
+    not up-to-date, i.e. the source subvolume has changed (more
-    generation has been increased since the last snapshot). If set to
+    precisely: the btrfs generation has been increased) since the last
-    ``ondemand'', snapshots are only created if at least one target
+    snapshot was created. If set to ``ondemand'', snapshots are only
-    subvolume is reachable (useful if you are tight on disk space and
+    created if at least one target subvolume is reachable (useful if
-    you only need btrbk for backups to an external disk which is not
+    you are tight on disk space and you only need btrbk for backups to
-    always connected). If set to ``no'', the snapshots are never
+    an external disk which is not always connected). If set to ``no'',
-    created (useful if another instance of btrbk is taking care of
+    the snapshots are never created (useful if another instance of
-    snapshot creation). Defaults to ``always''.
+    btrbk is taking care of snapshot creation). Defaults to
+    ``always''.
 
 *incremental* yes|no|strict::
     If set, incremental backups are created. If set to ``strict'',
@@ -167,7 +160,7 @@ not be certain about this, such operations are disallowed in
 
 *noauto* yes|no::
     If set, the context is skipped by all btrbk actions unless
-    explicitely enabled by a matching btrbk '<filter>' command line
+    explicitly enabled by a matching btrbk '<filter>' command line
     argument (e.g. "btrbk run myfilter").
 
 
@@ -176,8 +169,8 @@ not be certain about this, such operations are disallowed in
 *group* <group-name> [<group-name>]...::
     Add the current section (volume, subvolume or target) to
     user-defined groups, which can be used as filter for most btrbk
-    commands. This option can be set multiple times within the same
+    commands (see btrbk(1) FILTER STATEMENTS). This option can be set
-    context.
+    multiple times within the same context.
 
 
 === Retention Policy Options
@@ -192,16 +185,20 @@ not be certain about this, such operations are disallowed in
     Defines after what time (in full hours since midnight) a
     snapshot/backup is considered to be a "daily" backup. Daily,
     weekly, monthly and yearly backups are preserved on this hour (see
-    <<_retention_policy,RETENTION POLICY>> below). If you set this
+    <<_retention_policy,RETENTION POLICY>> below). Ignored on
-    option, make sure to also set 'timestamp_format' to ``long'' or
+    snapshots or backups without time information ('timestamp_format
-    ``long-iso'' (backups and snapshots having no time information
+    short'). Defaults to ``0''.
-    will ignore this option). Defaults to ``0''.
 
 *snapshot_preserve* no|<retention_policy>::
     Set retention policy for snapshots (see
     <<_retention_policy,RETENTION POLICY>> below). If set to ``no'',
     preserve snapshots according to 'snapshot_preserve_min'
     only. Defaults to ``no''.
++
+--
+Note that 'snapshot_preserve' has no effect if 'snapshot_preserve_min'
+is set to ``all'' (the default).
+--
 
 *snapshot_preserve_min* all|latest|<number>{h,d,w,m,y}::
     Preserve all snapshots for a minimum amount of hours (h), days
@@ -214,6 +211,11 @@ not be certain about this, such operations are disallowed in
     <<_retention_policy,RETENTION POLICY>> below). If set to ``no'',
     preserve backups according to 'target_preserve_min' only. Defaults
     to ``no''.
++
+--
+Note that 'target_preserve' has no effect if 'target_preserve_min' is
+set to ``all'' (the default).
+--
 
 *target_preserve_min*  all|latest|no|<number>{h,d,w,m,y}::
     Preserve all backups for a minimum amount of hours (h), days (d),
@@ -240,31 +242,29 @@ not be certain about this, such operations are disallowed in
 
 === SSH Options
 
-*ssh_identity* <file>::
+*ssh_identity* <file>|no::
-    Absolute path to a ssh identity file (private key). Note that if
+    Absolute path to a ssh identity file (private key). If not set,
-    the private key is password protected, btrbk will prompt for user
+    the ssh default is used (see ssh(1), "-i identity_file"). Note
-    input, which is usually not desired.
+    that if the identity key is password protected and no
+    authentication agent is used, btrbk will prompt for user input on
+    every connection attempt.
 
-*ssh_user* <username>::
+*ssh_user* <username>|no::
     Remote username for ssh. Defaults to ``root''. Make sure the
     remote user is able to run "btrfs" with root privileges (see
-    option 'backend' for details).
+    option 'backend' for details). If set to ``no'', the ssh default
+    is used.
 
 *ssh_compression* yes|no::
     Enables or disables the compression of ssh connections. Defaults
-    to ``no''.
+    to ``no''. Note that if *stream_compress* is enabled, ssh
+    compression will always be disabled for send/receive operations.
 
-*ssh_cipher_spec* <cipher_spec>::
+*ssh_cipher_spec* default|<cipher_spec>::
     Selects the cipher specification for encrypting the session
     (comma-separated list of ciphers in order of preference). See the
     "-c cipher_spec" option in ssh(1) for more information. Defaults
-    to ``default'' (the ciphers specified in 'ssh_config').
+    to ``default'' (the ciphers specified in ssh_config(5)).
-
-Previous versions btrbk allowed you to set a *ssh_port* option, this
-has been dropped in favor of the `ssh://hostname:port` notation in the
-'volume' and 'target' <<_sections,sections>>. If you want to set a
-global port for all SSH connections to remote hosts, set the ``Port''
-option in ssh_config(5).
 
 
 === Data Stream Options
@@ -274,7 +274,7 @@ option in ssh_config(5).
     remote locations. Defaults to ``no''. If enabled, make sure that
     '<compress_command>' is available on the source and target
     hosts. Supported '<compress_command>': gzip, pigz, bzip2, pbzip2,
-    xz, lzo, lz4.
+    bzip3, xz, lzo, lz4, zstd.
 
 *stream_compress_level* default|<number>::
     Compression level for the specified '<compress_command>'. Refer to
@@ -282,9 +282,18 @@ option in ssh_config(5).
     fastest compression). Defaults to ``default'' (the default
     compression level of '<compress_command>').
 
+*stream_compress_long* default|<number>::
+    Enable long distance matching for the specified
+    '<compress_command>'. Refer to the related man-page for details.
+    Only supported for "zstd".
+
 *stream_compress_threads* default|<number>::
     Number of threads to use for <compress_command>. Only supported
-    for "pigz", "pbzip2" and recent versions of "xz".
+    for "pigz", "pbzip2", "bzip3", "zstd" and recent versions of "xz".
+
+*stream_compress_adapt* yes|no::
+    Enable adaptive compression for <compress_command>. Only supported
+    for "zstd" (version >= 1.3.6). Defaults to ``no''.
 
 *stream_buffer* <size>|no::
     Add a buffer to the btrfs send stream (locally, on uncompressed
@@ -314,13 +323,20 @@ additional, multi-threaded process to the command pipe.
 *stream_buffer_remote* <size>|no::
     Add a buffer on remote hosts (either source or target). Defaults
     to ``no''.
++
+--
+Enable this if you prefer buffering on the remote side, or even on
+both sides: reasons for this depend on available memory, disk and cpu
+performance (btrfs send/receive, compression), as well as networking
+constraints.
+--
 
 *rate_limit* <rate>|no::
     Limit the read rate of the btrfs send stream to '<rate>' bytes per
     second (locally, on uncompressed send stream). A suffix of "k",
     "m", "g", or "t" can be added to denote kilobytes (*1024),
     megabytes, and so on. Defaults to ``no''. Note that 'rate_limit'
-    implicitely adds a stream buffer (see 'stream_buffer' option
+    implicitly adds a stream buffer (see 'stream_buffer' option
     above).
 
 *rate_limit_remote* <rate>|no::
@@ -343,64 +359,142 @@ additional, multi-threaded process to the command pipe.
     daemon, auth, lpr, news, cron, authpriv, local0..local7.
 
 *lockfile* <file>|no::
-    Create lockfile <file> on startup; checks lockfile before running
+    Place an exclusive lock on <file> during program execution, using
-    any btrfs commands (using perl "flock"), and exits if the lock is
+    flock(2). If the lock is held by another process, exit before
-    held by another btrbk instance. Ignored on dryrun ('-n',
+    running any actions. Ignored on dryrun ('-n', '--dry-run'). See
-    '--dry-run'). See also '--lockfile' command-line option.
+    also '--lockfile' command-line option.
 
-*backend* btrfs-progs|btrfs-progs-btrbk|btrfs-progs-sudo::
+*backend* <backend>::
     Backend filesystem utilities to be used for btrfs specific
-    operations. Defaults to ``btrfs-progs''.
+    operations. Available backends:
 +
 --
-btrfs-progs::
+*btrfs-progs*::
     Default backend, btrfs commands are called as specified in
     btrfs(8) (e.g. "btrfs subvolume show").
 
-btrfs-progs-btrbk::
+*btrfs-progs-btrbk*::
     btrfs commands are separated by a dash instead of a whitespace
     (e.g. "btrfs-subvolume-show" instead of "btrfs subvolume
     show"). Useful for setting suid or file capabilities (setcap) on
     specific btrfs commands, as implemented in
     <https://github.com/digint/btrfs-progs-btrbk>.
 
-btrfs-progs-sudo::
+*btrfs-progs-sudo*::
     btrfs commands are prefixed with "sudo -n" (e.g. "sudo -n btrfs
     subvolume show" instead of "btrfs subvolume show"). Make sure to
     have appropriate (root) permissions for the "btrfs" command groups
-    and the "readlink" command in /etc/sudoers.
+    as well as the "readlink" and "test" commands in /etc/sudoers.
---
+
-+
+*btrfs-progs-doas*::
-For convenience, it is also possible to set *backend_local* or
+    Similar to btrfs-progs-sudo, using prefix "doas -n".
-*backend_remote* options, which will override the backend only for
+
-local or remote sources/targets (e.g. "backend_remote
+If you want to set this option for local or remote hosts only, set
+*backend_local* or *backend_remote* (e.g. "backend_remote
 btrfs-progs-btrbk").
 
+If you want to set this option for regular (non-root) user only, set
+*backend_local_user*.
+--
+
+*compat* <compat-option>...::
+    Enable compatibility options. Available 'compat-option':
++
+--
+*busybox*::
+    Use busybox compatible commands, at the expense of slight overhead
+    while reading filesystem information.
+
+*ignore_receive_errors*  _*experimental*_::
+    Tell btrfs-receive(8) to not terminate on errors by setting
+    "--max-errors=0" option. Print warnings instead.
++
+A known use case for this are target hosts lacking xattr support
+(e.g. some Synology NAS), while the send-stream contains "lsetxattr"
+commands. Another case is targets failing to set otime, complaining
+with "ERROR: attribute 12 requested but not present".
++
+Note that there is *no guarantee that backups created with this
+option enabled can be restored at all*.
+
+If you want to set this option for local or remote hosts only, set
+*compat_local* or *compat_remote* (e.g. "compat_remote busybox").
+--
+
+*cache_dir* <directory>::
+    If set, cache extent maps for the "btrbk extents" command.
+
 
 === Btrfs Specific Options
 
-*btrfs_commit_delete* after|each|no::
+*incremental_prefs* <list-spec>[:<amount>]...::
-    If set, make sure the deletion of snapshot and backup subvolumes
+    Specify the preferences to determine the best common (correlated)
-    are committed to disk when btrbk terminates. Defaults to ``no''.
+    parent and clone sources for incremental backups, by choosing from
+    predefined candidate lists.
++
+--
+The 'list-spec' defines from what candidate list the next
+parent/clone-src should be appended to a result list; 'amount' defines
+how many (e.g. "sro:1 sro:1" is identical to "sro:2"), or all if
+omitted. Any candidate which is already in the results is dropped.
 
-*incremental_clones* <number>::
+The resulting list of subvolumes is then used as parameters for the
-    Maximum number of clone sources allowed for incremental send. If
+btrfs-send(8) command: the first for "-p <parent>", all others for
-    set, btrbk adds "-c <clone-src>" to the btrfs-send(8) command for
+"-c <clone-src>".
-    all present snapshot/backup pairs (correlated subvolumes matching
+
-    matching 'received_uuid', printed by "btrbk stats"). Set this to a
+Available 'list-spec' (candidate lists = filtered subsets of
-    high number if you want to make sure that no common data is missed
+correlated subvolumes):
-    on incremental backups, in expense of btrfs-send
+
-    performance. Defaults to 0.
+*sro*,*srn*:: All from 'snapshot_dir' matching 'snapshot_name', with
+ parent_uuid relationship, sorted by btrbk timestamp (o=older
+ n=newer).
+
+*sao*,*san*:: All from 'snapshot_dir' matching 'snapshot_name', sorted
+ by btrbk timestamp (o=older n=newer).
+
+*aro*,*arn*:: All from 'incremental_resolve', with parent_uuid
+ relationship, sorted by cgen (o=older n=newer).
+
+Defaults to "sro:1 srn:1 sao:1 san:1 aro:1 arn:1". Note that for
+most operations the default resolves a single parent, as there usually
+are no newer snapshots, and all "sro:1 sao:1 aro:1" resolve to the
+same snapshot.
+
+Example: "defaults,sao,san,aro,arn" takes the defaults, and adds clone
+sources for all (!) known candidates on the filesystem.
+--
+
+*incremental_clones* yes|no::
+    If enabled, btrbk adds "-c <clone-src>" to the btrfs-send(8)
+    command for all correlated subvolumes resolved by
+    'incremental_prefs'. If disabled, only "-p <parent>" is
+    used. Defaults to ``yes''.
 
 *incremental_resolve* mountpoint|directory::
     Specifies where to search for the best common parent for
     incremental backups. If set to ``mountpoint'', use parents in the
-    filesystem tree below mount points of source
+    filesystem tree below the mount point of the snapshot and target
-    "<volume-directory>/<snapshot-dir>" and target
+    directory. If set to ``directory'', use parents strictly below
-    "<target-directory>". If set to ``directory'', use parents
+    snapshot/target directories. Set this to ``directory'' if you get
-    strictly below source/target directories. Set this to
+    access problems (when not running btrbk as root). Defaults to
-    ``directory'' if you get access problems (when not running btrbk
+    ``mountpoint''.
-    as root). Defaults to ``mountpoint''.
+
+*btrfs_commit_delete* yes|no::
+    If set, wait for the transaction commit at the end of each
+    snapshot or backup deletion (sets '--commit-each' option for
+    "btrfs subvolume delete"). Defaults to ``no''.
+
+*send_protocol* <number>|no  _*experimental*_::
+    Use btrfs send protocol version N. If enabled on 'target', btrbk
+    adds "--proto <number>" to the btrfs-send(8) command. Defaults to
+    ``no'' (btrfs default).
+
+*send_compressed_data* yes|no  _*experimental*_::
+    Send data that is compressed on the filesystem directly without
+    decompressing it. This requires protocol version 2 or higher
+    (btrfs-progs >= 5.19), and implies "send_protocol 2". If enabled
+    on 'target', btrbk adds "--compressed-data" to the btrfs-send(8)
+    command. Defaults to ``no'' (btrfs default).
 
 *snapshot_qgroup_destroy* yes|no  _*experimental*_:: {blank}
 *target_qgroup_destroy* yes|no  _*experimental*_:: {blank}
@@ -411,30 +505,39 @@ btrfs-progs-btrbk").
     <https://bugzilla.kernel.org/show_bug.cgi?id=91751>
 
 
+=== Informative Options
+
+*warn_unknown_targets* yes|no::
+    If set, prints a warning if btrbk encounters a target subvolume at
+    a unknown location (i.e. not following btrbk naming scheme, or
+    outside the target directory). Defaults to ``no''.
+
+
 RETENTION POLICY
 ----------------
 
-btrbk uses separate retention policies for snapshots and backups,
+Retention policies are defined individually for snapshots, backups and
-which are defined by the 'snapshot_preserve_min', 'snapshot_preserve',
+archives (summarized as "backups" in the following text), using a
-'target_preserve_min', 'target_preserve', 'preserve_day_of_week' and
+combination of:
-'preserve_hour_of_day' configuration options.
 
-Within this section, any statement about "backups" is always valid for
+**_preserve_min* all|latest|no|<number>{h,d,w,m,y}::
-backups as well as snapshots, referring to 'target_preserve' or
+    Amount of time (duration) in which all backups are preserved.
-'snapshot_preserve' respectively.
+
+**_preserve* no|<retention_policy>::
+    Schedule (single points in time) for which individual backups are
+    preserved.
+
+Note that if "preserve_min" is set to ``all'' (the default), any
+setting of "preserve" obviously has no effect.
 
 The format for '<retention_policy>' is:
 
     [<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]
 
-With the following semantics:
-
 *hourly*::
     Defines how many hours back hourly backups should be
     preserved. The first backup of an hour is considered an hourly
-    backup. Note that if you use <hourly> scheduling, make sure to
+    backup.
-    also set 'timestamp_format' to ``long'' or ``long-iso'', or the
-    scheduler will interpret the time as "00:00" (midnight).
 
 *daily*::
     Defines how many days back daily backups should be preserved. The
@@ -461,29 +564,31 @@ Use an asterisk for ``all'' (e.g. "target_preserve 60d *m"
 states: "preserve daily backups for 60 days back, and all monthly
 backups").
 
-The reference time (which defines the beginning of a day, week, month
-or year) for all date/time calculations is the local time of the host
-running btrbk.
-
 Hint: Run btrbk with the '-S', '--print-schedule' option to get a
 comprehensive output of the scheduler results.
 
-Caveats:
 
-* If you run a setup with several btrbk instances (e.g. one
+=== Reference Time
-  snapshot-only instance per remote client, and a separate fetch-only
-  instance on the backup server), it makes perfectly sense to run
-  btrbk with different local time on the clients, in order to make
-  sure the backups from all the remote hosts are preserved for
-  "midnight", and not at "00:00 UTC" (which would be "14:00" in
-  Honolulu). If you want this behaviour, do NOT use "timestamp_format
-  long-iso".
 
-* If "timestamp_format long-iso" is set, running btrbk from different
+The local time on the host running btrbk defines the reference time
-  time zones leads to different interpretation of "first in day, week,
+for all date/time calculations, especially for "beginning of a day",
-  month, or year". Make sure to run btrbk with the same time zone on
+and as a consequence for the first daily, weekly, monthly or yearly
-  every host, e.g. by setting the TZ environment variable (see
+backups. The local time on remote hosts (ssh source/target) is never
-  tzset(3)).
+used.
+
+Unless "timestamp_format long-iso" is set, daily backups are preserved
+at "preserve_hour_of_day" (defaults to midnight) of the respective
+time zone (and not for "00:00 UTC", which would be "14:00" in
+Honolulu). This becomes relevant for setups with multiple btrbk
+instances, e.g. many snapshot-only instances (spread around the
+world), and a fetch-only instance on the backup server.
+
+Caveat:
+
+* If "timestamp_format long-iso" is set, each btrbk instance on has a
+  different interpretation of "first in day". Make sure to run btrbk
+  with the same time zone on every host, e.g. by setting the TZ
+  environment variable (see tzset(3)).
 
 
 TARGET TYPES
@@ -492,24 +597,19 @@ TARGET TYPES
 *send-receive*::
     Backup to a btrfs filesystem, using "btrfs send/receive". This is
     the recommended (standard) target type. The '<target-directory>'
-    must be an absolute path and point to a btrfs volume (or
+    must be an absolute path and point to a subvolume or directory
-    subvolume), or to a directory within a subvolume. See
+    within a btrfs file system. See btrfs-send(8), btrfs-receive(8).
-    btrfs-send(8), btrfs-receive(8).
 
 *raw*  _*experimental*_::
     Backup to a raw (filesystem independent) file from the output of
     btrfs-send(8), with optional compression and encryption.
 +
 --
-Note that the target preserve mechanism is currently disabled for
-incremental raw backups (btrbk does not delete any incremental raw
-files)!
-
 Raw backups consist of two files: the main data file containing the
 btrfs send stream, and a sidecar file ".info" containing metadata:
 
-  <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|.xz][.gpg]
+  <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg]
-  <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|.xz][.gpg].info
+  <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg].info
 
 For 'incremental' backups ("incremental yes"), please note that:
 
@@ -520,6 +620,11 @@ For 'incremental' backups ("incremental yes"), please note that:
   make sure that a non-incremental backup is triggered from time to
   time.
 
+* The scheduler will never delete dependent parents of backups
+  preserved by the retention policy (run btrbk with the '-S',
+  '--print-schedule' option to get a comprehensive output of the
+  scheduler results).
+
 * There is currently no support for rotation of incremental backups:
   if 'incremental' is set, a full backup must be triggered manually
   from time to time in order to be able to delete old backups.
@@ -529,9 +634,13 @@ Additional options for raw targets:
 
 *raw_target_compress* <compress_command>|no::
     Compression algorithm to use for raw backup target. Supported
-    '<compress_command>': gzip, pigz, bzip2, pbzip2, xz, lzo, lz4.
+    '<compress_command>': gzip, pigz, bzip2, pbzip2, bzip3, xz, lzo,
+    lz4, zstd.
 *raw_target_compress_level* default|<number>::
     Compression level for the specified <compress_command>.
+*raw_target_compress_long* default|<number>::
+    Enable long distance matching for the specified
+    '<compress_command>'.
 *raw_target_compress_threads* default|<number>::
     Number of threads to use for <compress_command>.
 *raw_target_split* <size>|no::
@@ -547,7 +656,7 @@ Additional options for "raw_target_encrypt gpg":
 
 *gpg_keyring* <file>::
     Keyring to use for gpg, e.g. "`/etc/btrbk/gpg/pubring.kbx`".
-*gpg_recipient* <name>::
+*gpg_recipient* <name>...::
     Encrypt for user id '<name>' (email address).
 
 

doc/install.md

@@ -9,28 +9,30 @@ file, choose one of the following methods:
 
 ### Generic Linux System
 
+Install [asciidoctor] or [asciidoc] if you want to build the
+documentation.
+
 Download and unpack the latest [btrbk source tarball] and type:
 
     sudo make install
 
+#### Try latest master from Github:
 
-### Gentoo Linux
+    wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
-
+    chmod +x btrbk
-btrbk is in portage:
+    sudo ./btrbk ls /
-
-    emerge app-backup/btrbk
 
 
 ### Debian Based Distros
 
-btrbk is in `stretch (testing) (utils)`: https://packages.debian.org/stretch/btrbk
+btrbk is in debian stable (utils): https://packages.debian.org/stable/utils/btrbk
 
 Packages are also available via NeuroDebian: http://neuro.debian.net/pkgs/btrbk.html
 
 
 ### Fedora Linux
 
-btrbk is in the official Fedora repos: https://apps.fedoraproject.org/packages/btrbk
+btrbk is in the official Fedora repos: https://src.fedoraproject.org/rpms/btrbk
 
     sudo dnf install btrbk
 
@@ -42,10 +44,18 @@ btrbk is in AUR: https://aur.archlinux.org/packages/btrbk/
 
 ### Alpine Linux
 
-btrbk is in `testing`, install with:
+btrbk is in the community repository
 
     apk add btrbk
 
+
+### Gentoo Linux
+
+btrbk is in portage:
+
+    emerge app-backup/btrbk
+
+
 ### Void Linux
 
 btrbk is in Void's `current` repository
@@ -54,3 +64,5 @@ btrbk is in Void's `current` repository
 
 
   [btrbk source tarball]: https://digint.ch/download/btrbk/releases/
+  [asciidoctor]: https://asciidoctor.org
+  [asciidoc]: https://asciidoc.org

doc/lsbtr.1.asciidoc

@@ -0,0 +1,126 @@
+lsbtr(1)
+========
+:date: 2023-03-25
+:release-version: 0.32.6
+:man manual: Btrbk Manual
+:man source: Btrbk {release-version}
+
+
+NAME
+----
+
+lsbtr - list btrfs subvolumes
+
+
+SYNOPSIS
+--------
+
+[verse]
+lsbtr [-h|--help] [--version]
+      [-l|--long] [-u|--uuid] [-1|--single-column] [--raw]
+      [--format <output-format>]
+      [-v|--verbose]
+      [-c|--config <file>]
+      [--override <config_option>=<value>]
+      [[--] <path>|<url>...]
+
+
+DESCRIPTION
+-----------
+
+List btrfs subvolumes and their mount points visible by the file
+system below '<path>'.
+
+*lsbtr* is part of *btrbk* (basically a shortcut for "btrbk ls"), and
+takes some global configuration options from btrbk.conf(5) if present.
+
+*lsbtr* requires 'root privileges' to run correctly. Alternatively,
+consider using "btrfs-progs-sudo" or "btrfs-progs-btrbk" backends,
+both of which allows you to run lsbtr as a regular user. Refer to
+configuration option 'backend' in btrbk.conf(5) for more details.
+
+
+OPTIONS
+-------
+
+-h, --help::
+    Prints the synopsis and a list of the commands.
+
+--version::
+    Prints the btrbk version.
+
+-l, --long::
+    Print output in long table format (additionally print subvolume
+    path).
+
+-u, --uuid::
+    Print UUID table (parent/received relations).
+
+-1, --single-column::
+    Print path column only (delimited by newline).
+
+--raw::
+    Print space-separated key="value" pairs (machine readable).
+
+--format table|long|raw|col:[h:]<columns>::
+    Print output in specified format. If set to "raw", prints
+    space-separated key="value" pairs (machine readable).
++
+If set to "col:", prints only the <columns> specified (comma-separated
+list). Header lines are omitted if the "h:" modifier is present.
+Columns prefixed with "-" are collapsed if empty. Columns postfixed
+with ":RALIGN" are right-aligned.
+
+-v, --verbose::
+    Increase the level of verbosity.
+
+-c, --config <file>::
+    Read the configuration from <file>.
+
+--override <config_option>=<value>::
+    Override a configuration option <config_option> with <value>.
+
+
+FILES
+-----
+
++/etc/btrbk.conf+::
++/etc/btrbk/btrbk.conf+::
+    Default configuration file. The file format and configuration
+    options are described in *btrbk.conf*(5).
+
+
+EXIT STATUS
+-----------
+
+*lsbtr* returns the following error codes:
+
+ifndef::backend-docbook,backend-manpage[]
+[horizontal]
+endif::backend-docbook,backend-manpage[]
+0::   No problems occurred.
+1::   Generic error code.
+2::   Parse error: when parsing command-line options or configuration
+      file.
+255:: Script error.
+
+
+AVAILABILITY
+------------
+
+Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
+for further details.
+
+
+SEE ALSO
+--------
+
+*btrbk*(1),
+*btrbk.conf*(5),
+*btrfs*(8)
+
+
+AUTHOR
+------
+
+Axel Burri <axel@tty0.ch>

doc/ssh_filter_btrbk.1.asciidoc

@@ -1,7 +1,7 @@
 ssh_filter_btrbk(1)
 ===================
-:date: 2019-10-27
+:date: 2023-03-25
-:release-version: 0.29.0
+:release-version: 0.32.6
 :man manual: Btrbk Manual
 :man source: Btrbk {release-version}
 
@@ -34,18 +34,18 @@ The accepted commands are specified by the "--source", "--target",
 
 The following commands are always allowed:
 
- - "btrfs subvolume show"
+ - "btrfs subvolume show" (not affected by "--restrict-path")
- - "btrfs subvolume list"
+ - "btrfs subvolume list" (not affected by "--restrict-path")
  - "readlink"
+ - "test -d" (only if "compat busybox" configuration option is set)
  - "cat /proc/self/mountinfo"
- - pipes through "gzip", "pigz", "bzip2", "pbzip2", "xz", "lzop",
+ - pipes through "gzip", "pigz", "bzip2", "pbzip2", "bzip3", "xz",
-   "lz4" (stream_compress)
+   "lzop", "lz4", "zstd" (stream_compress)
- - pipes through "mbuffer" (stream_buffer)
+ - pipes through "mbuffer" (stream_buffer, rate_limit)
- - pipes through "pv -L" (rate_limit)
 
 Example line in /root/.ssh/authorized_keys on a backup target host:
 
-    command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup" ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@mydomain.com
+    command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup",restrict ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@example.org
 
 
 OPTIONS
@@ -80,7 +80,13 @@ OPTIONS
     Allow btrfs receive command: "btrfs receive".
 
 -p, --restrict-path <path>::
-    Restrict btrfs commands to <path>.
+    Restrict commands to <path>. Note that "btrfs subvolume show",
+    "btrfs subvolume list" are NOT affected by this option.
++
+It is not possible to restrict commands to exact subvolume names, as
+btrfs-receive(8) takes a <path> as argument (directory, not including
+the subvolume file name to be created, this is encoded in the
+send-stream).
 
 -l, --log::
     Log ACCEPT and REJECT messages to the system log.

doc/upgrade_to_v0.23.0.md

@@ -1,92 +0,0 @@
-Upgrading to btrbk-v0.23.0
-==========================
-
-In order to keep btrbk simple and intuitive while adding new features,
-it became inevitable to change the semantics of the "retention policy"
-related configuration options.
-
-
-What has changed?
------------------
-
-### Preserve *first* instead of *last* snapshot/backup
-
-btrbk used to *always* transfer the latest snapshot to the target
-location, while considering the *last* snapshot/backup of a day as a
-daily backup (and also the last weekly as a monthly). This made it
-very cumbersome when running btrbk in a cron job as well as manually,
-because the last manually created snapshot was immediately transferred
-on every run, and used as the daily backup (instead of the one created
-periodically by the cron job).
-
-The new semantics are to consider the *first* (instead of *last*)
-snapshot of a hour/day/week/month as the one to be preserved, while
-only transferring the snapshots needed to satisfy the target retention
-policy.
-
-
-### Preserve snapshots for a minimum amount of time
-
-In order to specify a minimum amount of time in which *all* snapshots
-should be preserved, the new "snapshot_preserve_min" and
-"target_preserve_min" configuration options were introduced. This was
-previously covered by "snapshot_preserve_daily", which caused a lot of
-confusion among users.
-
-
-Upgrading the configuration file: /etc/btrbk/btrbk.conf
--------------------------------------------------------
-
-Please read the description of the "run" command in [btrbk(1)], as
-well as the "RETENTION POLICY" section in [btrbk.conf(5)] for a
-detailed description. Make sure to understand the new concept, and run
-`btrbk --print-schedule dryrun` after updating the configuration.
-
-
-### Upgrade retention policy
-
-If you want the same behaviour as before:
-
-    # replace this:
-    snapshot_preserve_daily   <daily>
-    snapshot_preserve_weekly  <weekly>
-    snapshot_preserve_monthly <monthly>
-
-    # with:
-    snapshot_preserve_min  <daily>d
-    snapshot_preserve      <weekly>w <monthly>m
-
-    # ... do the same with "target_preserve_*" options
-
-
-But what you probably want is something like:
-
-    snapshot_preserve_min  5d
-    snapshot_preserve      <daily>d <weekly>w <monthly>m
-
-    target_preserve_min    no
-    target_preserve        <daily>d <weekly>w <monthly>m *y
-
-This states:
-
-  * Keep all snapshots for five days (no matter how many there are)
-  * Transfer only the first snapshot of a day to the target
-  * Keep all "first snapshots of a day" for `<daily>` days, etc.
-
-
-### Upgrade "resume_missing"
-
-If you have a line: "resume_missing yes" somwhere in your config,
-simply remove it. btrbk always resumes missing backups.
-
-If you have "resume_missing no", you can imitate this behaviour by
-setting:
-
-    target_preserve_min  latest
-    target_preserve      no
-
-This states: "always transfer the latest snapshot to the target".
-
-
-  [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
-  [btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html

lsbtr

@@ -0,0 +1 @@
+btrbk

ssh_filter_btrbk.sh

@@ -1,9 +1,11 @@
-#!/bin/bash
+#!/bin/sh
 
-set -e
+# initialise and sanitise the shell execution environment
-set -u
+unset -v IFS
+export LC_ALL=C
+export PATH='/sbin:/bin:/usr/sbin:/usr/bin'
 
-export PATH=/sbin:/bin:/usr/sbin:/usr/bin
+set -e -u
 
 enable_log=
 restrict_path_list=
@@ -12,55 +14,70 @@ allow_exact_list=
 allow_rate_limit=1
 allow_stream_buffer=1
 allow_compress=1
-compress_list="gzip|pigz|bzip2|pbzip2|xz|lzop|lz4"
+compress_list='gzip|pigz|bzip2|pbzip2|bzip3|xz|lzop|lz4|zstd'
 
 # note that the backslash is NOT a metacharacter in a POSIX bracket expression!
 option_match='-[a-zA-Z0-9=-]+'   # matches short as well as long options
-file_match='[0-9a-zA-Z_@+./-]*'  # matches file path (equal to $file_match in btrbk)
+file_match_sane='/[0-9a-zA-Z_@+./-]*' # matches file path (equal to ${file_match} in btrbk < 0.32.0)
+file_match="/[^']*" # btrbk >= 0.32.0 quotes file arguments: match all but single quote
+file_arg_match="('${file_match}'|${file_match_sane})" # support btrbk < 0.32.0
 
 log_cmd()
 {
-    if [[ -n "$enable_log" ]]; then
+    local priority="$1"
-        logger -p $1 -t ssh_filter_btrbk.sh "$2 (Name: ${LOGNAME:-<unknown>}; Remote: ${SSH_CLIENT:-<unknown>})${3:+: $3}: $SSH_ORIGINAL_COMMAND"
+    local authorisation_decision="$2"
+    local reason="${3-}"
+
+    if [ -n "${enable_log}" ]; then
+        logger -p "${priority}" -t ssh_filter_btrbk.sh "${authorisation_decision} (Name: ${LOGNAME:-<unknown>}; Connection: ${SSH_CONNECTION:-<unknown>})${reason:+: ${reason}}: ${SSH_ORIGINAL_COMMAND}"
     fi
 }
 
 allow_cmd()
 {
-    allow_list="${allow_list}|$1"
+    local cmd="$1"
+
+    allow_list="${allow_list}|${cmd}"
 }
 
 allow_exact_cmd()
 {
-    allow_exact_list="${allow_exact_list}|$1"
+    local cmd="$1"
+
+    allow_exact_list="${allow_exact_list}|${cmd}"
 }
 
 reject_and_die()
 {
-    local reason=$1
+    local reason="$1"
-    log_cmd "auth.err" "btrbk REJECT" "$reason"
+
-    echo "ERROR: ssh_filter_btrbk.sh: ssh command rejected: $reason: $SSH_ORIGINAL_COMMAND" 1>&2
+    log_cmd 'auth.err' 'btrbk REJECT' "${reason}"
+    printf 'ERROR: ssh_filter_btrbk.sh: ssh command rejected: %s: %s\n' "${reason}" "${SSH_ORIGINAL_COMMAND}" >&2
     exit 255
 }
 
 run_cmd()
 {
-    log_cmd "auth.info" "btrbk ACCEPT"
+    log_cmd 'auth.info' 'btrbk ACCEPT'
-    eval " $SSH_ORIGINAL_COMMAND"
+    eval " ${SSH_ORIGINAL_COMMAND}"
 }
 
 reject_filtered_cmd()
 {
-    if [[ -n "$restrict_path_list" ]]; then
+    if [ -n "${restrict_path_list}" ]; then
-	# match any of restrict_path_list with or without trailing slash,
+	# match any of restrict_path_list,
 	# or any file/directory (matching file_match) below restrict_path
-	path_match="(${restrict_path_list})(/${file_match})?"
+	path_match="'(${restrict_path_list})(${file_match})?'"
+	path_match_legacy="(${restrict_path_list})(${file_match_sane})?"
     else
 	# match any absolute file/directory (matching file_match)
-	path_match="/${file_match}"
+	path_match="'${file_match}'"
+	path_match_legacy="${file_match_sane}"
     fi
+    # btrbk >= 0.32.0 quotes files, allow both (legacy)
+    path_match="(${path_match}|${path_match_legacy})"
 
-    if [[ -n "$allow_compress" ]]; then
+    if [ -n "${allow_compress}" ]; then
         decompress_match="(${compress_list}) -d -c( -[pT][0-9]+)?"
         compress_match="(${compress_list}) -c( -[0-9])?( -[pT][0-9]+)?"
     else
@@ -70,8 +87,8 @@ reject_filtered_cmd()
 
     # rate_limit_remote and stream_buffer_remote use combined
     # "mbuffer" as of btrbk-0.29.0
-    if [[ -n "$allow_stream_buffer" ]] || [[ -n "$allow_rate_limit" ]]; then
+    if [ -n "${allow_stream_buffer}" ] || [ -n "${allow_rate_limit}" ]; then
-        mbuffer_match="mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?"
+        mbuffer_match='mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?'
     else
         mbuffer_match=
     fi
@@ -81,35 +98,44 @@ reject_filtered_cmd()
     stream_in_match="(${decompress_match} \| )?(${mbuffer_match} \| )?"
     stream_out_match="( \| ${mbuffer_match})?( \| ${compress_match}$)?"
 
+    # `grep`’s `-q`-option is not used as it may cause an exit status of `0` even
+    # when an error occurred.
+
     allow_stream_match="^${stream_in_match}${allow_cmd_match}${stream_out_match}"
-    if [[ $SSH_ORIGINAL_COMMAND =~ $allow_stream_match ]] ; then
+    if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${allow_stream_match}" >/dev/null 2>/dev/null; then
         return 0
     fi
 
-    exact_cmd_match="^${allow_exact_list}$";
+    exact_cmd_match="^(${allow_exact_list})$";
-    if [[ $SSH_ORIGINAL_COMMAND =~ $exact_cmd_match ]] ; then
+    if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${exact_cmd_match}" >/dev/null 2>/dev/null; then
         return 0
     fi
 
-    reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${restrict_path_list//|/\", \"}\")}"
+    local formatted_restrict_path_list="$(printf '%s' "${restrict_path_list}" | sed 's/|/", "/g')"
+    reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${formatted_restrict_path_list}\")}"
 }
 
 
 # check for "--sudo" option before processing other options
 sudo_prefix=
-for key; do
+for key in "$@"; do
-    [[ "$key" == "--sudo" ]] && sudo_prefix="sudo -n "
+    if [ "${key}" = '--sudo' ]; then
+        sudo_prefix='sudo -n '
+    fi
+    if [ "${key}" = '--doas' ]; then
+        sudo_prefix='doas -n '
+    fi
 done
 
-while [[ "$#" -ge 1 ]]; do
+while [ "$#" -ge 1 ]; do
     key="$1"
 
-    case $key in
+    case "${key}" in
       -l|--log)
           enable_log=1
           ;;
 
-      --sudo)
+      --sudo|--doas)
           # already processed above
           ;;
 
@@ -154,26 +180,30 @@ while [[ "$#" -ge 1 ]]; do
           ;;
 
       *)
-          echo "ERROR: ssh_filter_btrbk.sh: failed to parse command line option: $key" 1>&2
+          printf 'ERROR: ssh_filter_btrbk.sh: failed to parse command line option: %s\n' "${key}" >&2
           exit 255
           ;;
     esac
     shift
 done
 
-allow_cmd "${sudo_prefix}btrfs subvolume show"; # subvolume queries are always allowed
+# NOTE: subvolume queries are NOT affected by "--restrict-path":
-allow_exact_cmd "${sudo_prefix}btrfs subvolume list ${file_match}"; # subvolume queries are always allowed
+# btrbk also calls show/list on the mount point of the subvolume
-allow_cmd "${sudo_prefix}readlink"              # used to resolve mountpoints
+allow_exact_cmd "${sudo_prefix}btrfs subvolume (show|list)( ${option_match})* ${file_arg_match}";
-allow_exact_cmd "cat /proc/self/mountinfo"      # used to resolve mountpoints
+allow_cmd "${sudo_prefix}readlink"                    # resolve symlink
-allow_exact_cmd "cat /proc/self/mounts"         # legacy, for btrbk < 0.27.0
+allow_exact_cmd "${sudo_prefix}test -d ${file_arg_match}" # check directory (only for compat=busybox)
+allow_exact_cmd 'cat /proc/self/mountinfo'            # resolve mountpoints
+allow_exact_cmd 'cat /proc/self/mounts'               # legacy, for btrbk < 0.27.0
 
 # remove leading "|" on alternation lists
-allow_list=${allow_list#\|}
+allow_list="${allow_list#\|}"
-allow_exact_list=${allow_exact_list#\|}
+allow_exact_list="${allow_exact_list#\|}"
-restrict_path_list=${restrict_path_list#\|}
+restrict_path_list="${restrict_path_list#\|}"
 
-case "$SSH_ORIGINAL_COMMAND" in
+case "${SSH_ORIGINAL_COMMAND}" in
     *\.\./*)  reject_and_die 'directory traversal'  ;;
+    *'
+'*)           reject_and_die 'unsafe character LF'  ;;
     *\$*)     reject_and_die 'unsafe character "$"' ;;
     *\&*)     reject_and_die 'unsafe character "&"' ;;
     *\(*)     reject_and_die 'unsafe character "("' ;;
@@ -182,7 +212,7 @@ case "$SSH_ORIGINAL_COMMAND" in
     *\<*)     reject_and_die 'unsafe character "<"' ;;
     *\>*)     reject_and_die 'unsafe character ">"' ;;
     *\`*)     reject_and_die 'unsafe character "`"' ;;
-    *\|*)     [[ -n "$allow_compress" ]] || [[ -n "$allow_rate_limit" ]] || [[ -n "$allow_stream_buffer" ]] || reject_and_die 'unsafe character "|"' ;;
+    *\|*)     [ -n "${allow_compress}" ] || [ -n "${allow_rate_limit}" ] || [ -n "${allow_stream_buffer}" ] || reject_and_die 'unsafe character "|"' ;;
 esac
 
 reject_filtered_cmd