btrbk: fix duplicate cmdline option -l in lsbtr

After 7 years and a lot of warnings we can drop this code. Note that
most of those refer to `upgrade_to_v0.23.0.md`, which was removed in
16364e2.

.gitattributes

@@ -0,0 +1,3 @@
+# remove files from deployment using `git archive`
+.gitattributes export-ignore
+.gitignore export-ignore

.github/FUNDING.yml

@@ -0,0 +1,3 @@
+# These are supported funding model platforms
+
+custom: ['https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S', 'https://btc.com/19DYtoEepxBmn9ZPspJGZrhCtySKCxPcP1']

.github/workflows/codespell.yml

@@ -0,0 +1,26 @@
+---
+name: Codespell
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+permissions:
+  contents: read
+
+jobs:
+  codespell:
+    name: Check for spelling errors
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Codespell
+        uses: codespell-project/actions-codespell@v2
+        with:
+          check_filenames: true
+          skip: ".git,*.pdf,*.svg"
+          ignore_words_list: uptodate

ChangeLog

@@ -1,10 +1,479 @@
+btrbk-0.32.6
+
+  * Fix backup of unrelated (by parent_uuid) snapshots (close #339).
+  * Remove echo -e for portability (close #506).
+  * Support btrfs send protocol v2 (send_protocol and
+    send_compressed_data config options).
+  * Add bzip3 support.
+  * Convert ssh_filter_btrbk.sh to POSIX sh, and harden it.
+  * Slight change in ssh_filter_btrbk.sh logging output.
+  * Minor bugfixes, stability and documentation improvements.
+
+btrbk-0.32.5
+
+  * Correct handling of zero-size raw info file (close #491).
+
+btrbk-0.32.4
+
+  * Fix regression: wrong deprecation warnings in some cases.
+
+btrbk-0.32.3
+
+  * Fix deletion of many subvolumes at once (close #476).
+  * Allow disabling ssh_identity and ssh_user options.
+  * Minor bugfixes and documentation improvements.
+
+btrbk-0.32.2
+
+  * Fix regression: archive on missing target directories.
+  * Fix action "config print".
+  * Print version and help message to stdout (close #452).
+  * Check results of filesystem usage (close #469).
+  * Add "compat ignore_receive_errors" option (close #383).
+  * Support multiple gpg recipients (close #471).
+  * Fix changelog: remove "compat missing_otime" item in v0.32.1.
+  * Fix changelog: correct "warn_unknown_targets" item in v0.31.3.
+  * Minor bugfixes and framework improvements.
+
+btrbk-0.32.1
+
+  * Fix regression: correctly read mountinfo (close #445).
+  * Fix regression: filter paths for "lsbtr" (action "ls").
+  * Add backend btrfs-progs-doas (close #444).
+  * Allow directory traversal for local command line arguments.
+  * Display all source subvolumes in "usage" action.
+  * Handle errors from btrfs filesystem usage.
+
+btrbk-0.32.0
+
+  * MIGRATION
+    - If timestamp_format is not configured, explicitly set
+      "timestamp_format short" to revert old behavior.
+    - Update ssh_filter_btrbk.sh on remote hosts.
+  * Change default for timestamp_format to "long".
+  * Optional "volume" config section.
+  * Use "volume" section only for grouping and relative paths.
+  * Allow absolute path for "subvolume" and "snapshot_dir"
+    (close #407, #121).
+  * Support subvolume names with UTF-8 characters (close #392, #213,
+    #120). Add "safe_commands" option for paranoid people.
+  * Add "incremental_prefs" configuration option (close #387).
+  * Change incremental prefs policy, giving snapshots created by btrbk
+    higher preference than the global ones resolved by parent-uuid.
+  * Change "incremental_clones" option to boolean.
+  * Drop support of deprecated raw file format (btrbk < 0.26.0).
+  * Drop support of deprecated "-r, --resume-only" cmdline option.
+  * Fix ambiguous naming in --format=raw output of "list" actions.
+  * Accept quoted values in config.
+  * Use single quotes for raw table output.
+
+btrbk-0.31.3
+
+  * Add "warn_unknown_targets" configuration option (close #393).
+  * Add -1,--single-column command-line option for listing actions.
+  * Allow relative path command line argument for all actions.
+  * Add support for zstd adaptive compression (close #397).
+  * Add contrib/tools/btrbk_restore_raw.py (close #401).
+  * Minor bugfixes and framework improvements.
+
+btrbk-0.31.2
+
+  * MIGRATION
+    - Update ssh_filter_btrbk.sh on remote hosts.
+  * ssh_filter_btrbk.sh: Fix security vulnerability.
+    Specially crafted commands may be executed without being properly
+    checked. Applies to remote hosts filtering ssh commands using
+    ssh_filter_btrbk.sh in authorized_keys.
+  * Warn if no subvolume defined in config (close #378).
+
+btrbk-0.31.1
+
+  * Bugfix: duplicate snapshot location check (close #360).
+  * Bugfix: action "list all" (close #367).
+  * btrbk-mail: optionally prefix command output lines.
+
+btrbk-0.31.0
+
+  * Add action "extents [diff]": print accurate disk space usage based
+    on block regions (extent data, FIEMAP ioctl).
+  * Improve action "list" and "stats" table output (close #334).
+  * Add subcommand "list all", make default for "list" action.
+  * Change semantics on output of action "list": show only btrbk
+    snapshots/backups (close #333).
+  * Print human readable units in action "diff" (close #345).
+  * Allow custom table format definitions from command line.
+  * Add Bash completion (close #343).
+  * Add support for zstd compression (close #358).
+  * Bugfix: fs_path fallback for action "origin" and logging.
+  * Bugfix: makefile bin-links race (close #341).
+
+btrbk-0.30.0
+
+  * MIGRATION
+    - In your scripts (e.g. cron jobs, systemd units), replace
+      "/usr/sbin/btrbk" with "/usr/bin/btrbk".
+  * Install to "/usr/bin" instead of "/usr/sbin".
+  * Support IPv6 addresses (close #332).
+  * Add "compat busybox" configuration option (close #327).
+  * Create "lsbtr" symlink (alias for "btrbk ls").
+  * Improve action "ls" (allow url, bugfixes).
+  * Add "backend_local_user" configuration option.
+  * btrbk-mail: add more elaborated email and rsync options.
+  * Minor bugfixes and framework improvements.
+
+btrbk-0.29.1
+
+  * Allow multiple path arguments for action "ls".
+  * Change default output to format=short for action "ls".
+  * ssh_filter_btrbk.sh: exclude "btrfs subvolume show|list" from
+    restrict-path (close #309).
+  * Bugfix: fix filter statement match on wildcards (close #311).
+  * Fix regression: print ssh errors only if needed.
+
+btrbk-0.29.0
+
+  * MIGRATION
+    - If stream_buffer or rate_limit is configured, please install
+      "mbuffer" (version >= 20180505). Note that mbuffer is only
+      required on remote hosts if stream_buffer_remote or
+      rate_limit_remote is set.
+    - In order to mimic old behavior, replace rate_limit with
+      rate_limit_remote, and read btrbk.conf(5).
+    - If you are using table output other than --format=raw in your
+      scripts, note that the column headings changed from pretty
+      two-line to (uppercase) one-line format. Either add --pretty
+      option to mimic old behavior, or adapt your scripts.
+  * Dropped run-time dependency on "pv" (in favor of "mbuffer").
+  * Combined stream_buffer and rate_limit: for rate_limit, use
+    "mbuffer" (on local host) instead of "pv" (on source host).
+  * Add stream_buffer_remote and rate_limit_remote options.
+  * Use "mbuffer" instead of "pv" for --progress display.
+  * Add stream_buffer functionality for raw targets.
+  * Add action "ls": list all btrfs subvolumes below given path.
+  * Change table output: print single-line, uppercase headings.
+  * ssh_filter_btrbk.sh: whitelist mkdir if -t,--target option is set,
+    used by "btrbk archive" action (close #297).
+  * Bugfix: handle subvolumes with special characters (close #288).
+  * Bugfix: don't display "<no_action>" in backup summary if volume is
+    skipped by --exclude or noauto (close #291).
+  * Bugfix: systemd: Use WantedBy=timers.target instead of
+    WantedBy=multi-user.target in btrbk.timer (close #293).
+
+btrbk-0.28.3
+
+  * Bugfix: return exitcode=10 if action skipped by stray subvolumes
+    (close: #285).
+  * Bugfix: correctly handle root subvolumes having uuid.
+  * Bugfix: fix mount point resolving.
+
+btrbk-0.28.2
+
+  * Fix regression: crash if raw target dir is empty (close  #281).
+  * Bugfix: honor incremental_resolve on targets: resolve from
+    mountpoint instead of directory.
+  * Bugfix: handle directories named "0" correctly.
+
+btrbk-0.28.1
+
+  * Fix regression: crash if deep parent/child relations exceed
+    depth=100 (close #279).
+  * Remove pre-generated man pages, requires "asciidoctor" for builds.
+
+btrbk-0.28.0
+
+  * MIGRATION
+    - Replace "ssh_port" options with "ssh://hostname[:port]" notation
+      on "volume" or "target" declarations.
+    - Check filter statements in your scripts ("btrbk run <filter>"),
+      especially when using groups having the same name as subvolumes.
+  * Fix table format "resolved" (btrbk list snapshots|backups).
+  * Add btrbk-verify: tool for automated backup integrity check based
+    on rsync (close #256).
+  * Use "ssh://hostname[:port]" notation instead of ambiguous
+    "ssh_port" option (now deprecated).
+  * Bugfix: correctly handle multiple volume sections with same host,
+    but distinct port numbers (virtual machines setups, close #233).
+  * Remove selected rows ("*host", "*port") from table output if all
+    values are empty.
+  * Consider all parent/child relations (close #255).
+  * Disallow unrelated parents for btrfs-send if incremental=strict.
+  * Add clone sources to btrfs-send if necessary.
+  * Add "incremental_resolve" configuration option.
+  * Add "incremental_clones" configuration option.
+  * Add "noauto" configuration option.
+  * Add "--exclude" command line option.
+  * Make target_type optional, defaults to "send-receive".
+  * Use more sophisticated filter arguments.
+  * Allow whitespace separated "group" option.
+
+btrbk-0.27.2
+
+  * Bump dependency >= btrfs-progs-4.12 (as of btrbk-0.27.0).
+  * Trigger autofs mount while resolving mountpoints (close #259).
+  * Bugfix: fix scheduler when overriding "target_preserve_min" in
+    combination with global "target" section.
+
+btrbk-0.27.1
+
+  * MIGRATION
+    - Update ssh_filter_btrbk.sh on remote hosts if using
+      backend=btrfs-progs-sudo.
+  * Bugfix (regression): call "sudo -n readlink" for
+    backend=btrfs-progs-sudo (close #257).
+  * Bugfix (regression): fix failing "config print" command.
+  * Don't print title and blank lines for --format=raw, syslog and
+    transaction log.
+
+btrbk-0.27.0
+
+  * MIGRATION
+    - Run "btrbk prune --dry-run --print-schedule" and check if any
+      snapshots/backups would get deleted [1] [2]. If you want to
+      forcibly preserve a snapshot/backup forever, rename it (btrbk
+      ignores subvolumes with unknown naming scheme), e.g.:
+      "mv mysubvol.YYYYMMDD mysubvol.YYYYMMDD.keep_forever"
+    - Update ssh_filter_btrbk.sh on remote hosts (btrbk always calls
+      "readlink" and "cat /proc/self/mountinfo").
+  * Add "preserve_hour_of_day" configuration option (close #202).
+  * Allow backup of filesystem root using "subvolume ." (close #240).
+  * Add "-S" command line option (shortcut for --print-schedule).
+  * Bugfix: correct scheduling of "first weekly backup in month/year"
+    (close #217) [1] [2].
+  * Bugfix: add "btrfs filesystem usage" backend mapping (close #222).
+  * Bugfix: do not fail (exitcode=10) if targets are not accessible on
+    "btrfs snapshot --preserve".
+  * Bugfix: if "incremental no" is configured on target, do not keep
+    latest common snapshot.
+  * Enhance internal data structures:
+    - Allow snapshot_dir to be a mountpoint.
+    - Search complete target tree for correlated subvolumes.
+    - Include snapshots from all mountpoints as candidates (disabled
+      due to upstream bug: github.com/kdave/btrfs-progs/issues/96).
+    - Read /proc/self/mountinfo instead of /proc/self/mounts.
+    - Always read /proc/self/mountinfo.
+    - Resolve realpath using readlink(1).
+  * Fallback to "asciidoctor" for manpage generation (close #219).
+
+      [1] https://github.com/digint/btrbk/issues/217
+      [2] https://github.com/digint/btrbk/commit/719fb5f
+
+btrbk-0.26.1
+
+  * Add "archive_exclude" configuration option.
+  * Add warning on redefined configuration option.
+  * Bugfix: fix parsing of "openssl_iv_size" configuration option.
+  * Bugfix: fix filter statement matching for volume=/ (close #209).
+
+btrbk-0.26.0
+
+  * MIGRATION
+    - If you are using raw targets, make sure to run the
+      "raw_suffix2sidecar" utility in each target directory.
+  * Support for btrfs-progs v4.13.2: adapt parsing of "btrfs sub list"
+    output (close #192).
+  * Add "resume" command, replacement for "-r, --resume-only" command
+    line option (which is now deprecated).
+  * Add "snapshot" command (close #150).
+  * Add "prune" command.
+  * Add "--preserve-snapshots" and "--preserve-backups" options.
+  * Add "--wipe" command line option (close #173).
+  * Change raw backup format (sidecar file instead of uuid in file).
+  * Honor target_preserve for raw targets (delete raw targets).
+  * Add symmetric encryption for raw targets (close #157).
+  * Add "{snapshot,target,archive}_qgroup_destroy" configuration
+    options (close #49, #189).
+  * Do not run in "perl taint mode" by default: remove "perl -T" in
+    hashbang; hardcode $PATH only if taint mode is enabled.
+  * Remove "duration" column from transaction_log/transaction_syslog.
+  * Resolve ancestors (recursive on parent_uuid chain) when searching
+    for latest common subvolume.
+  * Generate man pages from asciidoc (remove raw groff sources).
+  * Bugfix: ssh_filter_btrbk: accept mbuffer command (stream_buffer).
+  * Bugfix: print correct (end-)time in transaction_log.
+  * Bugfix: check path when expanding wildcards (close #181).
+  * Bugfix: never show failed deletes in summary (close #183).
+
+btrbk-0.25.1
+
+  * Support for btrfs-progs v4.12: fix parsing of "btrfs sub show"
+    output, which now prints relative paths (close #171).
+  * Add "stream_buffer" configuration option (close #154).
+  * Bugfix: accept "no" for "transaction_log", "transaction_syslog"
+    and "lockfile" configuration options.
+  * Show "up-to-date" status for backups in "stats" command.
+  * Show "correlated" status instead of "orphaned" in "stats" command.
+  * Check source subvolumes for readonly and received_uuid flags, and
+    abort if one of them is set.
+
+btrbk-0.25.0
+
+  * MIGRATION
+    - If you call ssh_filter_btrbk.sh with "--sudo" option, make sure
+      to set "backend btrfs-progs-sudo" in btrbk.conf for this host.
+    - If "rate_limit" is enabled, update ssh_filter_btrbk.sh on remote
+      source hosts, and make sure the "pv" command is available there.
+  * Allow converting backup disks to source disks (close #114).
+  * Add "backend btrfs-progs-sudo" configuration option (close #115).
+  * Show aggregate "size" and "used" for "usage" action (close #119).
+  * Add "raw_target_split" configuration option (close #125).
+  * Allow trailing comments in btrbk.conf (close #129).
+  * Bugfix: rate limiting must be done after compression (close #134).
+  * raw_target_encrypt: Always set "gpg --no-random-seed-file":
+    prevents creation of "~/.gnupg/random_seed" with slight performance
+    penalty.
+
+btrbk-0.24.0
+
+  * MIGRATION
+    - update ssh_filter_btrbk.sh on remote hosts if "stream_compress"
+      is enabled. Also add "--compress" option to ssh_filter_btrbk.sh
+      invocation if "stream_compress" is enabled.
+  * Add "stream_compress" configuration option.
+  * Perform extra metadata check on target subvolume after "btrfs
+    receive" (adds an additional call to "btrfs subvolume show").
+  * Bugfix: Replace "realpath" with "readlink" in ssh_filter_btrbk.sh
+  * Add "raw_target_block_size" configuration option (close #105).
+  * Add "backend" configuration option (experimental).
+  * Bugfix: fix "list latest" with no snapshots (close #111).
+  * Support for btrfs-progs v4.8.3: fix parsing of "btrfs sub show"
+    output, which has changed for toplevel subvolume.
+
+btrbk-0.23.3
+
+  * Replace "realpath -e" with "readlink -e" for mountpoint discovery
+    (close #92).
+  * Dynamically set BINDIR in btrbk.service by Makefile.
+  * Add ChangeLog to DOCDIR in Makefile.
+
+btrbk-0.23.2
+
+  * Added "lockfile" configuration option and --lockfile command line
+    option (close: #81).
+  * Bugfix: raw targets: correctly handle multiple backups in same
+    target directory (close: #87).
+  * Use relative instead of absolute binary calls in btrbk-mail.
+
+btrbk-0.23.1
+
+  * Bugfix: set correct parent section when propagating targets
+    (close: #85).
+  * Add syslog output of transaction log (close #82).
+  * Do not print headers to transaction log anymore.
+  * Explain "orphaned" status in "stats" command, and suppress it on
+    "list backups" command (close: #76).
+
+btrbk-0.23.0
+
+  * INCOMPATIBLE CONFIGURATION:
+    * Please read "doc/upgrade_to_v0.23.0.md" for details on updating
+      the configuration file (/etc/btrbk/btrbk.conf).
+    * Dropped "btrfs_progs_compat" option. Need btrfs-progs >= v3.18.
+    * Removed "resume_missing" configuration option.
+  * Create backups only if needed to satisfy retention policy.
+  * Preserve FIRST backup of hour/day/week/month instead of LAST.
+  * Replaced "{snapshot,target}_preserve_{daily,weekly,monthly}"
+    configuration options with "{snapshot,target}_preserve_min" and
+    "{snapshot,target}_preserve NNh NNd NNw NNm NNy" options.
+  * Added hourly/yearly retention policies (close: #36, #69).
+  * Allow regular directories for send-receive targets (close: #77).
+  * Allow wildcards in subvolume section (close: #71).
+  * Propagate targets defined in "volume" or "root" context to all
+    "subvolume" sections (close: #78).
+  * Added "archive" command (close: #79).
+  * Changed output format of "origin" command, add table formats.
+  * Added configuration option "rate_limit" (close: #72).
+  * Added new timestamp_format "long-iso", having a timezone postfix.
+  * Added "--print-schedule" command line option.
+  * Detect interrupted transfers of raw targets (close: #75).
+  * Always read "readonly" flag (additional call to btrfs-progs).
+  * Warn on receive targets at unexpected location (instead of abort).
+  * On incremental send/receive, use all snapshots sharing same
+    parent_uuid as candidates for "-p <parent>" instead of only older
+    snapshots. As last resort, use subvolumes in snapshot_dir matching
+    btrbk file name scheme as candidates (which allows incremental
+    backups after the parent vanished, e.g. after backup restore).
+  * Use perl built-in Time::Local instead of Date::Calc.
+  * Improvements of internal data structures.
+
+btrbk-0.22.2
+
+  * Bugfix: fix checks on "btrfs sub show" output, which resulted in
+    breakage on btrfs-progs < 4.1 (close: #68).
+
+btrbk-0.22.1
+
+  * Support for btrfs-progs v4.4 (close: #66).
+
+btrbk-0.22.0
+
+  * Bugfix: fix monthly schedule if older than 10 weeks (close: #59).
+  * Bugfix: fix sprintf used by config option "timestamp_format long"
+    when using perl-5.22.0 (close: #57).
+  * Bugfix: fix "--progress" option (close: #64).
+  * Added "clean" command (close: #61).
+  * Added "-n, --dry-run" option.
+  * Added configuration options "raw_target_compress_level",
+    "raw_target_compress_threads" (close: #60).
+  * Added "stats" command (close: #54).
+  * Print "$hostname:$path" instead of "{$hostname}$path" in summary
+    and logs.
+
+btrbk-0.21.0
+
+  * Added transaction log (configuration option "transaction_log").
+  * Added configuration option "group".
+  * Allow filtering of all commands by group as well as targets.
+  * Added "list backups|snapshots|latest|config|source|volume|target"
+    commands (while removing "tree" command).
+  * Added "config print" command.
+  * Added "--format=table|long|raw" and "-t,--table" command line
+    options, producing tabular and raw (machine-readable) output for
+    "(dry)run", "tree" and "list" commands.
+  * Print scheduler details if -v option is set on action run/dryrun.
+  * Added configuration option "ssh_cipher_spec" (close: #47).
+  * Added "target raw", with GnuPG and compression support
+    (experimental).
+  * Added configuration option "timestamp_format short|long".
+  * Replaced "info" command with "usage", with tabular output.
+  * Bugfix: correctly handle "incremental no" option.
+  * Bugfix: return exit status 10 instead of 0 if one or more backup
+    tasks aborted.
+  * Better error handling for send/receive commands (close: #33).
+  * Hardened ssh_filter_btrbk.sh script: fine-grained access control,
+    restrict-path option, sudo option (close: #45).
+  * Added example cron script.
+
+btrbk-0.20.0
+
+  * Added configuration option "ssh_port" (close: #39).
+  * Added configuration option "ssh_compression" (close: #44).
+  * Added command line option "--progress" (close: #42).
+  * Bugfix: correct handling of empty "snapshot_dir" (close: #43).
+  * Accept long options on command line.
+  * Documentation: added FAQ.
+
+btrbk-0.19.3
+
+  * Bugfix: fix sorting of dates in schedule().
+  * Bugfix: correct parsing of btrfs subvolume list (close: #28).
+  * Support for btrfs-progs v4.1-rc1.
+
+btrbk-0.19.2
+
+  * Bugfix: fix crash when using btrfs-progs < 3.17.3 (close: #24).
+
+btrbk-0.19.1
+
+  * Use "cgen" for snapshot comparison.
+  * Bugfix: fix crash in summary display (close: #22).
+
 btrbk-0.19.0
 
   * Added "snapshot_create onchange", which skips snapshot creation if
     the latest snapshot is up-to-date (i.e. has same generation as the
     source subvolume).
   * Improved handling of command line subvolume filter for "run",
-    "dryrun" and "tree" actions (closes: #21).
+    "dryrun" and "tree" actions (close: #21).
   * Bugfix: fixed crash in action "diff".
 
 btrbk-0.18.0
@@ -16,10 +485,10 @@ btrbk-0.18.0
       - "snapshot_create_always no"  -> "snapshot_create ondemand"
   * Set PATH variable instead of using absolute "/sbin/btrfs" for
     compatibility with all linux distros out there, which all install
-    'btrfs' in different locations (closes: #20).
+    'btrfs' in different locations (close: #20).
   * Added configuration option "snapshot_create", replacing option
     "snapshot_create_always". This allows setups with multiple btrbk
-    instances on several hosts (closes: #18).
+    instances on several hosts (close: #18).
   * Added command line option -r (resume only).
   * Catch and display errors from "btrfs subvolume show".
   * Include systemd service and timer unit for daily backups.
@@ -28,23 +497,23 @@ btrbk-0.17.1
 
   * Bugfix: send/receive: delete possibly left-behind garbled
     subvolume on failure. Fail with unrecoverable error if stray
-    target subvolume is in the way (closes: #17).
+    target subvolume is in the way (close: #17).
   * Bugfix: assume unreachable target as clean on snapshot creation if
-    snapshot_create_always is set (closes: #19).
+    snapshot_create_always is set (close: #19).
 
 btrbk-0.17.0
 
   * New versioning scheme using more common three-level versions.
   * Code refactoring: cleanup of data structures and handling of btrfs
     subvolume tree, as well as security related code parts.
-  * Correct handling of symlinks to btrfs subvolumes (closes: #12).
-  * Added configuration option "snapshot_name" (closes: #5).
+  * Correct handling of symlinks to btrfs subvolumes (close: #12).
+  * Added configuration option "snapshot_name" (close: #5).
   * Log messages now go to stderr, only the summary is printed on
     stdout.
-  * Bugfix: allow "0" as subvolume name (closes: #10).
-  * Bugfix: allow "/" as volume name (closes: #15).
+  * Bugfix: allow "0" as subvolume name (close: #10).
+  * Bugfix: allow "/" as volume name (close: #15).
   * Bugfix: check source AND targets for determining snapshot postfix
-    (closes: #11).
+    (close: #11).
   * Bugfix: fixed "diff" action (colses: #14).
   * Allow '+' character for subvolume names.
   * Filesystems on remote hosts are now printed as
@@ -57,9 +526,9 @@ btrbk-0.16
 btrbk-0.15
 
   * Added configuration option "btrfs_progs_compat", to be enabled if
-    using btrfs-progs < 3.17 (closes: #6).
+    using btrfs-progs < 3.17 (close: #6).
   * Added configuration option "resume_missing", for automatic resume
-    of missing backups (closes: #8).
+    of missing backups (close: #8).
   * Removed configuration option "receive_log" in favor of printing
     errors from "btrfs receive".
   * Bugfix: show correct exit code on external command failure.
@@ -68,18 +537,18 @@ btrbk-0.15
 btrbk-0.14
 
   * Bugfix: correctly handle empty target subvolumes (blocker for all
-    new users; closes: #4).
+    new users; close: #4).
 
 btrbk-0.13
 
   * Bugfix: allow '@' character for subvolume names (blocker for
     ubuntu users, since ubuntu prefixes all subvolumes with '@' in its
-    subvolume layout; closes: #3).
+    subvolume layout; close: #3).
 
 btrbk-0.12
 
   * Cleaner and more generic parsing of btrfs subvolume list.
-  * Bugfix: subvolumes are also allowed for "snapshot_dir" (closes:
+  * Bugfix: subvolumes are also allowed for "snapshot_dir" (close:
     #1, #2).
 
 btrbk-0.11

Makefile

@@ -1,42 +1,113 @@
-PN         = btrbk
+#
+# Btrbk is a single perl script, and does not require any special
+# installation procedures or libraries. There is no need to run the
+# "all" build target if you don't want to build the man pages (see
+# doc/Makefile).
+#
+# Note: systemd units (file names) are hardcoded in "install-systemd"
+# build target for simplicity.
+#
 
-PREFIX    ?= /usr
-CONFDIR    = /etc
-CRONDIR    = /etc/cron.daily
-BINDIR     = $(PREFIX)/sbin
-DOCDIR     = $(PREFIX)/share/doc/$(PN)
-SCRIPTDIR  = $(PREFIX)/share/$(PN)/scripts
-SYSTEMDDIR = $(PREFIX)/lib/systemd/system
-MAN1DIR    = $(PREFIX)/share/man/man1
-MAN5DIR    = $(PREFIX)/share/man/man5
+BIN         = btrbk
+BIN_LINKS   = lsbtr
+CONFIGS     = btrbk.conf.example
+DOCS        = ChangeLog \
+              README.md
+SCRIPTS     = ssh_filter_btrbk.sh \
+              contrib/cron/btrbk-mail \
+              contrib/cron/btrbk-verify \
+              contrib/migration/raw_suffix2sidecar \
+              contrib/crypt/kdf_pbkdf2.py \
+              contrib/tools/btrbk_restore_raw.py
 
-all:
-	@echo 'nothing to do for "all"'
+PN          = btrbk
+PREFIX     ?= /usr
+CONFDIR     = /etc
+CRONDIR     = /etc/cron.daily
+BINDIR      = $(PREFIX)/bin
+DOCDIR      = $(PREFIX)/share/doc/$(PN)
+SCRIPTDIR   = $(PREFIX)/share/$(PN)/scripts
+SYSTEMDDIR  = $(PREFIX)/lib/systemd/system
+BASHCOMPDIR = $(PREFIX)/share/bash-completion/completions
+MAN1DIR     = $(PREFIX)/share/man/man1
+MAN5DIR     = $(PREFIX)/share/man/man5
+
+ifeq ($(COMPRESS), yes)
+  DOCS := $(addsuffix .gz,$(DOCS))
+endif
+
+replace_vars = sed \
+	-e "s|@PN@|$(PN)|g" \
+	-e "s|@CONFDIR@|$(CONFDIR)|g" \
+	-e "s|@CRONDIR@|$(CRONDIR)|g" \
+	-e "s|@BINDIR@|$(BINDIR)|g" \
+	-e "s|@DOCDIR@|$(DOCDIR)|g" \
+	-e "s|@SCRIPTDIR@|$(SCRIPTDIR)|g" \
+	-e "s|@SYSTEMDDIR@|$(SYSTEMDDIR)|g" \
+	-e "s|@BASHCOMPDIR@|$(BASHCOMPDIR)|g" \
+	-e "s|@MAN1DIR@|$(MAN1DIR)|g" \
+	-e "s|@MAN5DIR@|$(MAN5DIR)|g"
+
+all: man
+
+install: install-bin install-bin-links install-etc install-completion install-systemd install-share install-man install-doc
 
 install-bin:
-	@echo 'installing main script and config...'
-	install -Dm644 btrbk.conf.example "$(DESTDIR)$(CONFDIR)/btrbk/btrbk.conf.example"
-	install -Dm755 $(PN) "$(DESTDIR)$(BINDIR)/$(PN)"
+	@echo 'installing binary...'
+	install -d -m 755 "$(DESTDIR)$(BINDIR)"
+	install -p -m 755 $(BIN) "$(DESTDIR)$(BINDIR)"
+
+install-bin-links: install-bin
+	@echo 'installing symlinks...'
+	for name in $(BIN_LINKS); do \
+		ln -s -n -f $(BIN) "$(DESTDIR)$(BINDIR)/$$name"; \
+	done
+
+install-etc:
+	@echo 'installing example configs...'
+	install -d -m 755 "$(DESTDIR)$(CONFDIR)/btrbk"
+	install -p -m 644 $(CONFIGS) "$(DESTDIR)$(CONFDIR)/btrbk"
+
+install-completion:
+	@echo 'installing bash completion...'
+	install -d -m 755 "$(DESTDIR)$(BASHCOMPDIR)"
+	install -p -m 644 contrib/bash/completion.bash "$(DESTDIR)$(BASHCOMPDIR)/$(BIN)"
+	for name in $(BIN_LINKS); do \
+		ln -s -n -f $(BIN) "$(DESTDIR)$(BASHCOMPDIR)/$$name"; \
+	done
 
 install-systemd:
 	@echo 'installing systemd service units...'
-	install -Dm644 contrib/systemd/btrbk.service "$(DESTDIR)$(SYSTEMDDIR)/btrbk.service"
-	install -Dm644 contrib/systemd/btrbk.timer "$(DESTDIR)$(SYSTEMDDIR)/btrbk.timer"
+	install -d -m 755 "$(DESTDIR)$(SYSTEMDDIR)"
+	$(replace_vars) contrib/systemd/btrbk.service.in > contrib/systemd/btrbk.service.tmp
+	$(replace_vars) contrib/systemd/btrbk.timer.in > contrib/systemd/btrbk.timer.tmp
+	install -p -m 644 contrib/systemd/btrbk.service.tmp "$(DESTDIR)$(SYSTEMDDIR)/btrbk.service"
+	install -p -m 644 contrib/systemd/btrbk.timer.tmp "$(DESTDIR)$(SYSTEMDDIR)/btrbk.timer"
+	rm contrib/systemd/btrbk.service.tmp
+	rm contrib/systemd/btrbk.timer.tmp
 
 install-share:
 	@echo 'installing auxiliary scripts...'
-	install -Dm644 ssh_filter_btrbk.sh "$(DESTDIR)$(SCRIPTDIR)/ssh_filter_btrbk.sh"
+	install -d -m 755 "$(DESTDIR)$(SCRIPTDIR)"
+	install -p -m 755 $(SCRIPTS) "$(DESTDIR)$(SCRIPTDIR)"
 
-install-man:
-	@echo 'installing manpages...'
-	install -Dm644 doc/btrbk.1 "$(DESTDIR)$(MAN1DIR)/btrbk.1"
-	install -Dm644 doc/btrbk.conf.5 "$(DESTDIR)$(MAN5DIR)/btrbk.conf.5"
-	gzip -9 "$(DESTDIR)$(MAN1DIR)/btrbk.1"
-	gzip -9 "$(DESTDIR)$(MAN5DIR)/btrbk.conf.5"
+install-man: man
+	@echo 'installing man pages...'
+	@$(MAKE) -C doc install-man
 
-install-doc:
+install-doc: $(DOCS)
 	@echo 'installing documentation...'
-	install -Dm644 README.md "$(DESTDIR)$(DOCDIR)/README.md"
-	gzip -9 "$(DESTDIR)$(DOCDIR)/README.md"
+	install -d -m 755 "$(DESTDIR)$(DOCDIR)"
+	install -p -m 644 $(DOCS) "$(DESTDIR)$(DOCDIR)"
+	@$(MAKE) -C doc install-doc
 
-install: install-bin install-systemd install-share install-man install-doc
+man:
+	@echo 'generating manpages...'
+	@$(MAKE) -C doc man
+
+clean:
+	rm -f *.gz
+	@$(MAKE) -C doc clean
+
+%.gz : %
+	gzip -9 -n -c $< > $@

README.md

@@ -1,7 +1,7 @@
 Introduction
 ============
 
-btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs
+Btrbk is a backup tool for btrfs subvolumes, taking advantage of btrfs
 specific capabilities to create atomic snapshots and transfer them
 incrementally to your backup locations.
 
@@ -9,232 +9,291 @@ The source and target locations are specified in a config file, which
 allows to easily configure simple scenarios like "laptop with locally
 attached backup disks", as well as more complex ones, e.g. "server
 receiving backups from several hosts via ssh, with different retention
-policy".
+policies".
 
 Key Features:
 
-- Atomic snapshots
-- Incremental backups
-- Configurable retention policy
-- Backups to multiple destinations
-- Transfer via ssh
-- Resume of backups (if backup target was not reachable for a while)
-- Display file changes between two backups
+  * Atomic snapshots
+  * Incremental backups
+  * Flexible retention policy
+  * Backups to multiple destinations
+  * Transfer via ssh
+  * Robust recovery from interrupted backups (for removable and mobile
+    devices)
+  * Archive to offline storage
+  * Encrypted backups to non-btrfs storage
+  * Wildcard subvolumes (useful for docker and lxc containers)
+  * Transaction log
+  * Comprehensive list and statistics output
+  * Resolve and trace btrfs parent-child and received-from relationships
+  * List file changes between backups
+  * Calculate accurate disk space usage based on block regions
 
-btrbk is intended to be run as a cron job.
+Btrbk is designed to run as a cron job for triggering periodic
+snapshots and backups, as well as from the command line (e.g. for
+instantly creating additional snapshots).
 
 
 Installation
 ============
 
-btrbk comes as a single executable file (perl script), without the
-need of any installation procedures. If you want the package and
-man-pages properly installed, follow the instructions below.
+Btrbk is a single perl script, and does not require any special
+installation procedures or libraries. Download the latest [btrbk
+source tarball], or try latest master:
 
-Prerequisites
--------------
+    wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
+    chmod +x btrbk
+    sudo ./btrbk ls /
 
-- [btrfs-progs]: Btrfs filesystem utilities (use "btrfs_progs_compat"
-  option for hosts running version prior to v3.17)
-- Perl interpreter: probably already installed on your system
-- [Date::Calc]: Perl module
+For more information, read the [installation documentation].
 
-  [btrfs-progs]: http://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
-  [Date::Calc]:  http://search.cpan.org/perldoc?Date::Calc
+  [btrbk source tarball]: https://digint.ch/download/btrbk/releases/
+  [installation documentation]: doc/install.md
 
-Instructions
-------------
 
-In order to install the btrbk executable along with the man-pages and
-an example configuration file, choose one of the following methods:
+### Prerequisites
 
-### Generic Linux System
+  * [btrfs-progs]: Btrfs filesystem utilities >= v4.12
+  * [Perl interpreter]: Probably already installed on your system
+  * [OpenSSH]: If you want to transfer backups from/to remote locations
+  * [mbuffer]: If you want rate limiting and progress bars
 
-Download and unpack the newest stable [btrbk source tarball] and type:
-
-    sudo make install
-
-### Debian Based Distros
-
-Download the newest stable [btrbk debian package], and
-
-    sudo dpkg --install btrbk_<version>_all.deb
-
-### Gentoo Linux
-
-Grab the digint portage overlay from:
-`git://dev.tty0.ch/portage/digint-overlay.git`
-
-    emerge btrbk
-
-### Arch Linux
-
-btrbk is in AUR: https://aur.archlinux.org/packages/btrbk/
-
-  [btrbk source tarball]: http://www.digint.ch/download/btrbk/releases/
-  [btrbk debian package]: http://www.digint.ch/download/btrbk/packages/debian/
+  [btrfs-progs]: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
+  [Perl interpreter]: https://www.perl.org
+  [OpenSSH]: https://www.openssh.com
+  [mbuffer]: https://www.maier-komor.de/mbuffer.html
 
 
 Synopsis
 ========
 
-Please consult the [btrbk(1) man-page] provided with this package for a
-full description of the command line options.
+Please consult the [btrbk(1)] man-page provided with this package for
+a full description of the command line options.
 
-  [btrbk(1) man-page]: http://www.digint.ch/btrbk/doc/btrbk.html
+  [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
 
 
-Configuration File
-==================
+Configuration
+=============
 
 Before running `btrbk`, you will need to create a configuration
 file. You might want to take a look at `btrbk.conf.example` provided
 with this package. For a detailed description, please consult the
-[btrbk.conf(5) man-page].
+[btrbk.conf(5)] man-page.
 
-When playing around with config-files, it is highly recommended to
-check the output using the `dryrun` command before executing the
-backups:
+After a configuration change, it is highly recommended to check it by
+running btrbk with the `-n,--dry-run` option:
 
-    btrbk -c myconfig -v dryrun
+    # btrbk -c /path/to/myconfig -v -n run
 
 This will read all btrfs information on the source/target filesystems
 and show what actions would be performed (without writing anything to
 the disks).
 
-  [btrbk.conf(5) man-page]: http://www.digint.ch/btrbk/doc/btrbk.conf.html
+The examples below assume that the btrfs subvolume containing `home`
+and `rootfs` is mounted at `/mnt/btr_pool`. This is usually the btrfs
+root subvolume, which always has `subvolid=5`.
+
+Mounting `subvolid=5` is *recommended* (mandatory for btrbk < v0.32.0)
+if you want to backup your root filesystem `/`.
+
+/etc/fstab:
+
+    /dev/sda1  /mnt/btr_pool  btrfs  subvolid=5,noatime  0 0
+
+Note that some default btrfs installations (e.g. Ubuntu) use subvolume
+names `@` for rootfs (mounted at `/`) and `@home` for `/home`, as a
+naming convention. If this is the case on your file system, replace
+the `subvolume` declarations in the examples accordingly.
+
+  [btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html
 
 
-Example: laptop with usb-disk for backups
------------------------------------------
+Example: Local Regular Snapshots (time-machine)
+-----------------------------------------------
 
-In this example, we assume you have a laptop with:
+The simplest use case is to only create snapshots of your data. This
+will obviously not protect it against hardware failure, but can be
+useful for:
 
-- a disk having a btrfs volume mounted as `/mnt/btr_pool`, containing
-  a subvolume `rootfs` for the root filesystem and a subvolume `home`
-  for the user data.
-- a backup disk having a btrfs volume mounted as `/mnt/btr_backup`,
-  containing a subvolume `mylaptop` for the incremental backups.
+  * protection against inadvertent changes or deletions
+  * keeping past states of copies from rsync or similar tools
 
-Retention policy:
-
-- keep snapshots for 14 days (very handy if you are on the road and
-  the backup disk is not attached)
-- keep monthly backups forever
-- keep weekly backups for 10 weeks
-- keep daily backups for 20 days
-
-/etc/btrbk/btrbk-mylaptop.conf:
-
-    snapshot_preserve_daily    14
-    snapshot_preserve_weekly   0
-    snapshot_preserve_monthly  0
-
-    target_preserve_daily      20
-    target_preserve_weekly     10
-    target_preserve_monthly    all
-
-    snapshot_dir               btrbk_snapshots
-
-    volume /mnt/btr_pool
-      subvolume rootfs
-        target send-receive    /mnt/btr_backup/mylaptop
-
-      subvolume home
-        target send-receive    /mnt/btr_backup/mylaptop
-
-
-/etc/cron.daily/btrbk:
-
-    #!/bin/bash
-    /usr/sbin/btrbk -c /etc/btrbk/btrbk-mylaptop.conf run
-
-
-- This will create snapshots on a daily basis:
-  - `/mnt/btr_pool/btrbk_snapshots/rootfs.YYYYMMDD`
-  - `/mnt/btr_pool/btrbk_snapshots/home.YYYYMMDD`
-- And create incremental backups in:
-  - `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD`
-  - `/mnt/btr_backup/mylaptop/home.YYYYMMDD`
-
-If you want the snapshots to be created only if the backup disk is
-attached, simply add the following line to the config:
-
-    snapshot_create            ondemand
-
-
-Example: host-initiated backup on fileserver
---------------------------------------------
-
-Let's say you have a fileserver at "myserver.mydomain.com" where you
-want to create backups of your laptop disk, the config would look like
-this:
-
-    ssh_identity               /etc/btrbk/ssh/id_rsa
-
-    volume /mnt/btr_pool
-      subvolume rootfs
-        target send-receive    /mnt/btr_backup/mylaptop
-        target send-receive    ssh://myserver.mydomain.com/mnt/btr_backup/mylaptop
-
-In addition to the backups on your local usb-disk mounted at
-`/mnt/btr_backup/mylaptop`, incremental backups would also be pushed
-to `myserver.mydomain.com`.
-
-
-Example: fileserver-initiated backups from several hosts
---------------------------------------------------------
-
-If you're a sysadmin and want to trigger backups directly from your
-fileserver, the config would be something like:
-
-    ssh_identity               /etc/btrbk/ssh/id_rsa
-
-    volume ssh://alpha.mydomain.com/mnt/btr_pool
-      subvolume rootfs
-        target send-receive    /mnt/btr_backup/alpha
-
-      subvolume home
-        target send-receive    /mnt/btr_backup/alpha
-
-    volume ssh://beta.mydomain.com/mnt/btr_pool
-      subvolume rootfs
-        target send-receive    /mnt/btr_backup/beta
-
-      subvolume dbdata
-        target send-receive    /mnt/btr_backup/beta
-
-This will pull backups from alpha/beta.mydomain.com and locally create:
-
-- `/mnt/btr_backup/alpha/rootfs.YYYYMMDD`
-- `/mnt/btr_backup/alpha/home.YYYYMMDD`
-- `/mnt/btr_backup/beta/rootfs.YYYYMMDD`
-- `/mnt/btr_backup/beta/dbdata.YYYYMMDD`
-
-
-Example: local time-machine (daily snapshots)
----------------------------------------------
-
-If all you want is to create snapshots of your home directory on a
-regular basis:
+Let's assume you need regular snapshots of your home directory, which
+is located in the subvolume `home` of the volume `/mnt/btr_pool`. The
+snapshots are to be stored in `btrbk_snapshots` (on the same volume).
 
 /etc/btrbk/btrbk.conf:
 
+    timestamp_format        long
+    snapshot_preserve_min   18h
+    snapshot_preserve       48h
+
     volume /mnt/btr_pool
       snapshot_dir btrbk_snapshots
       subvolume home
 
+Notice that the `target` option is not provided, and btrbk will only
+manage snapshots located on the same volume in `snapshot_dir`. Btrbk
+does not create subdirs by default, the snapshot directory must first
+be created manually:
+
+    # mkdir /mnt/btr_pool/btrbk_snapshots
+
+The "volume" section is merely used as a specifier for a base
+directory, and can be skipped if you prefer to configure everything
+using absolute paths. The above configuration can also be written as:
+
+    snapshot_dir /mnt/btr_pool/btrbk_snapshots
+    subvolume    /mnt/btr_pool/home
+
+If you don't want to mount the btrfs root filesystem to
+`/mnt/btr_pool`, you might as well configure it like this:
+
+    snapshot_dir /btrbk_snapshots
+    subvolume    /home
+
+Start a dry run (-n, --dry-run):
+
+    # btrbk run -n
+
+Create the first snapshot:
+
+    # btrbk run
+
+Print schedule (-S, --print-schedule):
+
+    # btrbk run -n -S
+
+If it works as expected, configure a cron job to run btrbk hourly:
+
+/etc/cron.hourly/btrbk:
+
+    #!/bin/sh
+    exec /usr/bin/btrbk -q run
+
+Snapshots will now be created every hour. All snapshots are preserved for at
+least 18 hours (`snapshot_preserve_min`), whether they are created by the cron
+job or manually by calling `sudo btrbk run` on the command line. Additionally,
+48 hourly snapshots are preserved (`snapshot_preserve`).
+
+
+Example: Backups to USB Disk
+----------------------------
+
+In this example, we assume you have a laptop with:
+
+  * a disk having a btrfs root subvolume (subvolid=5) mounted on
+    `/mnt/btr_pool`, containing a subvolume `rootfs` for the root
+    filesystem (i.e. mounted on `/`) and a subvolume `home` for the
+    user data,
+  * a directory or subvolume `/mnt/btr_pool/btrbk_snapshots` which
+    will hold the btrbk snapshots,
+  * a backup disk having a btrfs volume mounted as `/mnt/btr_backup`,
+    containing a subvolume or directory `mylaptop` for the incremental
+    backups.
+
+Retention policy:
+
+  * keep all snapshots for 2 days, no matter how frequently you (or
+    your cron job) run btrbk
+  * keep daily snapshots for 14 days (very handy if you are on
+    the road and the backup disk is not attached)
+  * keep monthly backups forever
+  * keep weekly backups for 10 weeks
+  * keep daily backups for 20 days
+
+/etc/btrbk/btrbk-mylaptop.conf:
+
+    snapshot_preserve_min   2d
+    snapshot_preserve      14d
+
+    # Create snapshots only if the backup disk is attached
+    #snapshot_create ondemand
+
+    target_preserve_min    no
+    target_preserve        20d 10w *m
+
+    snapshot_dir           btrbk_snapshots
+
+    volume /mnt/btr_pool
+      target /mnt/btr_backup/mylaptop
+      subvolume rootfs
+      subvolume home
+      [...]
+
+
 /etc/cron.daily/btrbk:
 
-    #!/bin/bash
-    /usr/sbin/btrbk run
-
-Note that you can run btrbk more than once a day, e.g. by creating the
-above script in `/etc/cron.hourly/btrbk`, or by calling `sudo btrbk
-run` from the command line.
+    #!/bin/sh
+    exec /usr/bin/btrbk -q -c /etc/btrbk/btrbk-mylaptop.conf run
 
 
-Example: multiple btrbk instances
+  * This will create snapshots on a daily basis:
+    * `/mnt/btr_pool/btrbk_snapshots/rootfs.YYYYMMDD`
+    * `/mnt/btr_pool/btrbk_snapshots/home.YYYYMMDD`
+  * And create incremental backups in:
+    * `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD`
+    * `/mnt/btr_backup/mylaptop/home.YYYYMMDD`
+
+If you prefer triggering the backups manually, change the cron command
+to run the `snapshot` action instead of `run`. Start the backups
+manually by running:
+
+    # btrbk resume
+
+For a quick additional snapshot of your home, run:
+
+    # btrbk snapshot home
+
+
+Example: Host-initiated Backup on Fileserver
+--------------------------------------------
+
+Let's say you have a fileserver at "myserver.example.org" where you
+want to create backups of your laptop disk. The config could look like
+this:
+
+    ssh_identity /etc/btrbk/ssh/id_rsa
+
+    volume /mnt/btr_pool
+      subvolume rootfs
+        target /mnt/btr_backup/mylaptop
+        target ssh://myserver.example.org/mnt/btr_backup/mylaptop
+
+In addition to the backups on your local usb-disk mounted at
+`/mnt/btr_backup/mylaptop`, incremental backups would also be pushed
+to `myserver.example.org`.
+
+
+Example: Fileserver-initiated Backups from Several Hosts
+--------------------------------------------------------
+
+If you're a sysadmin and want to trigger backups directly from your
+fileserver, the config would be something like:
+
+    ssh_identity /etc/btrbk/ssh/id_rsa
+
+    volume ssh://alpha.example.org/mnt/btr_pool
+      target /mnt/btr_backup/alpha
+      subvolume rootfs
+      subvolume home
+
+    volume ssh://beta.example.org/mnt/btr_pool
+      target /mnt/btr_backup/beta
+      subvolume rootfs
+      subvolume dbdata
+
+This will pull backups from alpha/beta.example.org and locally
+create:
+
+  * `/mnt/btr_backup/alpha/rootfs.YYYYMMDD`
+  * `/mnt/btr_backup/alpha/home.YYYYMMDD`
+  * `/mnt/btr_backup/beta/rootfs.YYYYMMDD`
+  * `/mnt/btr_backup/beta/dbdata.YYYYMMDD`
+
+
+Example: Multiple Btrbk Instances
 ---------------------------------
 
 Let's say we have a host (at 192.168.0.42) running btrbk with the
@@ -243,119 +302,389 @@ to only fetch the snapshots.
 
 /etc/btrbk/btrbk.conf (on backup server):
 
+    target_preserve_min        no
+    target_preserve            0d 10w *m
+
     volume ssh://192.168.0.42/mnt/btr_pool
+      target /mnt/btr_backup/my-laptop
       subvolume home
-        snapshot_dir             btrbk_snapshots
-        snapshot_preserve_daily  all
-        snapshot_create          no
-        resume_missing           yes
+        snapshot_dir           btrbk_snapshots
+        snapshot_preserve_min  all
+        snapshot_create        no
 
-        target_preserve_daily    0
-        target_preserve_weekly   10
-        target_preserve_monthly  all
+If the server runs btrbk with this config, 10 weeklies and all
+monthlies are received from 192.168.0.42. The source filesystem is
+never altered because of `snapshot_preserve_min all`.
 
-        target send-receive  /mnt/btr_backup/my-laptop.com
 
-If the server runs btrbk with this config, the latest snapshot (which
-is *always* transferred), 10 weeklies and all monthlies are received
-from 192.168.0.42. The source filesystem is never altered because of
-`snapshot_preserve_daily all`.
+Example: Virtual Machine Setup
+------------------------------
+
+Common virtual machine setups have multiple volume sections with same
+host, but distinct port numbers for each machine.
+
+/etc/btrbk/btrbk.conf:
+
+    # This propagates to all subvolume sections:
+    target /mnt/btr_backup/
+
+    volume ssh://localhost:2201/mnt/btr_pool
+      group vm vm01
+      subvolume home
+        snapshot_name vm01-home
+      subvolume data
+        snapshot_name vm01-data
+
+    volume ssh://localhost:2202/mnt/btr_pool
+      group vm vm02
+      subvolume home
+        snapshot_name vm02-home
+
+    volume ssh://localhost:2203/mnt/btr_pool
+      [...]
+
+This will create `/mnt/btr_backup/vm[NN]-home`, `vm[NN]-data`, ...
+
+Note that btrbk holds a single reference to every btrfs filesystem
+tree, regarding UUID's as "globally unique". If the configured
+subvolumes point to the same filesystem on different machines (ports),
+you will see log lines like this when running `btrbk -v`:
+
+```
+Assuming same filesystem: "ssh://localhost:2201/dev/sda1", "ssh://localhost:2202/dev/sda1"
+```
+
+
+Example: Backup from non-btrfs Source
+-------------------------------------
+
+If you want to make backups from a filesystem other than btrfs
+(e.g. ext4 or reiserfs), you need to create a *synchronization
+subvolume* on the backup disk:
+
+    # btrfs subvolume create /mnt/btr_backup/myhost_sync
+
+Configure btrbk to use `myhost_sync` as source subvolume:
+
+    volume /mnt/btr_backup
+      subvolume myhost_sync
+        snapshot_name           myhost
+
+        snapshot_preserve_min   latest
+        snapshot_preserve       14d 20w *m
+
+The btrbk package provides the "btrbk-mail" script, which automates
+the synchronization using rsync, and can be run as cron job or systemd
+timer unit. For configuration details, see the config section in
+"/contrib/cron/btrbk-mail".
+
+Alternatively, you can run any synchronization software prior to
+running btrbk. Something like:
+
+    #!/bin/sh
+    rsync -az --delete \
+          --inplace --numeric-ids --acls --xattrs \
+          -e 'ssh -i /etc/btrbk/ssh/id_rsa' \
+          myhost.example.org:/data/ \
+          /mnt/btr_backup/myhost_sync/
+
+    exec /usr/bin/btrbk -q run
+
+This will produce snapshots `/mnt/btr_backup/myhost.20150101`, with
+retention as defined with the snapshot_preserve option.
+
+
+Example: Encrypted Backup to non-btrfs Target
+---------------------------------------------
+
+If your backup server does not support btrfs, you can send your
+subvolumes to a raw file.
+
+This is an _experimental_ feature: btrbk supports "raw" targets,
+meaning that similar to the "send-receive" target the btrfs subvolume
+is being sent using `btrfs send` (mirroring filesystem level data),
+but instead of instantly being received (`btrfs receive`) by the
+target filesystem, it is being redirected to a file, optionally
+compressed and piped through GnuPG.
+
+/etc/btrbk/btrbk.conf:
+
+    raw_target_compress   xz
+    raw_target_encrypt    gpg
+    gpg_keyring           /etc/btrbk/gpg/pubring.gpg
+    gpg_recipient         btrbk@example.org
+
+    volume /mnt/btr_pool
+      subvolume home
+        target raw ssh://cloud.example.com/backup
+          ssh_user  btrbk
+          # incremental  no
+
+This will create a GnuPG encrypted, compressed files on the target
+host. For each backup, two files are created:
+
+  * `/backup/home.YYYYMMDD.btrfs.xz.gpg`: main data file containing
+    the btrfs send-stream,
+  * `/backup/home.YYYYMMDD.btrfs.xz.gpg.info`: sidecar file containing
+    metadata used by btrbk.
+
+If you are using raw _incremental_ backups, please make sure you
+understand the implications (see [btrbk.conf(5)], TARGET TYPES).
 
 
 Setting up SSH
 ==============
 
-Since btrbk needs root access on the remote side, it is *very
-advisable* to take all the security precautions you can. Usually
-backups are generated periodically without user interaction, so it is
-not possible to protect your ssh key with a password. The steps below
-will give you hints on how to secure your ssh server for a backup
-scenario.
+Since btrbk needs root access, it is *very advisable* to take all the
+security precautions you can. In most cases backups are generated
+periodically without user interaction, so it is not possible to
+protect your ssh key with a password. The steps below will give you
+hints on how to secure your ssh server for a backup scenario. Note
+that the btrbk package is not required on the remote side, but you
+will need the `btrfs` executable from the [btrfs-progs] package.
 
-btrbk provides a little shell script called "ssh_filter_btrbk.sh",
-which only allows sane calls to the /sbin/btrfs command needed for
-snapshot creation and send/receive operations. This is how it is used
-with ssh:
 
-**Step 1** (client): Create a ssh key dedicated to btrbk, without password protection:
+### Create SSH Key Pair
 
-    ssh-keygen -t rsa -b 2048 -f /etc/btrbk/ssh/id_rsa -C btrbk@mydomain.com -N ""
+On the client side, create a ssh key dedicated to btrbk, without
+password protection:
 
-**Step 2** (server): Copy the "ssh_filter_btrbk.sh" from the btrbk project to "/root/".
+    # ssh-keygen -t rsa -b 4096 -f /etc/btrbk/ssh/id_rsa -C btrbk@example.org -N ""
 
-**Step 3** (server): Add contents of the public key
-  (/etc/btrbk/ssh/id_rsa.pub) to "/root/.ssh/authorized_keys",
-  restricting access from a single host:
+The content of the public key (/etc/btrbk/ssh/id_rsa.pub) is used for
+authentication in "authorized_keys" on the server side (see [sshd(8)]
+for details).
 
-    from="192.168.0.42",command="/root/ssh_filter_btrbk.sh" ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@mydomain.com
 
-Now your ssh server allows connections only from 192.168.0.42, and
-will only execute commands needed by btrbk. Note that the btrbk
-executable is not needed on the remote side, but you will need
-"/sbin/btrfs" from the btrfs-progs package.
+### Allow Root Login
+
+The most straight forward setup is to allow root login on the remote
+host. If this is not an option for you, refer to the more complex
+"Dedicated Btrbk User Login" section below.
+
+/etc/ssh/sshd_config:
+
+    PermitRootLogin prohibit-password
+
+Add your btrbk public key to "/root/.ssh/authorized_keys" on the
+server, and you are good to go.
+
+
+### Restrict Access
+
+Restrict ssh access to a static IP address within your network. On the
+remote host, either add a "Match" block in:
+
+/etc/ssh/sshd_config:
+
+    Match Address 192.168.0.42
+
+Or restrict in authorized_keys:
+
+    from="192.168.0.42" <pubkey>...
+
+Consult the [sshd_config(5)] man-page for a detailed explanation and
+more options.
+
+
+Dedicated Btrbk User Login (optional)
+-------------------------------------
+
+If allowing root login is not an option for you, there are several
+ways to restrict SSH access to a regular user.
+
+
+### Option 1: Use sudo
+
+On the client side, configure btrbk use the sudo backend. This changes
+the ssh calls to btrfs commands to `sudo btrfs <subcommand>
+<options>`.
+
+/etc/btrbk/btrbk.conf:
+
+    backend_remote btrfs-progs-sudo
+
+On the remote host, grant root permissions for the "btrfs" command
+groups (subcommands) in "/etc/sudoers". If you are using
+[ssh_filter_btrbk(1)], also add the `ssh_filter_btrbk.sh --sudo`
+option in "authorized_keys" (see below).
+
+
+### Option 2: Use btrfs-progs-btrbk
+
+Instead of using the all-inclusive `btrfs` command,
+"btrfs-progs-btrbk" allows you to restrict privileges to its
+subcommands using linux capabilities(7) or setuid.
+
+Note that the "btrfs-progs-btrbk" package is not available on all
+linux distributions, you might need to build and install it on your
+own (refer to [btrfs-progs-btrbk] on GitHub for more details).
+
+/etc/btrbk/btrbk.conf:
+
+    backend_remote btrfs-progs-btrbk
+
+Make sure that only the required binaries with elevated privileges can
+be called by the btrbk user. For example, on a server acting as "btrbk
+source", allow only the following binaries for the "btrbk" group:
+
+    # getcap /usr/bin/btrfs-*
+    /usr/bin/btrfs-send cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-delete cap_dac_override,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-list cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-show cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    /usr/bin/btrfs-subvolume-snapshot cap_dac_override,cap_dac_read_search,cap_fowner,cap_sys_admin=ep
+    
+    # ls -l /usr/bin/btrfs-*
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-send
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-delete
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-list
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-show
+    -rwx--x--- 1 root btrbk  /usr/bin/btrfs-subvolume-snapshot
+
+
+Restrict Commands with "ssh_filter_btrbk.sh" (optional)
+-------------------------------------------------------
+
+Btrbk comes with a shell script "ssh_filter_btrbk.sh", which restricts
+ssh access to sane calls to the "btrfs" command needed for snapshot
+creation and send/receive operations (see [ssh_filter_btrbk(1)]).
+
+Copy "ssh_filter_btrbk.sh" to "/backup/scripts/", and configure sshd
+to run it whenever the key is used for authentication. Example
+"/root/.ssh/authorized_keys":
+
+    # example backup source (also allowing deletion of old snapshots)
+    command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete",restrict <pubkey>...
+
+    # example backup target (also allowing deletion of old snapshots)
+    command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete",restrict <pubkey>...
+
+    # example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no),
+    # restricted to subvolumes within /home or /data
+    command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data",restrict <pubkey>...
+
+
+  [ssh_filter_btrbk(1)]: https://digint.ch/btrbk/doc/ssh_filter_btrbk.1.html
+  [sshd(8)]: https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man8/sshd.8
+  [sshd_config(5)]: https://man.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man5/sshd_config
+  [btrfs-progs-btrbk]: https://github.com/digint/btrfs-progs-btrbk
 
 
 Restoring Backups
 =================
 
-btrbk does not provide any mechanism to restore your backups, this has
-to be done manually. In the examples below, we assume that you have a
-btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you want to
-have restored is at `/mnt/btr_pool/data`.
+Btrbk does not provide any mechanism to restore your backups, this has
+to be done manually. In the instructions below, we assume that you
+have a btrfs volume mounted at `/mnt/btr_pool`, and the subvolume you
+want to restore is at `/mnt/btr_pool/data`.
 
-Example: Restore a Snapshot
------------------------------
+**Important**: don't use `btrfs property set` to make a subvolume
+read-write after restoring. This is a low-level command, and leaves
+"Received UUID" in a false state which causes btrbk to fail on
+subsequent incremental backups. Instead, use `btrfs subvolume
+snapshot` (without `-r` flag) as described below.
 
-First, pick a snapshot to be restored:
 
-    btrbk tree
+### Step 0: Identify Subvolume
 
-From the list, pick the snapshot you want to restore. Let's say it's
+    # list snapshots managed by btrbk
+    btrbk list snapshots
+
+    # alternative: list all subvolumes
+    btrbk ls /
+    btrbk ls -L /
+
+From the list, identify the snapshot you want to restore. Let's say it's
 `/mnt/btr_pool/_btrbk_snap/data.20150101`.
 
-If the broken subvolume is still present, move it away:
 
-    mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
-
-Now restore the snapshot:
-
-    btrfs subvolume snapshot /mnt/btr_pool/_btrbk_snap/data.20150101 /mnt/btr_pool/data
-
-That's it; your `data` subvolume is restored. If everything went fine,
-it's time to nuke the broken subvolume:
-
-    btrfs subvolume delete /mnt/btr_pool/data.BROKEN
-
-
-Example: Restore a Backup
--------------------------
-
-First, pick a backup to be restored:
-
-    btrbk tree
-
-From the list, pick the backup you want to restore. Let's say it's
-`/mnt/btr_backup/data.20150101`.
-
-If the broken subvolume is still present, move it away:
-
-    mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
-
-Now restore the backup:
+### Step 1: Restore Backup
+(skip this step if you restore from a snapshot)
 
+    # locally mounted backup disk
     btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
+
+    # from / to remote host
+    ssh root@remote btrfs send /mnt/btr_backup/data.20150101 | btrfs receive /mnt/btr_pool/
+    btrfs send /mnt/btr_backup/data.20150101 | ssh root@remote btrfs receive /mnt/btr_pool/
+
+**Hint**: Try to send-receive backups incrementally if possible. In
+case you still have common snapshot / backup pairs (i.e. both
+"snapshot_subvol" and "target_subvol" are listed above), use `btrfs
+send -p <parent>`.
+
+From this point on, `data.20150101` on both disks can be used as
+parents for subsequent send-receive operations, and a *received_uuid*
+relationship is established (see below).
+
+
+### Step 2: Create read-write Subvolume
+
+    # if still present, move broken subvolume away
+    mv /mnt/btr_pool/data /mnt/btr_pool/data.BROKEN
+
+    # create read-write subvolume
     btrfs subvolume snapshot /mnt/btr_pool/data.20150101 /mnt/btr_pool/data
-    btrfs subvolume delete /mnt/btr_pool/data.20150101
 
-Alternatively, if you're restoring data on a remote host, do something
-like this:
+Your `data` subvolume is restored, you can carry on with incremental
+backups to `/mnt/btr_backup`.
 
-    btrfs send /mnt/btr_backup/data.20150101 | ssh root@my-remote-host.com btrfs receive /mnt/btr_pool/
 
-If everything went fine, nuke the broken subvolume:
+### Step 3: Cleanup
 
+    # if everything went fine, delete the broken subvolume
     btrfs subvolume delete /mnt/btr_pool/data.BROKEN
 
+Make sure to keep `data.20150101` subvolumes on both disks at least
+until you created a new backup using btrbk, in order to keep the
+incremental chain alive.
+
+
+Btrfs Relationship (technical note)
+-----------------------------------
+
+    btrbk origin -t /mnt/btr_backup/data.20150101
+    btrbk ls -L /mnt/btr_pool /mnt/btr_backup
+
+  * **received_uuid** relationship: *correlated*, *identical*
+    read-only subvolumes, cross-filesystem.
+
+        a.received_uuid = b.received_uuid
+        a.received_uuid = b.uuid
+
+    * Required for subvolumes used as parent (or clone-src) of
+      send-receive operations.
+    * Present on subvolumes created by `btrfs send | btrfs receive`.
+    * `/mnt/btr_pool/data.20150101 === /mnt/btr_backup/data.20150101`
+
+  * **parent_uuid** relationship: "is-snapshot-of"
+
+        a.parent_uuid = b.uuid
+
+    * Present on subvolumes created by `btrfs subvolume snapshot` or
+      `btrfs send -p | btrfs receive`.
+    * Used by btrbk to determine best parent.
+    * `/mnt/btr_pool/data.20150101 <-- /mnt/btr_pool/data`
+
+
+FAQ
+===
+
+Make sure to also read the [btrbk FAQ page](doc/FAQ.md).
+Help improve it by asking!
+
+
+Donate
+======
+
+So btrbk saved your day?
+
+I will definitively continue to develop btrbk for free. If you want to
+support my hard work with a donation, you are welcome to do so!
+
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S)
+
 
 Development
 ===========
@@ -363,10 +692,15 @@ Development
 Source Code Repository
 ----------------------
 
-The source code for btrbk is managed using Git. Check out the source
-like this:
+The source code for btrbk is managed using Git.
 
-    git clone git://dev.tty0.ch/btrbk.git
+Official repository:
+
+    git clone https://dev.tty0.ch/btrbk.git
+
+Mirror on GitHub:
+
+    git clone https://github.com/digint/btrbk.git
 
 
 How to Contribute
@@ -376,23 +710,23 @@ Your contributions are welcome!
 
 If you would like to contribute or have found bugs:
 
-- Visit the [btrbk project page on GitHub] and use the [issues
-  tracker] there.
-- Talk to us on Freenode in `#btrbk`.
-- Contact the author via email (the email address can be found in the
-  sources).
+  * Visit the [btrbk project page on GitHub] and use the
+    [issues tracker] there.
+  * Talk to us on [Libera.Chat] in `#btrbk`.
+  * Contact the author via email (the email address can be found in
+    the sources).
 
 Any feedback is appreciated!
 
-  [btrbk project page on GitHub]: http://github.com/digint/btrbk
-  [issues tracker]: http://github.com/digint/btrbk/issues
+  [btrbk project page on GitHub]: https://github.com/digint/btrbk
+  [issues tracker]: https://github.com/digint/btrbk/issues
+  [Libera.Chat]: https://libera.chat
 
 
 License
 =======
 
 btrbk is free software, available under the [GNU General Public
-License, Version 3][GPLv3].
-
-  [GPLv3]: http://www.gnu.org/licenses/gpl.html
+License, Version 3 or later][GPL-3.0-or-later].
 
+  [GPL-3.0-or-later]: https://www.gnu.org/licenses/gpl.html

btrbk.conf.example

@@ -2,13 +2,33 @@
 # Example btrbk configuration file
 #
 #
-# Please refer to the btrbk.conf(5) man-page for more details.
+# Please refer to the btrbk.conf(5) man-page for a complete
+# description of all configuration options.
+# For more examples, see README.md included with this package.
 #
-# Note that the options can be overridden in the
-# volume/subvolume/target sections.
+#   btrbk.conf(5): <https://digint.ch/btrbk/doc/btrbk.conf.5.html>
+#   README.md:     <https://digint.ch/btrbk/doc/readme.html>
+#
+# Note that the options can be overridden per volume/subvolume/target
+# in the corresponding sections.
 #
 
 
+# Enable transaction log
+transaction_log            /var/log/btrbk.log
+
+# Specify SSH private key for remote connections
+ssh_identity               /etc/btrbk/ssh/id_ed25519
+ssh_user                   root
+
+# Use sudo if btrbk or lsbtr is run by regular user
+backend_local_user         btrfs-progs-sudo
+
+# Enable stream buffer. Adding a buffer between the sending and
+# receiving side is generally a good idea.
+# NOTE: If enabled, make sure to install the "mbuffer" package!
+stream_buffer              256m
+
 # Directory in which the btrfs snapshots are created. Relative to
 # <volume-directory> of the volume section.
 # If not set, the snapshots are created in <volume-directory>.
@@ -16,7 +36,7 @@
 # If you want to set a custom name for the snapshot (and backups),
 # use the "snapshot_name" option within the subvolume section.
 #
-# NOTE: btrbk does not autmatically create this directory, and the
+# NOTE: btrbk does not automatically create this directory, and the
 # snapshot creation will fail if it is not present.
 #
 snapshot_dir               _btrbk_snap
@@ -30,116 +50,155 @@ snapshot_dir               _btrbk_snap
 # creation of non-incremental backups if no parent is found).
 #incremental                yes
 
-# Resume missing backups (set to "no" if you don't want to resume
-# missing backups).
-#resume_missing             yes
+# Specify after what time (in full hours after midnight) backups/
+# snapshots are considered as a daily backup/snapshot
+#preserve_hour_of_day       0
 
 # Specify on which day of week weekly/monthly backups are to be
 # preserved.
 #preserve_day_of_week       sunday
 
-# Preserve matrix for source snapshots:
-#snapshot_preserve_daily    all
-#snapshot_preserve_weekly   0
-#snapshot_preserve_monthly  all
+# Preserve all snapshots for a minimum period of time.
+#snapshot_preserve_min      1d
 
-# Preserve matrix for backup targets:
-#target_preserve_daily      all
-#target_preserve_weekly     0
-#target_preserve_monthly    all
+# Retention policy for the source snapshots.
+#snapshot_preserve          <NN>h <NN>d <NN>w <NN>m <NN>y
 
-# Specify SSH private key for "ssh://" volumes / targets:
-#ssh_identity               /etc/btrbk/ssh/id_ed25519
-#ssh_user                   root
+# Preserve all backup targets for a minimum period of time.
+#target_preserve_min        no
 
-# Don't wait for transaction commit on deletion. Set this to "after"
-# or "each" to make sure the deletion of subvolumes is committed to
-# disk when btrbk terminates.
-#btrfs_commit_delete        no
+# Retention policy for backup targets:
+#target_preserve            <NN>h <NN>d <NN>w <NN>m <NN>y
 
-# Set this to "yes" to enable btrfs-progs < 3.17 compatibility.
-# Set this either globally or in a specific "target" section.
-#btrfs_progs_compat         no
+# Retention policy for archives ("btrbk archive" command):
+#archive_preserve_min       no
+#archive_preserve           <NN>h <NN>d <NN>w <NN>m <NN>y
+
+# Enable compression for remote btrfs send/receive operations:
+#stream_compress            no
+#stream_compress_level      default
+#stream_compress_threads    default
+
+# Enable lock file support: Ensures that only one instance of btrbk
+# can be run at a time.
+#lockfile                   /var/lock/btrbk.lock
+
+# Don't wait for transaction commit on deletion. Enable this to make
+# sure the deletion of subvolumes is committed to disk when btrbk
+# terminates.
+#btrfs_commit_delete no
 
 
 #
-# Volume section: "volume <volume-directory>"
+# Volume section (optional): "volume <volume-directory>"
 #
-#   <volume-directory>  Directory of a btrfs volume (or subvolume)
-#                       containing the subvolume to be backuped
+#   <volume-directory>  Base path within a btrfs filesystem
+#                       containing the subvolumes to be backuped
 #                       (usually the mount-point of a btrfs filesystem
-#                       mounted with subvolid=0 option)
+#                       mounted with subvolid=5 option).
 #
-# Subvolume section: "subvolume <subvolume-name>
+# Subvolume section: "subvolume <subvolume-name>"
 #
 #   <subvolume-name>    Subvolume to be backuped, relative to
 #                       <volume-directory> in volume section.
 #
 # Target section: "target <type> <volume-directory>"
 #
-#   <type>              Backup type, currently only "send-receive".
-#   <volume-directory>  Directory of a btrfs volume (or subvolume)
+#   <type>              (optional) type, defaults to "send-receive".
+#   <volume-directory>  Directory within a btrfs filesystem
 #                       receiving the backups.
 #
 # NOTE: The parser does not care about indentation, this is only for
-# human readability. The options always apply to the last section
+# human readability. All options apply to the last section
 # encountered, overriding the corresponding option of the upper
-# section. This means that the global options must be set before any
-# "volume" section.
+# section. This means that the global options must be set on top,
+# before any "volume", "subvolume" or "target section.
 #
 
 
 #
-# Example configuration:
+# Example retention policy:
 #
-snapshot_preserve_daily    14
-snapshot_preserve_weekly   0
-snapshot_preserve_monthly  0
+snapshot_preserve_min   2d
+snapshot_preserve       14d
 
-target_preserve_daily      20
-target_preserve_weekly     10
-target_preserve_monthly    all
+target_preserve_min     no
+target_preserve         20d 10w *m
 
 
+#
+# Simple setup: Backup root and home to external disk
+#
+snapshot_dir /btrbk_snapshots
+target       /mnt/btr_backup
+subvolume    /
+subvolume    /home
+
+
+#
+# Complex setup
+#
+# In order to keep things organized, it is recommended to use "volume"
+# sections and mount the top-level subvolume (subvolid=5):
+#
+#  $ mount -o subvolid=5 /dev/sda1 /mnt/btr_pool
+#
 # Backup to external disk mounted on /mnt/btr_backup
 volume /mnt/btr_pool
-  # no action if external disk is not attached
-  snapshot_create  ondemand
+  # Create snapshots in /mnt/btr_pool/btrbk_snapshots
+  snapshot_dir btrbk_snapshots
 
-  subvolume  root_gentoo
-    target send-receive      /mnt/btr_backup/_btrbk
+  # Target for all subvolume sections:
+  target /mnt/btr_backup
 
-  subvolume  kvm
-    # use different preserve matrix for kvm backups
-    target_preserve_daily    7
-    target_preserve_weekly   4
-
-    target send-receive      /mnt/btr_backup/_btrbk
+  # Some default btrfs installations (e.g. Ubuntu) use "@" for rootfs
+  # (mounted at "/") and "@home" (mounted at "/home"). Note that this
+  # is only a naming convention.
+  #subvolume @
+  subvolume root
+  subvolume home
+  subvolume kvm
+    # Use different retention policy for kvm backups:
+    target_preserve 7d 4w
 
 
-# Backup to external disk as well as some remote host
+# Backup data to external disk as well as remote host
 volume /mnt/btr_data
-  subvolume  home
-    # always create snapshot, even if targets are unreachable
-    snapshot_create  always
-    target send-receive      /mnt/btr_backup/_btrbk
-    target send-receive      ssh://backup.my-remote-host.com/mnt/btr_backup
+  subvolume  data
+    # Always create snapshot, even if targets are unreachable
+    snapshot_create always
+    target /mnt/btr_backup
+    target ssh://backup.my-remote-host.com/mnt/btr_backup
 
 
 # Backup from remote host, with different naming
 volume ssh://my-remote-host.com/mnt/btr_pool
   subvolume data_0
-    snapshot_dir             snapshots/btrbk
-    snapshot_name            data_main
-    target send-receive      /mnt/btr_backup/_btrbk/my-remote-host.com
+    snapshot_dir   snapshots/btrbk
+    snapshot_name  data_main
+    target /mnt/btr_backup/my-remote-host.com
+
+
+# Backup on demand (noauto) to remote host running busybox, login as
+# regular user using ssh-agent with current user name (ssh_user no)
+# and default credentials (ssh_identity no).
+volume /home
+  noauto  yes
+  compat  busybox
+  backend_remote  btrfs-progs-sudo
+  ssh_user      no
+  ssh_identity  no
+
+  target ssh://my-user-host.com/mnt/btr_backup/home
+  subvolume  alice
+  subvolume  bob
 
 
 # Resume backups from remote host which runs its own btrbk instance
 # creating snapshots for "home" in "/mnt/btr_pool/btrbk_snapshots".
 volume ssh://my-remote-host.com/mnt/btr_pool
+  snapshot_dir           btrbk_snapshots
+  snapshot_create        no
+  snapshot_preserve_min  all
   subvolume home
-    snapshot_dir             btrbk_snapshots
-    snapshot_preserve_daily  all
-    snapshot_create          no
-    resume_missing           yes
-    target send-receive      /mnt/btr_backup/_btrbk/my-remote-host.com
+    target /mnt/btr_backup/my-remote-host.com

contrib/bash/completion.bash

@@ -0,0 +1,133 @@
+_btrbk_init_cmds()
+{
+  # set $cmds to an array of the commands so far
+  #
+  # for example, for this command:
+  #
+  #     btrbk -v --override warn_unknown_targets=yes list config --long
+  #
+  # then $cmds is:
+  #
+  #     cmds=(list config)
+  #
+  cmds=()
+
+  local i
+  for ((i = 1; i < cword; i++)); do
+    case "${words[i-1]}" in
+      '-c' | '--config' | '--exclude' | '-l' | '--loglevel' | '--format' | '--lockfile' | '--override')
+        continue
+        ;;
+    esac
+    [[ ${words[i]} != -* ]] && cmds+=(${words[i]})
+  done
+
+  return 0
+}
+
+_btrbk()
+{
+  local cur prev words cword split cmds
+  _init_completion -s || return
+  _btrbk_init_cmds || return
+
+  case "$prev" in
+    '-c' | '--config')
+      _filedir
+      return
+      ;;
+    '--exclude')
+      return
+      ;;
+    '-l' | '--loglevel')
+      COMPREPLY=($(compgen -W 'error warn info debug trace' -- "$cur"))
+      return
+      ;;
+    '--format')
+      COMPREPLY=($(compgen -W 'table long raw' -- "$cur"))
+      return
+      ;;
+    '--lockfile')
+      _filedir
+      return
+      ;;
+    '--override')
+      return
+      ;;
+  esac
+
+  $split && return
+
+  if [[ $cur == -* ]]; then
+    COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur"))
+    [[ $COMPREPLY == *= ]] && compopt -o nospace
+  else
+    if [[ ! -v 'cmds[0]' ]]; then
+      COMPREPLY=($(compgen -W 'run dryrun snapshot resume prune archive clean stats list usage origin diff extents ls' -- "$cur"))
+    fi
+  fi
+
+  case "${cmds[0]}" in
+    'archive')
+      # <source>
+      if [[ ! -v 'cmds[1]' ]]; then
+        _filedir -d
+      # <target>
+      elif [[ ! -v 'cmds[2]' ]]; then
+        _filedir -d
+      # [--raw]
+      elif [[ $cur == -* ]]; then
+        COMPREPLY+=($(compgen -W '--raw' -- "$cur"))
+      fi
+      ;;
+    'list')
+      if [[ ! -v 'cmds[1]' ]]; then
+        COMPREPLY=($(compgen -W 'all snapshots backups latest config source volume target' -- "$cur"))
+      fi
+      ;;
+    'origin')
+      # <subvolume>
+      if [[ ! -v 'cmds[1]' ]]; then
+        _filedir -d
+      fi
+      ;;
+    'ls')
+      # <path>|<url>...
+      _filedir -d
+      ;;
+    'extents')
+      # [diff] <path>... [exclusive] <path>...
+      if [[ ! -v 'cmds[1]' ]]; then
+          COMPREPLY+=($(compgen -W 'diff' -- "$cur"))
+      elif [[ ! ${cmds[*]} =~ (^|[[:space:]])"exclusive"($|[[:space:]]) ]]; then
+          COMPREPLY+=($(compgen -W 'exclusive' -- "$cur"))
+      fi
+      _filedir -d
+      ;;
+  esac
+} && complete -F _btrbk btrbk
+
+_lsbtr()
+{
+  local cur prev words cword split
+  _init_completion -s || return
+
+  case "$prev" in
+    '-c' | '--config')
+      _filedir
+      ;;
+    '--override')
+      ;;
+  esac
+  $split && return
+
+  if [[ $cur == -* ]]; then
+    COMPREPLY=($(compgen -W '$(_parse_help "$1")' -- "$cur"))
+    [[ $COMPREPLY == *= ]] && compopt -o nospace
+  else
+    # <path>|<url>...
+    _filedir -d
+  fi
+} && complete -F _lsbtr lsbtr
+
+# ex: filetype=bash

contrib/cron/btrbk-mail

@@ -0,0 +1,278 @@
+#!/bin/bash
+
+## Wrapper script running "btrbk" and sending email with results
+
+now=$(date +%Y%m%d)
+
+declare -A rsync_src rsync_dst rsync_log rsync_rsh rsync_opt
+declare -A sync_fs_onchange
+
+#####  start config section  #####
+
+# Email recipients, separated by whitespace:
+mailto=${MAILTO:-root}
+
+# Email subject:
+mail_subject_prefix="btrbk <${HOSTNAME:-localhost}>"
+
+# Add summary and/or detail (rsync/btrbk command output) to mail body.
+# If both are not set, a mail is only sent on errors.
+mail_summary=yes
+mail_detail=no
+
+# List of mountpoints to be mounted/unmounted (whitespace-separated)
+# mount_targets="/mnt/btr_pool /mnt/backup"
+mount_targets=
+
+# rsync declarations (repeat complete block for more declarations):
+rsync_src[example_data]="user@example.com:/data/"
+rsync_dst[example_data]="/mnt/backup/example.com/data/"
+rsync_log[example_data]="/mnt/backup/example.com/data-${now}.log"
+rsync_rsh[example_data]="ssh -i /mnt/backup/ssh_keys/id_rsa"
+rsync_opt[example_data]="-az --delete --inplace --numeric-ids --acls --xattrs"
+# If set, add "rsync_dst" to "sync_fs" (see below) if rsync reports files transferred
+#sync_fs_onchange[example_data]=yes
+
+# Enable all rsync declarations (all indices of rsync_src array)
+#rsync_enable=${!rsync_src[@]}
+# Explicitly enable rsync declarations (whitespace-separated list)
+#rsync_enable="example_data"
+rsync_enable=
+
+# If set, do not run btrbk if rsync reports no changes.
+# If set to "quiet", do not send mail.
+#skip_btrbk_if_unchanged=quiet
+
+# Array of directories to sync(1) prior to running btrbk. This is
+# useful for source subvolumes having "snapshot_create ondemand"
+# configured in btrbk.conf.
+#sync_fs=("/mnt/btr_data" "/mnt/btr_pool")
+
+# btrbk command / options:
+btrbk_command="run"
+btrbk_opts="-c /etc/btrbk/btrbk.conf"
+
+
+### Layout options:
+
+# Prefix command output: useful when using mail clients displaying
+# btrbk summary lines starting with ">>>" as quotations.
+#mail_cmd_block_prefix='\\u200B' # zero-width whitespace
+#mail_cmd_block_prefix=". "
+
+# Newline character
+BR=$'\n'
+
+#####  end config section  #####
+
+
+check_options()
+{
+    [[ -n "$btrbk_command" ]] || die "btrbk_command is not set"
+    for key in $rsync_enable; do
+        [[ -n "${rsync_src[$key]}" ]] || die "rsync_src is not set for \"$key\""
+        [[ -n "${rsync_dst[$key]}" ]] || die "rsync_dst is not set for \"$key\""
+        [[ -n "${rsync_opt[$key]}" ]] || die "rsync_opt is not set for \"$key\""
+    done
+}
+
+send_mail()
+{
+    # assemble mail subject
+    local subject="$mail_subject_prefix"
+    [[ -n "$has_errors" ]] && subject+=" ERROR";
+    [[ -n "$status"     ]] && subject+=" - $status";
+    [[ -n "$xstatus"    ]] && subject+=" (${xstatus:2})";
+
+    # assemble mail body
+    local body=
+    if [[ -n "$info" ]] && [[ -n "$has_errors" ]] || [[ "${mail_summary:-no}" = "yes" ]]; then
+        body+="$info"
+    fi
+    if [[ -n "$detail" ]] && [[ -n "$has_errors" ]] || [[ "${mail_detail:-no}" = "yes" ]]; then
+        [[ -n "$body" ]] && body+="${BR}${BR}DETAIL:${BR}"
+        body+="$detail"
+    fi
+
+    # skip sending mail on empty body
+    if [[ -z "$body" ]] && [[ -n "$has_errors" ]]; then
+        body+="FATAL: something went wrong (errors present but empty mail body)${BR}"
+    fi
+    [[ -z "$body" ]] && exit 0
+
+    # send mail
+    echo "$body" | mail -s "$subject" $mailto
+    if [[ $? -ne 0 ]]; then
+        echo "$0: Failed to send btrbk mail to \"$mailto\", dumping mail:${BR}" 1>&2
+        echo "<mail_subject>$subject</mail_subject>${BR}<mail_body>${BR}$body</mail_body>" 1>&2
+    fi
+}
+
+einfo()
+{
+    info+="$1${BR}"
+}
+
+ebegin()
+{
+    ebtext=$1
+    detail+="${BR}### $1${BR}"
+}
+
+eend()
+{
+    if [[ $1 -eq 0 ]]; then
+        eetext=${3-success}
+        detail+="${BR}"
+    else
+        has_errors=1
+        eetext="ERROR (code=$1)"
+        [[ -n "$2" ]] && eetext+=": $2"
+        detail+="${BR}### $eetext${BR}"
+    fi
+    info+="$ebtext: $eetext${BR}"
+    return $1
+}
+
+die()
+{
+    einfo "FATAL: ${1}, exiting"
+    has_errors=1
+    send_mail
+    exit 1
+}
+
+run_cmd()
+{
+    cmd_out=$("$@" 2>&1)
+    local ret=$?
+    detail+="++ ${@@Q}${BR}"
+    if [[ -n "${mail_cmd_block_prefix:-}" ]] && [[ -n "$cmd_out" ]]; then
+        detail+=$(echo -n "$cmd_out" | sed "s/^/${mail_cmd_block_prefix}/")
+        detail+="${BR}"
+    else
+        detail+=$cmd_out
+    fi
+    return $ret
+}
+
+mount_all()
+{
+    # mount all mountpoints listed in $mount_targets
+    mounted=""
+    for mountpoint in $mount_targets; do
+        ebegin "Mounting $mountpoint"
+        run_cmd findmnt -n $mountpoint
+        if [[ $? -eq 0 ]]; then
+            eend -1 "already mounted"
+        else
+            detail+="${BR}"
+            run_cmd mount --target $mountpoint
+            eend $? && mounted+=" $mountpoint"
+        fi
+    done
+}
+
+umount_mounted()
+{
+    for mountpoint in $mounted; do
+        ebegin "Unmounting $mountpoint"
+        run_cmd umount $mountpoint
+        eend $?
+    done
+}
+
+
+check_options
+mount_all
+
+
+#
+# run rsync for all $rsync_enable
+#
+for key in $rsync_enable; do
+    ebegin "Running rsync[$key]"
+    if [[ -d "${rsync_dst[$key]}" ]]; then
+        # There is no proper way to get a proper machine readable
+        # output of "rsync did not touch anything at destination", so
+        # we add "--info=stats2" and parse the output.
+        # NOTE: This also appends the stats to the log file (rsync_log).
+        # Another approach to count the files would be something like:
+        # "rsync --out-format='' | wc -l"
+        run_cmd rsync ${rsync_opt[$key]} \
+                    --info=stats2 \
+                    ${rsync_log[$key]:+--log-file="${rsync_log[$key]}"} \
+                    ${rsync_rsh[$key]:+-e "${rsync_rsh[$key]}"} \
+                    "${rsync_src[$key]}" \
+                    "${rsync_dst[$key]}"
+        exitcode=$?
+
+        # parse stats2 (count created/deleted/transferred files)
+        REGEXP=$'\n''Number of created files: ([0-9]+)'
+        REGEXP+='.*'$'\n''Number of deleted files: ([0-9]+)'
+        REGEXP+='.*'$'\n''Number of regular files transferred: ([0-9]+)'
+        if [[ $cmd_out =~ $REGEXP ]]; then
+            rsync_stats="${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}"
+            rsync_stats_long="${BASH_REMATCH[1]} created, ${BASH_REMATCH[2]} deleted, ${BASH_REMATCH[3]} transferred"
+            nfiles=$(( ${BASH_REMATCH[1]} + ${BASH_REMATCH[2]} + ${BASH_REMATCH[3]} ))
+        else
+            rsync_stats_long="failed to parse stats, assuming files transferred"
+            rsync_stats="-1/-1/-1"
+            nfiles=-1
+        fi
+
+        eend $exitcode "$rsync_stats_long" "$rsync_stats_long"
+        xstatus+=", rsync[$key]=$rsync_stats"
+
+        if [[ $nfiles -ne 0 ]]; then
+            # NOTE: on error, we assume files are transferred
+            rsync_files_transferred=1
+            [[ -n "${sync_fs_onchange[$key]}" ]] && sync_fs+=("${rsync_dst[$key]}")
+        fi
+    else
+        eend -1 "Destination directory not found, skipping: ${rsync_dst[$key]}"
+    fi
+done
+
+# honor skip_btrbk_if_unchanged (only if rsync is enabled and no files were transferred)
+if [[ -n "$rsync_enable" ]] && [[ -n "$skip_btrbk_if_unchanged" ]] && [[ -z "$rsync_files_transferred" ]]; then
+     einfo "No files transferred, exiting"
+     status="No files transferred"
+     umount_mounted
+     if [[ "$skip_btrbk_if_unchanged" != "quiet" ]] || [[ -n "$has_errors" ]]; then
+         send_mail
+     fi
+     exit 0
+fi
+
+
+#
+# sync filesystems in sync_fs
+#
+if [[ ${#sync_fs[@]} -gt 0 ]]; then
+    ebegin "Syncing filesystems at ${sync_fs[@]}"
+    run_cmd sync -f "${sync_fs[@]}"
+    eend $?
+fi
+
+
+#
+# run btrbk
+#
+ebegin "Running btrbk"
+run_cmd btrbk ${btrbk_opts:-} ${btrbk_command}
+exitcode=$?
+case $exitcode in
+    0)  status="All backups successful"
+        ;;
+    3)  status="Another instance of btrbk is running, no backup tasks performed!"
+        ;;
+    10) status="At least one backup task aborted!"
+         ;;
+    *)  status="btrbk failed with error code $exitcode"
+        ;;
+esac
+eend $exitcode "$status"
+
+umount_mounted
+send_mail

contrib/cron/btrbk-verify

@@ -0,0 +1,424 @@
+#!/bin/bash
+#
+# NAME
+#
+#   btrbk-verify - check latest btrbk snapshot/backup pairs
+#
+#
+# SYNOPSIS
+#
+#   btrbk-verify [options] <command> [filter...]
+#
+#
+# DESCRIPTION
+#
+#   Compare btrbk backups. Reads all files and attributes, and
+#   compares checksums of source and target. Uses rsync(1) as backend,
+#   in dry-run mode with all preserve options enabled.
+#
+#   Resolves snapshot/backup pairs by evaluating the output of
+#   "btrbk list latest [filter...]". The filter argument is passed
+#   directly to btrbk, see btrbk(1) FILTER STATEMENTS.
+#
+#   Restrictions:
+#    - ".d..t...... ./" lines are ignored by default:
+#      Root folder timestamp always differ.
+#    - "cd+++++++++ .*" lines are ignored by default:
+#      Nested subvolumes appear as new empty directories.
+#    - btrbk raw targets are skipped
+#    - rsync needs root in most cases (see --ssh-* options)
+#
+#   NOTE: Depending on your setup (hardware, btrfs mount options),
+#   btrbk-verify may eat all your CPU power and use high bandwidth!
+#   Consider nice(1), ionice(1).
+#
+#   Incomplete resource eater list:
+#    - rsync: checksums, heavy disk I/O
+#    - btrfs: decompression, encryption
+#    - ssh: compression, encryption
+#
+#
+# EXAMPLES
+#
+#   btrbk-verify latest /mnt/btr_pool
+#
+#     Verify latest backups from targets configured in
+#     /etc/btrbk/btrbk.conf, matching the "/mnt/btr_pool" filter.
+#
+#   btrbk-verify all
+#
+#     Verify ALL backups from targets in /etc/btrbk/btrbk.conf.
+#     NOTE: This really re-checksums ALL files FOR EACH BACKUP,
+#     even if they were not touched between backups!
+#
+#   btrbk-verify latest -n -v -v
+#
+#     Print detailed log as well as command executed by this script,
+#     without actually executing rsync commands (-n, --dry-run).
+#
+#   btrbk-verify --ssh-agent --ssh-user root --ssh-identity /etc/btrbk/ssh/id_ed25519
+#
+#     Use "ssh -i /etc/btrbk/ssh/id_ed25519 -l root" for rsync rsh
+#     (override settings from btrbk.conf), start an ssh-agent(1) for
+#     this session and verify all latest snapshot / backups.
+#
+#
+# SEE ALSO
+#
+#   btrbk(1), btrbk.conf(5), rsync(1), nice(1), ionice(1)
+#
+#
+# AUTHOR
+#
+#   Axel Burri <axel@tty0.ch>
+#
+
+set -u
+set -e
+set -o pipefail
+
+btrbk_version_min='0.32.0'
+
+# defaults: ignore subvol dirs and root folder timestamp change
+ignore_nested_subvolume_dir=1
+ignore_root_folder_timestamp=1
+ssh_identity=
+ssh_user=
+ssh_start_agent=
+
+verbose=0
+stats_enabled=
+dryrun=
+
+print_usage()
+{
+#80-----------------------------------------------------------------------------
+    cat 1>&2 <<EOF
+usage: btrbk-verify [options] <command> [btrbk-list-options...] [filter...]
+
+options:
+   -h, --help            display this help message
+   -c, --config FILE     specify btrbk configuration file
+   -n, --dry-run         perform a trial run without verifying subvolumes
+   -v, --verbose         be verbose (set twice for debug loglevel)
+   --stats               print rsync stats to stderr (--info=stats2)
+   --strict              treat all rsync diffs as errors
+   --ignore-acls         ignore acls when verifying subvolumes
+   --ignore-xattrs       ignore xattrs when verifying subvolumes
+   --ssh-identity FILE   override ssh_identity from btrbk.conf(5) with FILE,
+                         and clear all other ssh_* options (use with --ssh-user)
+   --ssh-user USER       override ssh_user from btrbk.conf(5) with USER, and
+                         clear all other ssh_* options(use with --ssh-identity)
+   --ssh-agent           start ssh-agent(1) and add identity
+
+commands:
+   latest    verify most recent snapshots and backups (btrbk list latest)
+   all       verify all snapshots and backups (btrbk list backups)
+
+For additional information, see <https://digint.ch/btrbk/>
+EOF
+#80-----------------------------------------------------------------------------
+    exit ${1:-0}
+}
+
+list_subcommand=
+btrbk_args=()
+rsync_args=(-n --itemize-changes --checksum -a --delete --numeric-ids --hard-links --acls --xattrs --devices --specials)
+
+while [[ "$#" -ge 1 ]]; do
+    key="$1"
+    case $key in
+      latest)
+          [[ -n "$list_subcommand" ]] && print_usage 2;
+          list_subcommand="latest"
+          ;;
+      all)
+          [[ -n "$list_subcommand" ]] && print_usage 2;
+          list_subcommand="backups"
+          ;;
+      -n|--dry-run)
+          dryrun=1
+          ;;
+      --stats)
+          # enable rsync stats2 (transfer statistics)
+          rsync_args+=(--info=stats2)
+          stats_enabled=1
+          ;;
+      --strict)
+          # treat all rsync diffs as errors:
+          #  - empty directories (nested subvolumes)
+          #  - root folder timestamp mismatch
+          ignore_nested_subvolume_dir=
+          ignore_root_folder_timestamp=
+          ;;
+      --ignore-*)  # --ignore-acls, --ignore-xattrs, --ignore-device, ...
+          # remove "--xxx" flag from rsync_args for --ignore-xxx
+          rsync_args=(${rsync_args[@]/"--"${key#"--ignore-"}})
+          ;;
+      --ssh-identity)
+          # use different ssh identity (-i option) for rsync rsh.
+          # NOTE: this overrides all btrbk ssh_* options
+          ssh_identity="$2"
+          shift
+          ;;
+      --ssh-user)
+          # use different ssh user (-l option) for rsync rsh
+          # NOTE: this overrides all btrbk ssh_* options
+          ssh_user="$2"
+          shift
+          ;;
+      --ssh-agent)
+          ssh_start_agent=1
+          ;;
+      -v|--verbose)
+          verbose=$((verbose+1))
+          btrbk_args+=("-v")
+          ;;
+      -h|--help)
+          print_usage 0
+          ;;
+      *)
+          # all other args are passed to btrbk (filter, -c,--config=FILE)
+          btrbk_args+=("$key")
+          ;;
+    esac
+    shift
+done
+
+BR=$'\n'
+
+log_line()
+{
+    echo "$@" 1>&2
+}
+log_stats ()  { [[ -n "$stats_enabled" ]] && log_line "$@" ; return 0; }
+log_verbose() { [[ $verbose -ge 1 ]] && log_line "$@" ; return 0; }
+log_debug()   { [[ $verbose -ge 2 ]] && log_line "$@" ; return 0; }
+log_cmd()
+{
+    local prefix=""
+    [[ -n "$dryrun" ]] && prefix="(dryrun) "
+    log_debug "### ${prefix}$@"
+}
+
+tlog()
+{
+    # same output as btrbk transaction log
+    local status=$1
+    local comment=${2:-}
+    [[ -n "$dryrun" ]] && [[ "$status" == "starting" ]] && status="dryrun_starting"
+    local line="$(date --iso-8601=seconds) verify-rsync ${status} ${target} ${source} - -"
+    [[ -n "$comment" ]] && line="$line # $comment";
+    tlog_text+="$line${BR}"
+    log_debug "$line"
+}
+tlog_print()
+{
+    # tlog goes to stdout
+    echo "${BR}TRANSACTION LOG${BR}---------------${BR}${tlog_text:-}"
+}
+
+# parse "rsync -i,--itemize-changes" output.
+# prints ndiffs to stdout, and detailed log messages to stderr
+count_rsync_diffs()
+{
+    local nn=0
+    local rsync_line_match='^(...........) (.*)$'
+    local dump_stats_mode=
+
+    # unset IFS: no word splitting, trimming (read literal line)
+    while IFS= read -r rsync_line; do
+        local postfix_txt=""
+        if [[ -n "$dump_stats_mode" ]]; then
+            # dump_stats_mode enabled, echo to stderr
+            log_stats "${rsync_line}"
+        elif [[ "$rsync_line" == "" ]]; then
+            # empty line denotes start of --info=stats, enable dump_stats_mode
+            dump_stats_mode=1
+            log_stats "--- BEGIN rsync stats2 dump ---"
+        elif [[ "$rsync_line" =~ $rsync_line_match ]]; then
+            rl_flags="${BASH_REMATCH[1]}"
+            rl_path="${BASH_REMATCH[2]}"
+            if [[ -n "$ignore_root_folder_timestamp" ]] && [[ "$rsync_line" == ".d..t...... ./" ]]; then
+                # ignore timestamp on root folder, for some reason this does not match
+                postfix_txt="   # IGNORE reason=ignore_root_folder_timestamp"
+            elif [[ -n "$ignore_nested_subvolume_dir" ]] && [[ "$rl_flags" == "cd+++++++++" ]]; then
+                # nested subvolumes appear as new empty directories ("cd+++++++++") in rsync (btrfs bug?)
+                postfix_txt="   # IGNORE reason=ignore_nested_subvolume_dir"
+            else
+                nn=$((nn+1))
+                postfix_txt="   # FAIL ndiffs=$nn"
+            fi
+            log_verbose "[rsync] ${rsync_line}${postfix_txt}"
+        else
+            nn=$((nn+1))
+            log_line "btrbk-verify: ERROR: failed to parse rsync line: ${rsync_line}"
+        fi
+    done
+    [[ -n "$dump_stats_mode" ]] && log_stats "--- END rsync stats2 dump ---"
+    echo $nn
+    return 0
+}
+
+rsync_rsh()
+{
+    # btrbk v0.27.0 sets source_rsh="ssh [flags...] ssh_user@ssh_host"
+    # this returns "ssh [flags...] -l ssh_user"
+    local rsh=$1
+    local rsh_match="(.*) ([a-z0-9_-]+)@([a-zA-Z0-9.-]+)$"
+
+    if [[ -z "$rsh" ]]; then
+        return
+    elif [[ -n "$ssh_user" ]] || [[ -n "$ssh_identity" ]]; then
+        # override btrbk.conf from command line arguments
+        log_debug "Overriding all ssh_* options from btrbk.conf"
+        local cmd="ssh -q"
+        [[ -n "$ssh_identity" ]] && cmd="$cmd -i '$ssh_identity'"
+        [[ -n "$ssh_user" ]] && cmd="$cmd -l '$ssh_user'"
+        echo "$cmd"
+    elif [[ $rsh =~ $rsh_match ]]; then
+        echo "${BASH_REMATCH[1]} -l ${BASH_REMATCH[2]}"
+    else
+        log_line "btrbk-verify: ERROR: failed to parse source_rsh: $rsh"
+        exit 1
+    fi
+}
+
+kill_ssh_agent()
+{
+    echo "Stopping SSH agent"
+    eval `ssh-agent -k`
+}
+
+start_ssh_agent()
+{
+    if [[ -z "$ssh_identity" ]]; then
+        log_line "btrbk-verify: ERROR: no SSH identity specified for agent"
+        print_usage 2
+    fi
+    echo "Starting SSH agent"
+    eval `ssh-agent -s`
+    ssh_agent_running=1
+    trap 'exit_trap_action' EXIT
+    ssh-add "$ssh_identity"
+}
+
+
+eval_btrbk_resolved_line()
+{
+    local line=" $1"
+    local prefix=$2
+    local required_keys=$3
+    # reset all variables first
+    for vv in $required_keys; do
+        eval "${prefix}${vv}="
+    done
+    for vv in $required_keys; do
+        # basic input validation, set prefixed variable (eval)
+        local match=" ${vv}='([^']*('\\\\''[^']*)*)'"
+        if [[ $line =~ $match ]] ; then
+            eval "${prefix}${vv}='${BASH_REMATCH[1]}'" || return 1
+        else
+            log_line "btrbk-verify: ERROR: Missing variable \"${vv}\""
+            return 1
+        fi
+    done
+}
+
+exit_trap_action()
+{
+    [[ -n "${ssh_agent_running:-}" ]] && kill_ssh_agent
+    [[ $verbose -gt 0 ]] && tlog_print
+}
+
+# start ssh-agent(1)
+[[ -n "$ssh_start_agent" ]] && start_ssh_agent
+
+# run "btrbk list"
+[[ -z "$list_subcommand" ]] && print_usage 2
+log_verbose "Resolving btrbk $list_subcommand"
+btrbk_cmd=("btrbk" "list" "$list_subcommand" "--format=raw" "-q" "${btrbk_args[@]}")
+log_debug "### ${btrbk_cmd[@]}"
+btrbk_list=$("${btrbk_cmd[@]}")
+btrbk_list_exitstatus=$?
+if [[ $btrbk_list_exitstatus -ne 0 ]]; then
+    log_line "btrbk-verify: ERROR: Command execution failed (status=$btrbk_list_exitstatus): ${btrbk_cmd[@]}"
+    exit 1
+fi
+log_debug "--- BEGIN btrbk list $list_subcommand ---"
+log_debug "$btrbk_list"
+log_debug "--- END btrbk list $list_subcommand ---"
+
+tlog_text=""
+exitstatus=0
+# trap on EXIT (includes all signals)
+trap 'exit_trap_action' EXIT
+
+while read -r btrbk_list_line; do
+    # set R_xxx variables from format=raw line (table format "resolved")
+    log_debug "Evaluating [btrbk list] line: $btrbk_list_line"
+    [[ -z "$btrbk_list_line" ]] && continue
+    if ! eval_btrbk_resolved_line "$btrbk_list_line" \
+         "R_" "snapshot_subvolume target_subvolume source_host target_host target_type source_rsh target_rsh"
+    then
+        log_line "btrbk-verify: ERROR: Parse error of command output: ${btrbk_cmd[@]}"
+        log_line "Make sure to have >=btrbk-${btrbk_version_min} installed!"
+        exitstatus=1
+        break
+    fi
+
+    source="${R_snapshot_subvolume}/"
+    target="${R_target_subvolume}/"
+    [[ -n "$R_source_host" ]] && source="${R_source_host}:${source}"
+    [[ -n "$R_target_host" ]] && target="${R_target_host}:${target}"
+
+    if [[ -z "$R_snapshot_subvolume" ]]; then
+        log_line "WARNING: Skipping task (missing snapshot): target=$target"
+    elif [[ -z "$R_target_subvolume" ]]; then
+        log_line "Skipping task (no target): source=$source"
+    elif [[ "$R_target_type" != "send-receive" ]]; then
+        log_line "Skipping task (target_type=$R_target_type): source=$source, target=$target"
+    elif [[ -n "$R_source_rsh" ]] && [[ -n "$R_target_rsh" ]]; then
+        log_line "WARNING: Skipping task (SSH for both source and target is not supported): target=$target"
+    else
+        log_line "Comparing [rsync] $source $target"
+
+        # rsync rsh is either source_rsh or target_rsh or empty
+        eff_rsh="$R_source_rsh"
+        [[ -z "$eff_rsh" ]] && eff_rsh="$R_target_rsh"
+        eff_rsh=$(rsync_rsh "$eff_rsh")
+
+        rsync_cmd=("rsync" "${rsync_args[@]}")
+        [[ -n "$eff_rsh" ]] && rsync_cmd+=("-e" "$eff_rsh")
+        rsync_cmd+=("${source}" "${target}")
+        log_cmd "${rsync_cmd[@]}"
+        [[ -n "$dryrun" ]] && rsync_cmd=("cat" "/dev/null")
+
+        #rsync_cmd=("echo" '........... SHOULD/FAIL/');   # simulate failure
+        #rsync_cmd=("echo" 'cd+++++++++ SHOULD/IGNORE/'); # simulate ignored
+
+        # execute rsync
+        tlog "starting"
+        set +e
+        ndiffs=$("${rsync_cmd[@]}" | count_rsync_diffs)
+        rsync_exitstatus=$?
+        set -e
+
+        if [[ $rsync_exitstatus -ne 0 ]] || [[ -z "$ndiffs" ]]; then
+            log_line "btrbk-verify: ERROR: Command execution failed (status=$rsync_exitstatus): ${rsync_cmd[@]}"
+            tlog "ERROR"
+            exitstatus=10
+        elif [[ $ndiffs -gt 0 ]]; then
+            log_line "VERIFY FAIL (ndiffs=$ndiffs): ${source} ${target}"
+            tlog "fail" "ndiffs=$ndiffs"
+            exitstatus=10
+        else
+            log_verbose "Compare success (ndiffs=$ndiffs)"
+            tlog "success"
+        fi
+    fi
+done <<< "$btrbk_list"
+#done < <(echo "$btrbk_list")  # more posix'ish
+
+# NOTE: this triggers exit_trap_action()
+exit $exitstatus

contrib/crypt/kdf_pbkdf2.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+#
+# kdf_pbkdf2.py - (kdf_backend for btrbk)
+#
+# Copyright (c) 2017 Axel Burri
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# ---------------------------------------------------------------------
+# The official btrbk website is located at:
+# https://digint.ch/btrbk/
+#
+# Author:
+# Axel Burri <axel@tty0.ch>
+# ---------------------------------------------------------------------
+
+import sys
+import os
+import getpass
+import hashlib
+
+def passprompt():
+    pprompt = lambda: (getpass.getpass("Passphrase: "), getpass.getpass("Retype passphrase: "))
+    p1, p2 = pprompt()
+    while p1 != p2:
+        print("No match, please try again", file=sys.stderr)
+        p1, p2 = pprompt()
+    return p1
+
+if len(sys.argv) <= 1:
+    print("Usage: {} <dklen>".format(sys.argv[0]), file=sys.stderr)
+    sys.exit(1)
+
+hash_name = "sha256"
+iterations = 300000
+dklen = int(sys.argv[1])
+salt = os.urandom(16)
+password = passprompt().encode("utf-8")
+
+dk = hashlib.pbkdf2_hmac(hash_name=hash_name, password=password, salt=salt, iterations=iterations, dklen=dklen)
+
+salt_hex = "".join(["{:02x}".format(x) for x in salt])
+dk_hex = "".join(["{:02x}".format(x) for x in dk])
+
+print("KEY=" + dk_hex);
+print("algorithm=pbkdf2_hmac");
+print("hash_name=" + hash_name);
+print("salt=" + salt_hex);
+print("iterations=" + str(iterations));

contrib/migration/raw_suffix2sidecar

@@ -0,0 +1,181 @@
+#!/usr/bin/perl
+#
+# raw_suffix2sidecar - migrate to btrbk raw target sidecar files
+#
+# Copyright (C) 2017 Axel Burri
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# ---------------------------------------------------------------------
+# The official btrbk website is located at:
+# https://digint.ch/btrbk/
+#
+# Author:
+# Axel Burri <axel@tty0.ch>
+# ---------------------------------------------------------------------
+
+# Create raw sidecar ".info" files from uuid-suffixed raw backup files
+# generated by btrbk < v0.26.0.
+
+use strict;
+use warnings FATAL => qw( all );
+use Getopt::Long qw(GetOptions);
+
+our $VERSION      = '0.26.0'; # match btrbk version
+our $AUTHOR       = 'Axel Burri <axel@tty0.ch>';
+our $PROJECT_HOME = '<https://digint.ch/btrbk/>';
+
+my  $VERSION_INFO = "raw_suffix2sidecar (btrbk migration script), version $VERSION";
+
+my $compress_format_alt = 'gz|bz2|xz|lzo|lz4';
+my $file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/;  # note: ubuntu uses '@' in the subvolume layout: <https://help.ubuntu.com/community/btrfs>
+my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/;
+my $timestamp_postfix_match = qr/\.(?<YYYY>[0-9]{4})(?<MM>[0-9]{2})(?<DD>[0-9]{2})(T(?<hh>[0-9]{2})(?<mm>[0-9]{2})((?<ss>[0-9]{2})(?<zz>(Z|[+-][0-9]{4})))?)?(_(?<NN>[0-9]+))?/;  # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]"
+my $raw_postfix_match = qr/--(?<received_uuid>$uuid_match)(\@(?<parent_uuid>$uuid_match))?\.btrfs?(\.(?<compress>($compress_format_alt)))?(\.(?<encrypt>gpg))?(\.(?<split>split_aa))?(\.(?<incomplete>part))?/;  # matches ".btrfs_<received_uuid>[@<parent_uuid>][.gz|.bz2|.xz|...][.gpg][.split_aa][.part]"
+
+my $dryrun;
+
+my %raw_info_sort = (
+  TYPE                 => 1,
+  FILE                 => 2,
+  RECEIVED_UUID        => 3,
+  RECEIVED_PARENT_UUID => 4,
+  INCOMPLETE           => 5,
+  compress             => 9,
+  split                => 10,
+  encrypt              => 11,
+ );
+
+sub VERSION_MESSAGE
+{
+  print STDERR $VERSION_INFO . "\n\n";
+}
+
+sub HELP_MESSAGE
+{
+  print STDERR "usage: raw_suffix2sidecar <dir>...\n";
+  print STDERR "\n";
+  print STDERR "options:\n";
+  #            "--------------------------------------------------------------------------------"; # 80
+  print STDERR "   -h, --help                display this help message\n";
+  print STDERR "       --version             display version information\n";
+  print STDERR "   -n, --dry-run             perform a trial run with no changes made\n";
+  print STDERR "\n";
+  print STDERR "For additional information, see $PROJECT_HOME\n";
+}
+
+sub write_raw_info($$)
+{
+  my $file = shift // die;
+  my $raw_info = shift // die;
+
+  my $info_file = $file . '.info';
+  my @line;
+  push @line, "#raw_suffix2sidecar-v$VERSION";
+  push @line, "# Do not edit this file";
+
+  # sort by %raw_info_sort, then by key
+  foreach(sort { (($raw_info_sort{$a} || 99) <=> ($raw_info_sort{$b} || 99)) || ($a cmp $b) } keys %$raw_info) {
+    push @line, ($_ . '=' . $raw_info->{$_}) if($raw_info->{$_});
+  }
+
+  print "Creating info file: $info_file\n";
+
+  unless($dryrun) {
+    open (INFOFILE, ">> $info_file") || die "Failed to open $info_file";
+    print INFOFILE join("\n", @line) . "\n";
+    close(INFOFILE);
+  }
+
+  return $info_file;
+}
+
+
+MAIN:
+{
+  Getopt::Long::Configure qw(gnu_getopt);
+  unless(GetOptions(
+    'help|h'             => sub { VERSION_MESSAGE(); HELP_MESSAGE(0); exit 0; },
+    'version'            => sub { VERSION_MESSAGE(); exit 0; },
+    'dry-run|n'          => \$dryrun,
+   ))
+  {
+    VERSION_MESSAGE();
+    HELP_MESSAGE(0);
+    exit 2;
+  }
+  unless(@ARGV) {
+    VERSION_MESSAGE();
+    HELP_MESSAGE();
+    exit 1;
+  }
+
+  foreach my $target_dir (@ARGV) {
+    $target_dir =~ s/\/+$//;
+    print "Processing directory: $target_dir/\n";
+    opendir(my($dh), $target_dir) || die "Failed to open directory '$target_dir': $!";
+    my @files = readdir($dh);
+    closedir $dh;
+
+    my @splitfiles = @files;
+    foreach my $file (@files) {
+      if($file =~ /^(?<basename>$file_match$timestamp_postfix_match)$raw_postfix_match$/) {
+        print "\nProcessing raw backup: $file\n";
+
+        my $newname = $+{basename} || die;
+        my %raw_info = (
+          TYPE => 'raw',
+          RECEIVED_UUID => $+{received_uuid},
+          RECEIVED_PARENT_UUID => $+{parent_uuid},
+          INCOMPLETE    => $+{incomplete} ? 1 : 0,
+          compress      => $+{compress},
+          split         => ($+{split} ? (-s $file) : undef), # file size
+          encrypt       => $+{encrypt},
+         );
+        die "Missing received uuid in file: $file" unless $raw_info{RECEIVED_UUID};
+        $newname .= '.btrfs';
+        $newname .= '.' . $raw_info{compress} if($raw_info{compress});
+        $newname .= '.' . $raw_info{encrypt} if($raw_info{encrypt});
+        $raw_info{FILE} = $newname;
+        write_raw_info("$target_dir/$newname", \%raw_info);
+
+        if($raw_info{split}) {
+          my $sfile = $file;
+          $sfile =~ s/_aa$//; # we match on ".split_aa" above
+          foreach my $splitfile (@splitfiles) {
+            if($splitfile =~ /^${sfile}(_[a-z]+)$/) {
+              my $suffix = $1 // die;
+              print "Renaming file: $target_dir/$splitfile -> $target_dir/$newname.split$suffix\n";
+              unless($dryrun) {
+                rename("$target_dir/$splitfile", "$target_dir/$newname.split$suffix") || die "Failed to rename file: $target_dir/$splitfile -> $target_dir/${newname}.split$suffix: $!";
+              }
+            }
+          }
+        }
+        else {
+          print "Renaming file: $target_dir/$file -> $target_dir/$newname\n";
+          unless($dryrun) {
+            rename("$target_dir/$file", "$target_dir/$newname") || die "Failed to rename file: $target_dir/$file -> $target_dir/$newname";
+          }
+        }
+      }
+    }
+  }
+
+  if($dryrun) {
+    print "\nNOTE: Dryrun was active, none of the operations above were actually executed!\n";
+  }
+}
+
+1;

contrib/systemd/btrbk.service.in

@@ -1,6 +1,7 @@
 [Unit]
 Description=btrbk backup
+Documentation=man:btrbk(1)
 
 [Service]
 Type=oneshot
-ExecStart=/usr/sbin/btrbk run
+ExecStart=@BINDIR@/btrbk run

contrib/systemd/btrbk.timer.in

@@ -7,4 +7,4 @@ AccuracySec=10min
 Persistent=true
 
 [Install]
-WantedBy=multi-user.target
+WantedBy=timers.target

contrib/tools/btrbk_restore_raw.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python3
+
+import os
+import logging
+import subprocess
+import argparse
+
+
+logger = logging.getLogger(__name__)
+
+
+class TransformProcess:
+    def run(self, bfile, options, **kw):
+        return subprocess.Popen(self.get_cmd(bfile, options), **kw)
+
+    def get_cmd(self, bfile, options):
+        raise NotImplementedError()
+
+    @classmethod
+    def add_parser_options(cls, parser):
+        pass
+
+
+class TransformOpensslDecrypt(TransformProcess):
+    @staticmethod
+    def get_cmd(bfile, options):
+        return [
+            'openssl', 'enc', '-d', '-' + bfile.info['cipher'], '-K',
+            open(options.openssl_keyfile, 'r').read(), '-iv', bfile.info['iv']
+        ]
+
+    @staticmethod
+    def add_parser_options(parser):
+        parser.add_argument('--openssl-keyfile', help="path to private encryption key file")
+
+
+class TransformDecompress(TransformProcess):
+    def __init__(self, program):
+        self.p = program
+
+    def get_cmd(self, bfile, options):
+        return [self.p, '-d']
+
+
+class TransformBtrfsReceive(TransformProcess):
+    @classmethod
+    def run(cls, bfile, options, **kw):
+        return subprocess.Popen(cls.get_cmd(bfile, options), **kw)
+
+    @staticmethod
+    def get_cmd(bfile, options):
+        return ['btrfs', 'receive', options.restore_dir]
+
+
+TRANSFORMERS = (
+    TransformOpensslDecrypt, TransformDecompress, TransformBtrfsReceive
+)
+
+
+class BtrfsPipeline:
+    def __init__(self, bfile):
+        self.bfile = bfile
+        self.processors = []
+
+    def append(self, transformer):
+        self.processors.append(transformer)
+
+    def run(self, options):
+        processes = []
+        with open(self.bfile.data_file, 'rb') as next_input:
+            for transformer in self.processors:
+                process = transformer.run(
+                    self.bfile, options,
+                    stdin=next_input, stdout=subprocess.PIPE,
+                    stderr=subprocess.PIPE)
+                next_input = process.stdout
+                processes.append(process)
+            btrfs_process = TransformBtrfsReceive.run(
+                self.bfile, options, stdin=next_input,
+                stderr=subprocess.PIPE, stdout=subprocess.DEVNULL)
+            processes.append(btrfs_process)
+            # warning: the code below is pretty ugly and hacky
+            terminated = 0
+            while terminated < len(processes):
+                for p in processes:
+                    if p.returncode is not None:
+                        continue
+                    msg = None
+                    try:
+                        p.wait(timeout=1)
+                    except subprocess.TimeoutExpired as e:
+                        pass
+                    except Exception as e:
+                        msg = e
+                    else:
+                        msg = p.stderr.read().decode('utf-8').strip()
+                    finally:
+                        if p.returncode is not None:
+                            terminated += 1
+                            if p.returncode != 0:
+                                for p_other in processes:
+                                    p_other.terminate()
+                                    terminated += 1
+                                if msg:
+                                    logger.error(f"error running {p.args}: {msg}")
+
+    def get_cmd(self, options):
+        command_pipe = [['cat', self.bfile.data_file]]
+        for transformer in self.processors:
+            command_pipe.append(transformer.get_cmd(self.bfile, options))
+        command_pipe.append(TransformBtrfsReceive.get_cmd(self.bfile, options))
+        return ' | '.join(' '.join(x) for x in command_pipe)
+
+
+class BackupFile:
+    def __init__(self, path):
+        assert path.endswith('.info')
+        self.info_file = path
+        self.info = self._parse_info()
+        self.uuid = self.info['RECEIVED_UUID']
+        self.data_file = os.path.join(os.path.dirname(path), self.info['FILE'])
+        self.parent = self.info.get('RECEIVED_PARENT_UUID')
+        self.is_restored = False
+
+    def _parse_info(self):
+        config = {}
+        with open(self.info_file, 'r') as fh:
+            # skip command option line
+            for line in fh.readlines():
+                if '=' not in line:
+                    continue
+                key, val = line.strip().split('=', maxsplit=1)
+                config[key] = val
+        return config
+
+    def get_transformers(self):
+        if 'encrypt' in self.info:
+            if self.info['encrypt'] == 'gpg':
+                raise NotImplementedError('gpg encryption')
+            elif self.info['encrypt'] == 'openssl_enc':
+                yield TransformOpensslDecrypt()
+            else:
+                raise Exception(f'unknown encryption type: "{self.info["encrypt"]}"')
+        if 'compress' in self.info:
+            yield TransformDecompress(self.info['compress'])
+
+    def restore_file(self, options):
+        assert self.info.get('TYPE') == 'raw'
+        assert not self.info.get('INCOMPLETE')
+        pipeline = BtrfsPipeline(self)
+        for transformer in self.get_transformers():
+            pipeline.append(transformer)
+        if options.dry_run:
+            print(pipeline.get_cmd(options))
+        else:
+            logger.info(f"restoring backup {os.path.basename(self.data_file)}")
+            pipeline.run(options)
+        self.is_restored = True
+
+
+def restore_from_path(backup, options):
+    path = os.path.dirname(backup)
+    info_files = {}
+    backup_file = BackupFile(backup + '.info')
+    restored_files = set()
+    for entry in os.scandir(path):
+        if entry.is_file() and entry.name.endswith('.info'):
+            info = BackupFile(entry.path)
+            info_files[info.uuid] = info
+    restored_files.update(restore_backup(backup_file, info_files, options))
+    logger.info(f"finished; restored {len(restored_files)} backup files")
+
+
+def restore_backup(bfile, parents, options):
+    if bfile.is_restored:
+        return
+    if bfile.parent:
+        parent = parents.get(bfile.parent)
+        if not parent:
+            msg = (f"missing parent {bfile.parent} for"
+                   f"'{os.path.basename(bfile.info_file)}'")
+            if options.ignore_missing:
+                logger.warning(msg)
+            else:
+                raise Exception(msg)
+        else:
+            yield from restore_backup(parent, parents, options)
+    bfile.restore_file(options)
+    yield bfile.uuid
+
+
+def main():
+    parser = argparse.ArgumentParser(description="restore btrbk raw backup")
+    parser.add_argument('backup', help="backup file to restore; for incremental"
+                        " backups the parent files must be in the same directory")
+    parser.add_argument('restore_dir', help="target directory for restored subvolumes"
+                                            " (path argument for \"btrfs receive\")")
+    parser.add_argument('-n', '--dry-run', action='store_true',
+                        help="print commands that would be executed")
+    parser.add_argument('--ignore-missing', action='store_true',
+                        help="do not fail on missing parent snapshots")
+
+    for transformer in TRANSFORMERS:
+        transformer.add_parser_options(parser)
+
+    args = parser.parse_args()
+
+    if args.dry_run:
+        logger.setLevel('ERROR')
+
+    restore_from_path(args.backup, args)
+
+
+if __name__ == '__main__':
+    logger.setLevel('INFO')
+    logging.basicConfig(format='%(asctime)s %(levelname)s - %(message)s')
+    main()

doc/.gitignore

@@ -0,0 +1,2 @@
+*.[1-8]
+*.[1-8].gz

doc/FAQ.md

@@ -0,0 +1,306 @@
+btrbk FAQ
+=========
+
+How can I auto-mount btrfs filesystems used by btrbk?
+-----------------------------------------------------
+
+Given that the "volume" lines in the btrbk configuration file are
+valid mount-points, you can loop through the configuration and mount
+the volumes like this:
+
+    #!/bin/sh
+    btrbk list volume --format=raw | while read line; do
+        eval $line
+        $volume_rsh mount $volume_path
+    done
+
+Note that the `btrbk list` command accepts filters (see [btrbk(1)],
+FILTER STATEMENTS), which means you can e.g. add "group automount"
+tags in your configuration and dump only the volumes of this group:
+`btrbk list volume automount`.
+
+  [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
+
+
+How can I setup a debian pre-install hook?
+------------------------------------------
+
+Create a file `/etc/apt/apt.conf.d/70btrbk`, e.g.:
+
+    // create a btrfs snapshot before (un)installing packages
+    Dpkg::Pre-Invoke {"/usr/bin/btrbk run /mnt/btr_pool/rootfs";};
+
+In order to make sure that the snapshots are always generated and
+nothing is deleted, add the btrbk command line options `--preserve
+--override=snapshot_create=always`.
+
+
+Why is "subvolume ." configuration not recommended?
+---------------------------------------------------
+
+Referring to a btrbk configuration like this:
+
+    volume /
+      subvolume .
+        snapshot_name rootfs
+
+Btrbk is designed to operate on the subvolumes *within* a root
+subvolume. In the config above, the btrbk snapshots would be created
+*inside* the source subvolume, altering it (from user perspective).
+From btrfs perspective this is not a problem, as the snapshots are
+separate subvolumes referring to the source subvolume and mapped into
+the file system tree below the source subvolume.
+
+The recommended way is to split your data into subvolumes, e.g.:
+
+     # btrfs subvolume create /mnt/data/www
+     # btrfs subvolume create /mnt/data/mysql
+     # btrfs subvolume create /mnt/data/projectx
+
+This way you make full advantage of the btrfs filesystem, as all your
+data now has a name, which helps organizing things a lot. This gets
+even more important as soon as you start snapshotting and
+send/receiving.
+
+The btrbk configuration for this would be:
+
+    volume /mnt/data
+      subvolume www
+        [...]
+      subvolume mysql
+        [...]
+      subvolume projectx
+        [...]
+
+
+How should I organize my btrfs filesystem?
+------------------------------------------
+
+There's lots of ways to do this, and each one of them has its reason
+to exist. Make sure to read the [btrfs SysadminGuide on
+kernel.org](https://btrfs.wiki.kernel.org/index.php/SysadminGuide) as
+a good entry point.
+
+<!-- TODO: add links to recommendations for ubuntu and other distros -->
+
+
+### btrfs root
+
+If your linux root filesystem is btrfs, I recommend booting linux from
+a btrfs subvolume, and use the btrfs root only as a container for
+subvolumes (i.e. NOT booting from "subvolid=5"). This has the big
+advantage that you can choose the subvolume in which to boot by simply
+switching the `rootflags=subvol=<subvolume>` kernel boot option.
+
+Example (/boot/grub/grub.cfg):
+
+    menuentry 'Linux' {
+      linux /boot/vmlinuz root=/dev/sdb3 ro rootflags=subvol=rootfs quiet
+    }
+    menuentry 'Linux (testing)' {
+      linux /boot/vmlinuz root=/dev/sdb3 ro rootflags=subvol=rootfs_testing
+    }
+
+Note that btrbk snapshots and backups are read-only, this means you
+have to create a run-time (rw) snapshot before booting into it:
+
+    # btrfs subvolume snapshot /mnt/btr_pool/backup/btrbk/rootfs-20150101 /mnt/btr_pool/rootfs_testing
+
+
+How do I convert '/' (subvolid=5) into a subvolume?
+---------------------------------------------------
+
+There's several ways to achieve this, the solution described below
+guarantees not to create new files (extents) on disk.
+
+### Step 1: make a snapshot of your root filesystem
+
+Assuming that '/' is mounted with `subvolid=5`:
+
+    # btrfs subvolume snapshot / /rootfs
+
+Note that this command does NOT make any physical copy of the files of
+your subvolumes within "/", it will only add some metadata.
+
+
+### Step 2: (optional) add the toplevel subvolume to fstab
+
+Add mount point for subvolid=5 to fstab, something like this:
+
+/rootfs/etc/fstab:
+
+    /dev/sda1  /mnt/btr_pool  btrfs  subvolid=5,noatime  0 0
+
+> This step is not critical for a proper root change, but will save
+> your time by preventing further configurations/reboots and manually
+> mounting the toplevel subvolume.
+
+
+### Step 3: boot from the new subvolume "rootfs".
+
+Either add `rootflags=subvol=rootfs` to grub.cfg, or set subvolume
+"rootfs" as default:
+
+    # btrfs subvolume set-default <subvolid> /
+
+You can obtain `<subvolid>` via
+`btrfs subvolume show /rootfs | grep "Subvolume ID"`
+
+> Editing grub.cfg manually may lead you some troubles if you perform
+> some actions that will fire `grub-mkconfig`.
+
+
+### Step 4: after reboot, check if everything went fine:
+
+First check your **system log** for btrfs errors:
+
+    cat /var/log/messages | grep -i btrfs | grep -i error
+
+then check if current `/` is our new subvolume:
+
+    # btrfs subvolume show /
+            Name:                   rootfs
+            ...
+
+Great, this tells us that we just booted into our new snapshot!
+
+    # btrfs subvolume show /mnt/btr_pool
+    /mnt/btr_pool is toplevel subvolume
+
+This means that the root volume (subvolid=5) is correctly mounted.
+
+
+### Step 5: delete old (duplicate) files
+
+Carefully delete all old files from `/mnt/btr_pool`, except "rootfs"
+and any other subvolumes within "/mnt/btr_pool". In other words,
+delete any folders that are NOT LISTED by `btrfs subvolume list -a
+/mnt/btr_pool`:
+
+    # cd /mnt/btr_pool
+    # mkdir TO_BE_REMOVED
+    # mv bin sbin usr lib var ... TO_BE_REMOVED
+
+Then reboot. If everything went fine, remove the directory:
+
+    # cd /mnt/btr_pool
+    # rm -rf TO_BE_REMOVED
+
+
+What is the most efficient way to clone btrfs storage?
+------------------------------------------------------
+
+It is very common (and avisable!) to keep backups on a separate
+location. In some situations, it is also required to transport the
+data physically, either to the datacenter or to your safe in the
+basement.
+
+
+### Answer 1: Use "btrbk archive"
+
+A robust approach is to use external disks as archives (secondary
+backups), and regularly run "btrbk archive" on them. As a nice side
+effect, this also detects possible read-errors on your backup targets
+(Note that a "btrfs scrub" is still more effective for that purpose).
+
+See **btrbk archive** command in [btrbk(1)] for more details.
+
+
+### Answer 2: Use external storage as "stream-fifo"
+
+This example uses a USB disk as "stream-fifo" for transferring
+(cloning) of btrfs subvolumes:
+
+1. For all source subvolumes (in order of generation):
+
+    `btrfs send /source/subvolX -p PARENT > /usbdisk/streamN`
+
+2. At the target location, restore the streams (in order of
+   generation):
+
+    `cat /usbdisk/streamN | btrfs receive /target`
+
+This approach has the advantage that you don't need to reformat your
+USB disk. This works fine, but be aware that you may run into trouble
+if a single stream gets corrupted, making all subsequent streams
+unusable.
+
+
+### Warning: Avoid using "dd" on btrfs filesystems!
+
+If you use `dd` (e.g. in order to clone a partition), make sure you
+don't mount the cloned filesystem at the same time as the original
+one. You will end up having multiple filesystems **sharing identical
+UUID**, which will break things. If you _really_ want to do this, make
+sure to run:
+
+    btrfstune -u /dev/sdaX
+
+which changes the UUID of the given device. Note that the btrfs
+subvolumes still share identical UUID's, but at least the kernel can
+cope with it (see
+[this post on stackexchange](https://unix.stackexchange.com/questions/246976/btrfs-subvolume-uuid-clash)
+).
+
+Btrbk on the other hand relies on subvolume UUID's being *universally
+unique*, and uses them as hash keys for identifying and caching
+filesystem and subvolume trees, which leads to undefined behavior if
+multiple identical UUID's are processed.
+
+
+I'm getting an error: Aborted: "Received UUID" is set
+-----------------------------------------------------
+
+You probably restored a backup with send-receive, and made it
+read/write using `btrfs property set`. This is bad, as all snapshots
+and backups will inherit this identical "Received UUID", which results
+in all these subvolumes will be treated as "containing same data".
+
+To fix this, create a "proper" snapshot:
+
+    # cd /mnt/btr_pool
+    # mv mysubvolume mysubvolume.broken
+    # btrfs subvolume snapshot mysubvolume.broken mysubvolume
+
+Now, `mysubvolume` should have an empty "Received UUID". Note that in
+order to have a clean environment, you also need to fix all subvolumes
+(snapshots as well as backups) that you created with the broken
+subvolume.
+
+Check if there are more broken subvolumes:
+
+    # btrfs subvolume show mysubvolume.broken
+    # btrfs subvolume list -a -R /mnt/btr_pool | grep <"Received UUID" from above>
+    # btrfs subvolume list -a -R /mnt/btr_backup | grep <"Received UUID" from above>
+
+Either delete them (they won't be used for incremental send-receive
+anyways), or clean them as follows:
+
+    # btrfs subvolume snapshot listed_ro_subvol listed_ro_subvol.rw
+    # btrfs subvolume delete listed_ro_subvol
+    # btrfs subvolume snapshot -r listed_ro_subvol.rw listed_ro_subvol
+    # btrfs subvolume delete listed_ro_subvol.rw
+
+Finally, don't forget to delete the broken source subvolume:
+
+    # btrfs subvolume delete mysubvolume.broken
+
+You should now have a clean environment, and btrbk will not complain
+any more.
+
+
+I'm getting an error: Aborted: subvolume has no UUID
+----------------------------------------------------
+
+If your file system was created with btrfs-progs < 4.16, the btrfs
+root subvolume (id=5) has no UUID. You can check this by calling:
+
+    # btrfs subvolume show /mnt/btr_pool
+    /
+        Name:    <FS_TREE>
+        UUID:    -
+        [...]
+
+Without a UUID, the snapshots would get no parent_uuid, leaving btrbk
+unable to track parent/child relationships. In this case, btrbk
+refuses to create snapshots and backups.

doc/Makefile

@@ -0,0 +1,69 @@
+DOCS       = FAQ.md
+MAN_MAN1   = btrbk.1 \
+             lsbtr.1 \
+             ssh_filter_btrbk.1
+MAN_MAN5   = btrbk.conf.5
+
+PN         = btrbk
+PREFIX    ?= /usr
+DOCDIR     = $(PREFIX)/share/doc/$(PN)
+MAN1DIR    = $(PREFIX)/share/man/man1
+MAN5DIR    = $(PREFIX)/share/man/man5
+
+ifeq ($(COMPRESS), yes)
+  DOCS     := $(addsuffix .gz,$(DOCS))
+  MAN_MAN1 := $(addsuffix .gz,$(MAN_MAN1))
+  MAN_MAN5 := $(addsuffix .gz,$(MAN_MAN5))
+endif
+
+# convert using "asciidoctor": <https://asciidoctor.org>
+# fallback to "a2x" from asciidoc package: <http://asciidoc.org>
+ifneq (, $(shell command -v asciidoctor 2> /dev/null))
+  ASCIIDOC_MANPAGE = asciidoctor -d manpage -b manpage
+  ASCIIDOC_HTML    = asciidoctor -b html5 -d article
+else ifneq (, $(shell command -v a2x 2> /dev/null))
+  # NOTE: using -L (--no-xmllint), as xmllint is a separate package on many distros.
+  ASCIIDOC_MANPAGE = a2x -L -d manpage -f manpage
+  ASCIIDOC_HTML    = asciidoc -b html -d article
+else
+  ASCIIDOC_ERR = $(error "please install either asciidoc or asciidoctor")
+  ASCIIDOC_MANPAGE = $(ASCIIDOC_ERR)
+  ASCIIDOC_HTML    = $(ASCIIDOC_ERR)
+endif
+
+# reproducible builds: reference date is ":date:" attribute from asciidoc source
+date_attr = $(shell sed -rn 's/:date:\s*//p' $(1))
+source_date_epoch = $(shell date +%s -u -d $(call date_attr,$(1)))
+
+
+all: man
+man: man1 man5
+man1: $(MAN_MAN1)
+man5: $(MAN_MAN5)
+
+install: install-man install-doc
+
+install-man: man
+	install -d -m 755 "$(DESTDIR)$(MAN1DIR)"
+	install -d -m 755 "$(DESTDIR)$(MAN5DIR)"
+	install -p -m 644 $(MAN_MAN1) "$(DESTDIR)$(MAN1DIR)"
+	install -p -m 644 $(MAN_MAN5) "$(DESTDIR)$(MAN5DIR)"
+
+install-doc: $(DOCS)
+	install -d -m 755 "$(DESTDIR)$(DOCDIR)"
+	install -p -m 644 $(DOCS) "$(DESTDIR)$(DOCDIR)"
+
+clean:
+	rm -f *.md.gz *.[15] *.[15].{gz,html}
+
+%.gz : %
+	gzip -9 -n -c $< > $@
+
+%.1 : %.1.asciidoc
+	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $<
+
+%.5 : %.5.asciidoc
+	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $<
+
+%.html : %.asciidoc
+	SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_HTML) -o $@ $<

doc/btrbk.1

@@ -1,138 +0,0 @@
-.TH "btrbk" "1" "2015-05-27" "btrbk v0.19.0" ""
-.SH NAME
-btrbk \- backup tool for btrfs volumes
-.SH SYNOPSIS
-\fBbtrbk\fR [\-\-version] [\-\-help] [\-c <file>] [\-p] [\-v] [\-q] [\-l <level>] <command> [<args>]
-.SH DESCRIPTION
-\fBbtrbk\fR is a backup tool for btrfs subvolumes, taking advantage of
-btrfs specific capabilities to create atomic snapshots and transfer
-them incrementally to target volumes. It is able to perform backups
-from one source to multiple destinations.
-.PP
-Snapshots as well as backup subvolume names are created in form:
-.IP
-<source_name>.YYYYMMDD[_N]
-.PP
-Where YYYY is the year, MM is the month, and DD is the day of
-creation, and, if multiple backups are created on the same day, N will
-be incremented on each backup, starting at 1.
-.SH OPTIONS
-.TP
-\-\-version
-Prints the btrbk version.
-.TP
-\-\-help
-Prints the synopsis and a list of the commands.
-.TP
-\-c <file>
-Read the configuration from <file>.
-.TP
-\-p
-Preserve all backups. Skips deletion of old backups, even if specified
-in the configuration file.
-.TP
-\-v
-Verbose output. Identical to: \-l info.
-.TP
-\-q
-Quiet operation. If set, btrbk does not print the summary after
-executing the "run" command.
-.TP
-\-l <level>
-Set the level of verbosity. Accepted levels are warn, info, debug,
-and trace.
-.SH COMMANDS
-.PP
-.B run
-[subvolume...]
-.RS 4
-Perform backup operations as specified in the configuration file. If
-the optional [subvolume...] arguments are present, backups are only
-performed for the specified subvolumes (which must match a
-volume/subvolume declaration in the configuration file), and the -p
-(preserve backups) option is implied.
-.PP
-First, btrbk reads information from the source and target btrfs
-filesystems in order to perform sanity checks and identify
-parent/child and received-from relationships.
-.PP
-If the checks succeed, btrbk creates snapshots for all the source
-subvolumes specified in the configuration file.
-.PP
-Then, for each specified target, btrbk creates the backups as follows:
-If the \fIresume_missing\fR option is set (the default), btrbk
-transfers all missing snapshots needed to satisfy the configured
-\fItarget_preserve_{daily,weekly,monthly}\fR retention policy, always
-incrementally from the latest common parent subvolume found. If no
-common parent subvolume is found, a full backup is created for the
-first transfer. Note that the latest snapshot (the one created in the
-first step) is always transferred, regardless of the retention policy.
-.PP
-As a last step, unless the -p (preserve backups) option is set,
-snapshots and backup subvolumes that are not preserved by their
-configured retention policy will be deleted. Note that the latest
-snapshot as well as the latest backup is always preserved, regardless
-of the retention policy.
-.RE
-.PP
-.B dryrun
-[subvolume...]
-.RS 4
-Don't run btrfs commands, just show the snapshots and backup
-subvolumes that would be created/deleted by the \fBrun\fR command. Use
-in conjunction with \fI\-l debug\fR to see the btrfs commands that
-would be executed.
-.RE
-.PP
-.B info
-[subvolume...]
-.RS 4
-Print filesystem usage information for all source/target
-volumes. Optionally filtered by [subvolume...] arguments.
-.RE
-.PP
-.B tree
-[subvolume...]
-.RS 4
-Print the snapshots and their corresponding backup subvolumes as a
-tree. Optionally filtered by [subvolume...] arguments.
-.RE
-.PP
-.B origin
-<subvolume>
-.RS 4
-Print origin information for the given backup subvolume, showing the
-parent-child relationship as well as the received-from information.
-.RE
-.PP
-.B diff
-<from> <to>
-.RS 4
-Print new files since subvolume <from> for subvolume <to>.
-.RE
-.SH FILES
-.TP
-/etc/btrbk.conf
-.PD 0
-.TP
-/etc/btrbk/btrbk.conf
-Default configuration file. The file format and configuration options
-are described in
-.BR btrbk.conf (5).
-.PD
-.SH EXIT STATUS
-.sp
-\fBbtrbk\fR returns a zero exit status if it succeeds. Non-zero is
-returned in case of failure.
-.SH AVAILABILITY
-Please refer to the btrbk project page
-\fBhttp://www.digint.ch/btrbk/\fR for further
-details.
-.SH SEE ALSO
-.BR btrbk.conf (5),
-.BR btrfs (1)
-.PP
-For more information about btrfs and incremental backups, see the web
-site at https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
-.SH AUTHOR
-Axel Burri <axel@tty0.ch>

doc/btrbk.1.asciidoc

@@ -0,0 +1,533 @@
+btrbk(1)
+========
+:date: 2023-03-25
+:release-version: 0.32.6
+:man manual: Btrbk Manual
+:man source: Btrbk {release-version}
+
+
+NAME
+----
+
+btrbk - backup tool for btrfs subvolumes
+
+
+SYNOPSIS
+--------
+
+[verse]
+btrbk [-h|--help] [--version]
+      [-c|--config <file>] [-n|--dry-run] [--exclude <filter>]
+      [-p|--preserve] [--preserve-snapshots] [--preserve-backups]
+      [-v|--verbose] [-q|--quiet] [-l|--loglevel <level>]
+      [-t|--table] [-L|--long] [-1|--single-column]
+      [--format <output-format>] [--pretty]
+      [-S|--print-schedule] [--progress]
+      [--lockfile <file>]
+      [--override <config_option>=<value>]
+      <command> [[--] <filter>...]
+
+
+DESCRIPTION
+-----------
+
+*btrbk* is a backup tool for btrfs subvolumes, taking advantage of
+btrfs specific capabilities to create atomic snapshots and transfer
+them incrementally to a target btrfs filesystem. It is able to perform
+backups from one source to multiple destinations.
+
+For most operations, *btrbk* requires 'root privileges' to run
+correctly. Alternatively, consider using "btrfs-progs-sudo" or
+"btrfs-progs-btrbk" backends, both of which allows you to run btrbk as
+a regular user. Refer to configuration option 'backend' in
+btrbk.conf(5) for more details.
+
+
+=== Snapshots and Backups
+
+Snapshots as well as backup subvolumes are created in the form:
+
+    <snapshot-name>.<timestamp>[_N]
+
+Where '<snapshot-name>' is identical to the source subvolume name,
+unless the configuration option 'snapshot_name' is set. '<timestamp>'
+is a timestamp describing the creation time (local time of the host
+running btrbk) of the snapshot/backup. The format can be configured
+using the 'timestamp_format' option, refer to btrbk.conf(5) for
+details. If multiple snapshots/backups are created on the same
+date/time, 'N' will be incremented on each snapshot, starting at 1.
+
+If a snapshot or backup does not match the naming scheme above
+(e.g. if it has been renamed manually), btrbk will leave it untouched.
+
+Note that in btrfs terminology, a 'snapshot' is a ``subvolume with
+a given initial content of the original subvolume'' (showing a
+parent-uuid, see btrfs-subvolume(8)), and they can be read-write
+(default) or read-only. In btrbk terminology, 'snapshot' means
+``read-only btrfs snapshot'', and 'backup' means ``read-only subvolume
+created with send/receive'' (showing a received-uuid).
+
+
+OPTIONS
+-------
+
+-h, --help::
+    Prints the synopsis and a list of the commands.
+
+--version::
+    Prints the btrbk version.
+
+-c, --config <file>::
+    Read the configuration from <file>.
+
+-n, --dry-run::
+    Don't run anything that would alter the filesystem, just show the
+    snapshots and backup subvolumes that would be created/deleted by
+    the *run*, *snapshot*, *resume*, *prune*, *archive* and *clean*
+    commands. Use in conjunction with '-l debug' to see the btrfs
+    commands that would be executed.
+
+--exclude <filter>::
+    Exclude configured sections matching '<filter>' (see
+    <<_filter_statements,FILTER STATEMENTS>> below), or any specific
+    snapshot from being backuped or deleted, or any specific backup
+    from being deleted.
++
+Note that excluding specific snapshots from being backuped has impact
+on scheduling: e.g. if the "first snapshot of the day" is excluded,
+the "second snapshot of the day" shifts to "first", creating a backup
+as "first backup of the day".
+
+-p, --preserve::
+    Preserve all snapshots and backups. Skips deletion of any
+    snapshots and backups, even if specified in the configuration file
+    (shortcut for "--preserve-snapshots --preserve-backups").
+
+--preserve-snapshots::
+    Preserve all snapshots. Skips deletion of any snapshots, even if
+    specified in the configuration file.
+
+--preserve-backups::
+    Preserve all backups. Skips deletion of any backups, even if
+    specified in the configuration file.
+
+--wipe::
+    Ignore configured snapshot retention policy, delete all but the latest
+    snapshots instead. All snapshots needed for incremental backup
+    (latest common) are also preserved. Useful if you are getting low
+    on disk space (ENOSPC).
+
+-v, --verbose::
+    Increase the logging level, see "--loglevel".
+
+-q, --quiet::
+    Quiet operation. If set, btrbk does not print the summary after
+    executing the *run*, *snapshot*, *resume*, *prune*, or *archive*
+    commands.
+
+-l, --loglevel <level>::
+    Set the level of verbosity for the stderr logging. Accepted levels
+    are: error, warn, info, debug, and trace. Default is info.
+
+-t, --table::
+    Print output in table format (shortcut for "--format=table").
+
+-L, --long::
+    Print output in long table format (shortcut for "--format=long").
+
+-1, --single-column::
+    Print output as single column (not available for all commands).
+
+--format table|long|raw|col:[h:]<columns>::
+    Print output in specified format. If set to "raw", prints
+    space-separated, quoted key=value pairs (machine readable).
++
+If set to "col:", prints only the <columns> specified (comma-separated
+list). Header lines are omitted if the "h:" modifier is present.
+Columns prefixed with "-" are collapsed if empty. Columns postfixed
+with ":RALIGN" are right-aligned.
+
+--pretty::
+    Print table output with lowercase, underlined column headings
+    (instead of single-line uppercase headings).
+
+-S, --print-schedule::
+    Print detailed scheduler information on *run*, *snapshot*,
+    *resume*, *prune* and *archive* commands. Use the '--format'
+    command line option to switch between different output formats.
+
+--progress::
+    Show progress bar on send-receive operation. Requires "mbuffer"
+    command (version >= 20180505) installed on the host running btrbk.
+
+--lockfile <file>::
+    Place an exclusive lock on <file> during program execution, using
+    flock(2). If the lock is held by another process, exit before
+    running any actions. Overrides configuration option
+    "lockfile". Ignored on dryrun ('-n', '--dry-run').
+
+--override <config_option>=<value>::
+    Override a configuration option <config_option> with
+    <value>. Globally, for ALL contexts. Use with care!
+
+
+COMMANDS
+--------
+
+=== Actions
+
+The following commands are used to create snapshots and/or
+backups. All actions can operate in dry-run mode ('-n', '--dry-run').
+Use the '--format' command line option to switch between different
+output formats.
+
+See section RETENTION POLICY in *btrbk.conf*(5) for information on
+configuring the retention policy.
+
+*run* [filter...]::
+    Perform snapshot and backup operations as specified in the
+    configuration file. If the optional [filter...] arguments are
+    present, snapshots and backups are only performed for the
+    subvolumes/targets matching a filter statement (see
+    <<_filter_statements,FILTER STATEMENTS>> below).
++
+*Step 0: Read Data*;;
+    Read information from the source and target btrfs filesystems in
+    order to perform sanity checks and identify parent/child and
+    received-from relationships.
++
+*Step 1: Create Snapshots*;;
+    If the checks succeed, btrbk creates snapshots for the source
+    subvolumes specified in the configuration file, according to the
+    'snapshot_create' option.
++
+*Step 2: Create Backups*;;
+    For each specified target, btrbk creates the backups as follows:
+    After comparing the backups to the source snapshots, btrbk
+    transfers all missing snapshots needed to satisfy the configured
+    target retention policy, incrementally from the latest common
+    parent subvolume found. If no common parent subvolume is found (or
+    if the 'incremental' option is set to ``no''), a full
+    (non-incremental) backup is created.
++
+*Step 3: Delete Backups*;;
+    Unless the -p, --preserve or --preserve-backups option is set,
+    backup subvolumes that are not preserved by their configured
+    retention policy will be deleted. Note that the latest
+    snapshot/backup pair are always preserved, regardless of the
+    retention policy.
++
+*Step 4: Delete Snapshots*;;
+    Unless the -p, --preserve or --preserve-snapshots option is set,
+    snapshots that are not preserved by their configured retention
+    policy will be deleted. Note that the latest snapshot (the one
+    created in step 1) as well as the latest snapshot/backup pair are
+    always preserved, regardless of the retention policy. If any
+    target is unreachable or has errors, all snapshots are preserved
+    in order not to break the incremental chain.
+
+*dryrun* [filter...]::
+    Don't run any btrfs commands that would alter the filesystem, just
+    show the snapshots and backup subvolumes that would be
+    created/deleted by the *run* command. Use in conjunction with '-l
+    debug' to see the btrfs commands that would be executed.
+
+*snapshot* [filter...]::
+    Snapshot only: skips backup creation and deletion (steps 2 and
+    3). Use in conjunction with -p, --preserve (or
+    --preserve-snapshots) if you also want to skip snapshot deletion
+    (step 4).
++
+Note that snapshot deletion is skipped if the target is not
+accessible, as it is still required in order to determine the latest
+snapshot/backup pair (which is always preserved, regardless of the
+retention policy).
+
+*resume* [filter...]::
+    Resume backups: skips snapshot creation (step 1), transfers and
+    deletes snapshots/backups in order to satisfy their configured
+    retention policy. Use in conjunction with -p, --preserve,
+    --preserve-backups, --preserve-snapshots if you want to skip
+    backup and/or snapshot deletion (steps 3, 4).
+
+*prune* [filter...]::
+    Prune snapshots and backups: skips snapshot and backup creation
+    (steps 1, 2), only deletes snapshots and backups in order to
+    satisfy their configured retention policy. Useful for cleaning the
+    disk after changing the retention policy. Use in conjunction with
+    --preserve-backups, --preserve-snapshots if you want to skip
+    backup or snapshot deletion (steps 3, 4).
++
+Note that deletion is skipped if source or target is not accessible,
+as it is still required in order to determine the latest
+snapshot/backup pair (which is always preserved, regardless of the
+retention policy).
+
+*archive* <source> <target> [--raw]::
+    Recursively copy all subvolumes created by btrbk from <source> to
+    <target> directory, optionally rescheduled using
+    'archive_preserve_*' configuration options. Also creates directory
+    tree on <target>. Useful for creating extra archive copies
+    (clones) from your backup disks. Note that you can continue using
+    btrbk after swapping your backup disk with the archive disk.
++
+If you want to use nested subvolumes on the target filesystem, you
+need to create them by hand (e.g. by running "btrfs subvolume create
+<target>/dir"). Check the output of --dry-run if unsure.
++
+Note that this feature needs a *linux kernel >=4.4* to work correctly!
++
+If '--raw' option is set, creates raw targets (experimental, see
+btrbk.conf(5), TARGET TYPES).
+
+*clean* [filter...]::
+    Delete incomplete (garbled) backups. Incomplete backups can be
+    left behind on network errors or kill signals while a send/receive
+    operation is ongoing, and are identified by the "received_uuid"
+    flag not being set on a target (backup) subvolume.
+
+The following table gives a quick overview of the action commands and
+resulting snapshot creation (S+), backup creation (B+), snapshot
+deletion (S-), and backup deletion (B-):
+
+ifdef::backend-docbook,backend-manpage[]
+....
+Command   Option                 S+ B+ S- B-
+--------------------------------------------
+run                              x  x  x  x
+run       --preserve             x  x
+run       --preserve-snapshots   x  x     x
+run       --preserve-backups     x  x  x
+snapshot                         x     x
+snapshot  --preserve             x
+resume                              x  x  x
+resume    --preserve                x
+resume    --preserve-snapshots      x     x
+resume    --preserve-backups        x  x
+prune                                  x  x
+prune     --preserve-snapshots            x
+prune     --preserve-backups           x
+....
+endif::backend-docbook,backend-manpage[]
+ifndef::backend-docbook,backend-manpage[]
+[cols="2*<m,4*^", options="header,autowidth,compact", style="monospaced"]
+|=======
+|Command  |Option                |S+ |B+ |S- |B-
+|run      |                      | x | x | x | x
+|run      |--preserve            | x | x |   |
+|run      |--preserve-snapshots  | x | x |   | x
+|run      |--preserve-backups    | x | x | x |
+|snapshot |                      | x |   | x |
+|snapshot |--preserve            | x |   |   |
+|resume   |                      |   | x | x | x
+|resume   |--preserve            |   | x |   |
+|resume   |--preserve-snapshots  |   | x |   | x
+|resume   |--preserve-backups    |   | x | x |
+|prune    |                      |   |   | x | x
+|prune    |--preserve-snapshots  |   |   |   | x
+|prune    |--preserve-backups    |   |   | x |
+|=======
+endif::backend-docbook,backend-manpage[]
+
+
+=== Informative Commands
+
+The following commands are informative only, and will not alter the
+file system.
+
+*stats* [filter...]::
+    Print statistics of snapshot and backup subvolumes. Optionally
+    filtered by [filter...] arguments (see <<_filter_statements,FILTER
+    STATEMENTS>> below).
+
+*list* <subcommand> [filter...]::
+    Print information defined by <subcommand> in a tabular
+    form. Optionally filtered by [filter...] arguments (see
+    <<_filter_statements,FILTER STATEMENTS>> below).
++
+Available subcommands (default ``all''):
++
+--
+ifndef::backend-docbook,backend-manpage[]
+[horizontal]
+endif::backend-docbook,backend-manpage[]
+
+*all*;;       List all snapshots and backups created by btrbk.
+*snapshots*;; List all snapshots created by btrbk.
+*backups*;;   List all backups (and correlated snapshots) created by
+              btrbk.
+*latest*;;    List most recent common snapshot/backup pair, or most
+              recent snapshot if no common found.
+*config*;;    List configured source/snapshot/target relations.
+*source*;;    List configured source/snapshot relations.
+*volume*;;    List configured volume sections.
+*target*;;    List configured targets.
+--
++
+Use the '--format' command line option to switch between different
+output formats.
+
+
+*usage* [filter...]::
+    Print filesystem usage information for all source/target volumes,
+    optionally filtered by [filter...] arguments (see
+    <<_filter_statements,FILTER STATEMENTS>> below). Note that the
+    "free" value is an estimate of the amount of data that can still
+    be written to the file system.
+
+*origin* <subvolume>::
+    Print the subvolume origin tree: Shows the parent-child
+    relationships as well as the received-from information. Use the
+    '--format' command line option to switch between different output
+    formats.
+
+*diff* <from> <to>::
+    List the modified files since generation (transid) of subvolume
+    <from> in subvolume <to>. Columns:
++
+------------
+SIZE   file was modified for a total of SIZE bytes
+COUNT  file was modified in COUNT generations
+FLAGS  "+"  file accessed at offset 0 (at least once)
+       "c"  COMPRESS flag is set (at least once)
+       "i"  INLINE flag is set (at least once)
+------------
+
+*extents* [diff] <subvolume>... [exclusive <subvolume>...]::
+   Print accurate disk space usage and diff based on extent data
+   (FIEMAP ioctl, slow!).
++
+--
+Subvolumes following the 'exclusive' keyword are added to a separate
+set, and additional set-exclusive data is printed at the end of the
+list. This gives a hint of how much data will be freed if deleting all
+subvolumes in the set. Example:
+
+    btrbk extents diff /backup/data.* exclusive /backup/data.2010*
+
+The EXCLUSIVE column shows the set-exclusive data of all other listed
+(!) subvolumes (relative complement of block regions). Provided that
+all related subvolumes (holding references to extents) are also
+listed, this amount of disk space would be freed when deleting the
+subvolume.
+
+The DIFF column shows the data added to the previous subvolume
+(relative complement of block regions).
+
+If called with the '--related' option, btrbk also lists all related
+subvolumes. This is not recommended for backups, as parent-uuid
+relations break for received subvolumes as soon as an intermediate
+subvolume is deleted.
+
+Note that reading all extents is a disk-intensive task, expect long
+execution times and high ram usage. Consider setting 'cache_dir'.
+--
+
+*ls* <path>|<url>...::
+    List all btrfs subvolumes below <path>. Use the '--format' command
+    line option to switch between different output formats. See
+    lsbtr(1).
+
+*config* print|print-all::
+    Prints the parsed configuration file.
+
+
+FILTER STATEMENTS
+-----------------
+
+Filter arguments are accepted in form:
+
+<group-name>::
+    Matches the 'group' configuration option of 'volume', 'subvolume'
+    or 'target' sections.
+
+<hostname>[:<port>]::
+    Matches the 'hostname' portion from '<url>' of 'volume' or
+    'target' sections.
+
+<directory>|<url>::
+    Matches 'volume', 'subvolume' or 'target' sections by either
+    relative or absolute path (if starting with "/" or "ssh://" or
+    "<hostname>:/"), accepting wildcard character "*". Relative paths
+    are matched against the end of the pathname. Either:
++
+--
+<volume-directory>::
+    Matches 'volume' sections.
+
+<volume-directory>/<subvolume-name>::
+    Matches 'subvolume' sections.
+
+<volume-directory>/<snapshot-dir>/<snapshot-name>::
+    Matches 'subvolume' sections defining snapshots with the
+    configured 'snapshot_dir' and 'snapshot_name'.
+
+<target-directory>::
+    Matches 'target' sections.
+
+<target-directory>/<snapshot-name>::
+    Matches 'target' sections within 'subvolume' sections defining
+    snapshots with the configured 'snapshot_name'.
+
+Accepted formats for '<url>' are:
+
+  ssh://<hostname>[:<port>]/<directory>
+  <hostname>:<directory>
+--
+
+Note that for *run* and *snapshot* commands, a filter matching a
+'target' configuration section also enables snapshot creation of the
+surrounding 'subvolume' section. If this is not desired, consider
+running *snapshot* and *resume* commands separately.
+
+Filter statements can match multiple times (e.g. on group as well as
+host name). In such a case, all matches are processed.
+
+
+FILES
+-----
+
++/etc/btrbk.conf+::
++/etc/btrbk/btrbk.conf+::
+    Default configuration file. The file format and configuration
+    options are described in *btrbk.conf*(5).
+
+
+EXIT STATUS
+-----------
+
+*btrbk* returns the following error codes:
+
+ifndef::backend-docbook,backend-manpage[]
+[horizontal]
+endif::backend-docbook,backend-manpage[]
+0::   No problems occurred.
+1::   Generic error code.
+2::   Parse error: when parsing command-line options or configuration
+      file.
+3::   Lockfile error: if lockfile is present on startup.
+10::  Backup abort: At least one backup task aborted.
+255:: Script error.
+
+
+AVAILABILITY
+------------
+
+Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
+for further details.
+
+
+SEE ALSO
+--------
+
+*btrbk.conf*(5),
+*btrfs*(8)
+
+For more information about btrfs and incremental backups, see the web
+site at https://btrfs.wiki.kernel.org/index.php/Incremental_Backup
+
+
+AUTHOR
+------
+
+Axel Burri <axel@tty0.ch>

doc/btrbk.conf.5

@@ -1,140 +0,0 @@
-.TH "btrbk.conf" "5" "2015-05-27" "btrbk v0.19.0" ""
-.SH NAME
-btrbk.conf \- btrbk configuration file
-.SH SYNOPSIS
-.B /etc/btrbk.conf
-.br
-.B /etc/btrbk/btrbk.conf
-.SH DESCRIPTION
-The btrbk configuration file specifies which btrfs subvolumes on the
-filesystem are to be processed, what target subvolumes should be used
-to create the backups, and where the snapshots should be
-generated. The retention policy, as well as most other options can be
-defined either globally or within a section.
-.PP
-The options specified always apply to the last section encountered,
-superseding the values set in upper-level sections. This means that
-global options must be set before any sections are defined.
-.PP
-The sections are:
-.TP
-\fBvolume\fR  <volume-directory>|<url>
-Directory of a btrfs volume containing the source subvolume(s) to be
-backed up. \fI<volume-directory>\fR must be an absolute path and point
-to a btrfs volume (or subvolume). Usually the mount point of a btrfs
-filesystem mounted with the \fIsubvolid=0\fR option.
-.TP
-\fBsubvolume\fR  <subvolume-name>
-Subvolume to be backed up, relative to the \fI<volume-directory>\fR
-specified in the \fIvolume\fR section.
-.TP
-\fBtarget\fR  <type> <volume-directory>|<url>
-Target type and directory where the backup subvolumes are to be
-created. \fI<volume-directory>\fR must be an absolute path and point
-to a btrfs volume (or subvolume). Currently the the only valid
-\fI<type>\fR is \[lq]send\-receive\[rq].
-.PP
-For the \fIvolume\fR and \fItarget\fR sections, you can also specify a
-ssh-url instead of a local directory. The syntax for \fI<url>\fR is:
-.IP
-ssh://host.xz/path/to/volume
-.PP
-Note that btrfs is very picky on file names (mainly for security
-reasons), only the characters [0-9] [a-z] [A-Z] and "._+-@" are
-allowed.
-.PP
-The configuration options are:
-.TP
-\fBsnapshot_dir\fR  <directory>
-Directory in which the btrfs snapshots are created, relative to
-\fI<volume-directory>\fR of the \fIvolume\fR section. Note that btrbk
-does not autmatically create this directory, and the snapshot creation
-will fail if it is not present.
-.TP
-\fBsnapshot_name\fR <basename> Base name of the created snapshot (and
-backup). This option is only valid in the \fItarget\fR
-section. Defaults to \fI<subvolume-name>\fR.
-.TP
-\fBsnapshot_create\fR  always|ondemand|onchange|no
-If set to \[lq]ondemand\[rq], snapshots are only created if the target
-subvolume is reachable (useful if you are tight on disk space and you
-only need btrbk for backups to an external disk which is not always
-connected).  If set to \[lq]onchange\[rq], snapshots are only created
-if the source subvolume has changed since the last snapshot (more
-precisely: if the btrfs generation has been increased since the last
-snapshot). If set to \[lq]always\[rq], snapshots are always
-created. If set to \[lq]no\[rq], the snapshots are never created
-(useful in conjunction with the \fIresume_missing\fR option if another
-instance of btrbk is taking care of snapshot creation). Defaults to
-\[lq]always\[rq].
-.TP
-\fBincremental\fR  yes|no|strict
-If set, incremental backups are created. If set to \[lq]strict\[rq],
-non-incremental (initial) backups are never created. Defaults to
-\[lq]yes\[rq].
-.TP
-\fBresume_missing\fR  yes|no
-If set, the backups in the target directory are compared to the source
-snapshots, and missing backups are created if needed (complying to the
-target preserve matrix). Defaults to \[lq]yes\[rq].
-.TP
-\fBtarget_preserve_daily\fR  all|<number>
-How many days of backups should be preserved. Defaults to \[lq]all\[rq].
-.TP
-\fBtarget_preserve_weekly\fR  all|<number>
-Defines for how many weeks back weekly backups should be
-preserved. Every backup created at \fIpreserve_day_of_week\fR (or
-the next backup in this week if none was made on the exact day) is
-considered as a weekly backup. Defaults to \[lq]0\[rq].
-.TP
-\fBtarget_preserve_monthly\fR  all|<number>
-Defines for how many months back monthly backups should be
-preserved. Every last weekly backup in a month is considered a
-monthly backup. Defaults to \[lq]all\[rq].
-.TP
-\fBsnapshot_preserve_daily\fR
-.PD 0
-.TP
-\fBsnapshot_preserve_weekly\fR
-.TP
-\fBsnapshot_preserve_monthly\fR
-Defines retention policy for the snapshots, with same semantics as the
-\fItarget_preserve_*\fR options.
-.PD
-.TP
-\fBpreserve_day_of_week\fR  monday|tuesday|...|sunday
-Defines on what day a backup/snapshot is considered as a weekly
-backup. Defaults to \[lq]sunday\[rq].
-.TP
-\fBssh_identity\fR  <file>
-Absolute path to a ssh identity file (private key). Note that if the
-private key is password protected, btrbk will prompt for user input,
-which is usually not desired.
-.TP
-\fBssh_user\fR  <username>
-Remote username for ssh. Defaults to \[lq]root\[rq]. Note that you will
-have to make sure that the remote user is able to run /sbin/btrfs
-(which needs root privileges).
-.TP
-\fBbtrfs_commit_delete\fR  after|each|no
-If set, make sure the deletion of snapshot and backup subvolumes are
-committed to disk when btrbk terminates. Defaults to \[lq]no\[rq].
-.TP
-\fBbtrfs_progs_compat\fR  yes|no \fI*experimental*\fR
-Enable compatibility mode for btrfs-progs < 3.17 (\fIbtrfs
---version\fR). This option can be set either globally or within a
-\fItarget\fR section.  If enabled, the latest common snapshots are
-determined by subvolume names instead of \fIreceived_uuid\fR, which
-can lead to false guesses if the snapshot or target subvolumes are
-manipulated by hand (moved, deleted).
-.PP
-Lines that contain a hash character (#) in the first column are
-treated as comments.
-.SH AVAILABILITY
-Please refer to the btrbk project page
-\fBhttp://www.digint.ch/btrbk/\fR for further
-details.
-.SH SEE ALSO
-.BR btrbk (1)
-.SH AUTHOR
-Axel Burri <axel@tty0.ch>

doc/btrbk.conf.5.asciidoc

@@ -0,0 +1,705 @@
+btrbk.conf(5)
+=============
+:date: 2023-03-25
+:release-version: 0.32.6
+:man manual: Btrbk Manual
+:man source: Btrbk {release-version}
+
+
+NAME
+----
+
+btrbk.conf - btrbk configuration file
+
+
+SYNOPSIS
+--------
+
+[verse]
+/etc/btrbk.conf
+/etc/btrbk/btrbk.conf
+
+
+DESCRIPTION
+-----------
+
+The btrbk configuration file specifies which btrfs subvolumes on the
+filesystem are to be processed, what target subvolumes should be used
+to create the backups, and where the snapshots should be
+generated. The retention policy, as well as most other options can be
+defined either globally or within a section.
+
+The options specified always apply to the last section encountered,
+superseding the values set in upper-level sections. This means that
+global options must be set before any sections are defined.
+
+Blank lines are ignored. A hash character (#) starts a comment
+extending until end of line.
+
+
+SECTIONS
+--------
+
+*volume* <volume-directory>|<url> (optional)::
+    Absolute path pointing to a btrfs file system containing the
+    source subvolume(s) to be backed up. Usually the mount point of a
+    btrfs filesystem mounted with the 'subvolid=5' option.
+
+*subvolume* <subvolume-name>::
+    Subvolume to be backed up, relative to the '<volume-directory>' of
+    the 'volume' section, or absolute if the 'volume' section is
+    omitted. Accepts wildcard character "*".
++
+--
+Note that if this subvolume is btrfs root (id=5), it needs to have a
+valid UUID, which is not the case for file systems created with
+btrfs-progs < 4.16.
+--
+
+*target* [send-receive|raw] <target-directory>|<url>::
+    Target directory where the backup subvolumes are to be
+    created. The optional target type defaults to ``send-receive'',
+    see <<_target_types,TARGET TYPES>> below for details.
++
+--
+Multiple 'target' sections are allowed, in any context: a 'target'
+defined in 'volume' or global context will be used for all underlying
+'subvolume' sections (hint: run "btrbk list" or "btrbk config print"
+to see the resulting configuration).
+--
+
+If a '<url>' is specified, btrbk actions (shell commands) are executed
+remotely via ssh, using the <<_ssh_options,SSH Options>> described
+below. Accepted formats are:
+
+  ssh://<hostname>[:<port>]/<directory>
+  <hostname>:<directory>
+
+Where '<hostname>' is either a host name, an IPv4 address in
+dotted-decimal form, or an IP literal encapsulated within square
+brackets (e.g. "[2001:db8::7]").
+
+If you are connecting to virtual machines, consider configuring
+several 'volume' sections for a '<hostname>', with distinct '<port>'
+numbers for each machine.
+
+
+OPTIONS
+-------
+
+The options described here can be specified in 'global context' as
+well as 'volume', 'subvolume' and 'target' sections, unless stated
+otherwise.
+
+
+=== Basic Options
+
+*timestamp_format* short|long|long-iso::
+    Timestamp format used as postfix for new snapshot subvolume
+    names. Defaults to ``long''.
++
+--
+ifndef::backend-docbook,backend-manpage[]
+[horizontal]
+endif::backend-docbook,backend-manpage[]
+*short*;;    +YYYYMMDD[_N]+  (e.g. "20150825", "20150825_1")
+*long*;;     +YYYYMMDD<T>hhmm[_N]+  (e.g. "20150825T1531")
+*long-iso*;; +YYYYMMDD<T>hhmmss&plusmn;hhmm[_N]+  (e.g. "20150825T153123+0200")
+--
++
+Note that a postfix "_N" is appended to the timestamp if a snapshot or
+backup already exists with the timestamp of current date/time.
++
+Use ``long-iso'' if you want to make sure that btrbk never
+creates ambiguous time stamps (which can happen if multiple
+snapshots are created during a daylight saving time clock
+change).
++
+Note that using ``long-iso'' has implications on the scheduling, see
+<<_reference_time,Reference Time>> below.
+
+*snapshot_dir* <directory>::
+    Directory in which the btrfs snapshots are created, relative to
+    '<volume-directory>' of the 'volume' section, or absolute if the
+    'volume' section is omitted. Note that btrbk does not
+    automatically create this directory, and the snapshot creation
+    will fail if it is not present.
+
+*snapshot_name* <basename>::
+    Base name of the created snapshot (and backup). This option is
+    only valid in the 'subvolume' section. Defaults to
+    '<subvolume-name>'.
+
+*snapshot_create* always|onchange|ondemand|no::
+    If set to ``always'', snapshots are always created. If set to
+    ``onchange'', snapshots are only created if the last snapshot is
+    not up-to-date, i.e. the source subvolume has changed (more
+    precisely: the btrfs generation has been increased) since the last
+    snapshot was created. If set to ``ondemand'', snapshots are only
+    created if at least one target subvolume is reachable (useful if
+    you are tight on disk space and you only need btrbk for backups to
+    an external disk which is not always connected). If set to ``no'',
+    the snapshots are never created (useful if another instance of
+    btrbk is taking care of snapshot creation). Defaults to
+    ``always''.
+
+*incremental* yes|no|strict::
+    If set, incremental backups are created. If set to ``strict'',
+    non-incremental (initial) backups are never created, and
+    incremental backups are restricted to 'related parents' (by
+    parent-uuid relationship). Defaults to ``yes''.
++
+--
+Note that even if the parent-uuid chain is broken, snapshots and
+backups can still share data (which is especially true for backups
+created with 'incremental' option enabled), and are perfectly suitable
+as parents for incremental send-receive operations. But as btrbk can
+not be certain about this, such operations are disallowed in
+"incremental strict" mode.
+--
+
+*noauto* yes|no::
+    If set, the context is skipped by all btrbk actions unless
+    explicitly enabled by a matching btrbk '<filter>' command line
+    argument (e.g. "btrbk run myfilter").
+
+
+=== Grouping Options
+
+*group* <group-name> [<group-name>]...::
+    Add the current section (volume, subvolume or target) to
+    user-defined groups, which can be used as filter for most btrbk
+    commands (see btrbk(1) FILTER STATEMENTS). This option can be set
+    multiple times within the same context.
+
+
+=== Retention Policy Options
+
+*preserve_day_of_week* monday|tuesday|...|sunday::
+    Defines on what day a snapshot/backup is considered to be a
+    "weekly" backup. Weekly, monthly and yearly backups are preserved
+    on this day of week (see <<_retention_policy,RETENTION POLICY>>
+    below). Defaults to ``sunday''.
+
+*preserve_hour_of_day* [0..23]::
+    Defines after what time (in full hours since midnight) a
+    snapshot/backup is considered to be a "daily" backup. Daily,
+    weekly, monthly and yearly backups are preserved on this hour (see
+    <<_retention_policy,RETENTION POLICY>> below). Ignored on
+    snapshots or backups without time information ('timestamp_format
+    short'). Defaults to ``0''.
+
+*snapshot_preserve* no|<retention_policy>::
+    Set retention policy for snapshots (see
+    <<_retention_policy,RETENTION POLICY>> below). If set to ``no'',
+    preserve snapshots according to 'snapshot_preserve_min'
+    only. Defaults to ``no''.
++
+--
+Note that 'snapshot_preserve' has no effect if 'snapshot_preserve_min'
+is set to ``all'' (the default).
+--
+
+*snapshot_preserve_min* all|latest|<number>{h,d,w,m,y}::
+    Preserve all snapshots for a minimum amount of hours (h), days
+    (d), weeks (w), months (m) or years (y), regardless of how many
+    there are. If set to ``all'', preserve all snapshots forever. If
+    set to ``latest'', preserve latest snapshot. Defaults to ``all''.
+
+*target_preserve* no|<retention_policy>::
+    Set retention policy for backups (see
+    <<_retention_policy,RETENTION POLICY>> below). If set to ``no'',
+    preserve backups according to 'target_preserve_min' only. Defaults
+    to ``no''.
++
+--
+Note that 'target_preserve' has no effect if 'target_preserve_min' is
+set to ``all'' (the default).
+--
+
+*target_preserve_min*  all|latest|no|<number>{h,d,w,m,y}::
+    Preserve all backups for a minimum amount of hours (h), days (d),
+    weeks (w), months (m) or years (y), regardless of how many there
+    are. If set to ``all'', preserve all backups forever. If set to
+    ``latest'', always preserve the latest backup (useful in
+    conjunction with "target_preserve no", if you want to keep the
+    latest backup only). If set to ``no'', only the backups following
+    the 'target_preserve' policy are created. Defaults to ``all''.
+
+*archive_preserve* no|<retention_policy>::
+    Set retention policy for archives ("btrbk archive" command), with
+    same semantics as 'target_preserve'.
+
+*archive_preserve_min* all|latest|no|<number>{h,d,w,m,y}::
+    Set retention policy for archives ("btrbk archive" command), with
+    same semantics as 'target_preserve_min'.
+
+*archive_exclude* <pattern>::
+    Exclude subvolumes matching <pattern> from archiving. The pattern
+    accepts wildcard character "*", and is matched against the end of
+    the pathname.
+
+
+=== SSH Options
+
+*ssh_identity* <file>|no::
+    Absolute path to a ssh identity file (private key). If not set,
+    the ssh default is used (see ssh(1), "-i identity_file"). Note
+    that if the identity key is password protected and no
+    authentication agent is used, btrbk will prompt for user input on
+    every connection attempt.
+
+*ssh_user* <username>|no::
+    Remote username for ssh. Defaults to ``root''. Make sure the
+    remote user is able to run "btrfs" with root privileges (see
+    option 'backend' for details). If set to ``no'', the ssh default
+    is used.
+
+*ssh_compression* yes|no::
+    Enables or disables the compression of ssh connections. Defaults
+    to ``no''. Note that if *stream_compress* is enabled, ssh
+    compression will always be disabled for send/receive operations.
+
+*ssh_cipher_spec* default|<cipher_spec>::
+    Selects the cipher specification for encrypting the session
+    (comma-separated list of ciphers in order of preference). See the
+    "-c cipher_spec" option in ssh(1) for more information. Defaults
+    to ``default'' (the ciphers specified in ssh_config(5)).
+
+
+=== Data Stream Options
+
+*stream_compress* <compress_command>|no::
+    Compress the btrfs send stream before transferring it from/to
+    remote locations. Defaults to ``no''. If enabled, make sure that
+    '<compress_command>' is available on the source and target
+    hosts. Supported '<compress_command>': gzip, pigz, bzip2, pbzip2,
+    bzip3, xz, lzo, lz4, zstd.
+
+*stream_compress_level* default|<number>::
+    Compression level for the specified '<compress_command>'. Refer to
+    the related man-page for details (usually [1..9], where 1 means
+    fastest compression). Defaults to ``default'' (the default
+    compression level of '<compress_command>').
+
+*stream_compress_long* default|<number>::
+    Enable long distance matching for the specified
+    '<compress_command>'. Refer to the related man-page for details.
+    Only supported for "zstd".
+
+*stream_compress_threads* default|<number>::
+    Number of threads to use for <compress_command>. Only supported
+    for "pigz", "pbzip2", "bzip3", "zstd" and recent versions of "xz".
+
+*stream_compress_adapt* yes|no::
+    Enable adaptive compression for <compress_command>. Only supported
+    for "zstd" (version >= 1.3.6). Defaults to ``no''.
+
+*stream_buffer* <size>|no::
+    Add a buffer to the btrfs send stream (locally, on uncompressed
+    data), with a maximum size of '<size>'. This can give a speed
+    improvement (measured up to 20%) on both local or remote
+    operations, but also increases system load. A suffix of "k", "m",
+    "g", or "%" can be added to '<size>' to denote kilobytes (*1024),
+    megabytes, gigabytes, or a percentage of total physical
+    memory. Defaults to ``no''.
++
+--
+If enabled, make sure that the "mbuffer" command (at least version
+20180505) is available on the host running btrbk. As of btrbk-0.29.0,
+mbuffer(1) is used for both 'rate_limit' and 'stream_buffer' options:
+
+    mbuffer [-m <stream_buffer>] [-r <rate_limit>]
+
+Note that mbuffer(1) always reads defaults from "`/etc/mbuffer.rc"`
+and "`~/.mbuffer.rc`".
+
+Leave this option disabled if your main concern is a stable backup
+process: while recent versions of mbuffer have proven reliable, it is
+often desirable to keep things simple rather than adding an
+additional, multi-threaded process to the command pipe.
+--
+
+*stream_buffer_remote* <size>|no::
+    Add a buffer on remote hosts (either source or target). Defaults
+    to ``no''.
++
+--
+Enable this if you prefer buffering on the remote side, or even on
+both sides: reasons for this depend on available memory, disk and cpu
+performance (btrfs send/receive, compression), as well as networking
+constraints.
+--
+
+*rate_limit* <rate>|no::
+    Limit the read rate of the btrfs send stream to '<rate>' bytes per
+    second (locally, on uncompressed send stream). A suffix of "k",
+    "m", "g", or "t" can be added to denote kilobytes (*1024),
+    megabytes, and so on. Defaults to ``no''. Note that 'rate_limit'
+    implicitly adds a stream buffer (see 'stream_buffer' option
+    above).
+
+*rate_limit_remote* <rate>|no::
+    Add rate limit on remote hosts (either source or target). Defaults
+    to ``no''. Note that it usually does not make much sense to enable
+    both 'rate_limit' and 'rate_limit_remote'.
+
+=== System Options
+
+*transaction_log* <file>|no::
+    If set, all transactions (snapshot create, subvolume send-receive,
+    subvolume delete) as well as abort messages are logged to <file>,
+    in a space-separated table format: "localtime type status
+    target_url source_url parent_url message".
+
+*transaction_syslog* <facility>|no::
+    If set, all transactions (as described in 'transaction_log' above)
+    are logged to syslog. The program name used in the messages is
+    "btrbk".  Accepted parameters for '<facility>': user, mail,
+    daemon, auth, lpr, news, cron, authpriv, local0..local7.
+
+*lockfile* <file>|no::
+    Place an exclusive lock on <file> during program execution, using
+    flock(2). If the lock is held by another process, exit before
+    running any actions. Ignored on dryrun ('-n', '--dry-run'). See
+    also '--lockfile' command-line option.
+
+*backend* <backend>::
+    Backend filesystem utilities to be used for btrfs specific
+    operations. Available backends:
++
+--
+*btrfs-progs*::
+    Default backend, btrfs commands are called as specified in
+    btrfs(8) (e.g. "btrfs subvolume show").
+
+*btrfs-progs-btrbk*::
+    btrfs commands are separated by a dash instead of a whitespace
+    (e.g. "btrfs-subvolume-show" instead of "btrfs subvolume
+    show"). Useful for setting suid or file capabilities (setcap) on
+    specific btrfs commands, as implemented in
+    <https://github.com/digint/btrfs-progs-btrbk>.
+
+*btrfs-progs-sudo*::
+    btrfs commands are prefixed with "sudo -n" (e.g. "sudo -n btrfs
+    subvolume show" instead of "btrfs subvolume show"). Make sure to
+    have appropriate (root) permissions for the "btrfs" command groups
+    as well as the "readlink" and "test" commands in /etc/sudoers.
+
+*btrfs-progs-doas*::
+    Similar to btrfs-progs-sudo, using prefix "doas -n".
+
+If you want to set this option for local or remote hosts only, set
+*backend_local* or *backend_remote* (e.g. "backend_remote
+btrfs-progs-btrbk").
+
+If you want to set this option for regular (non-root) user only, set
+*backend_local_user*.
+--
+
+*compat* <compat-option>...::
+    Enable compatibility options. Available 'compat-option':
++
+--
+*busybox*::
+    Use busybox compatible commands, at the expense of slight overhead
+    while reading filesystem information.
+
+*ignore_receive_errors*  _*experimental*_::
+    Tell btrfs-receive(8) to not terminate on errors by setting
+    "--max-errors=0" option. Print warnings instead.
++
+A known use case for this are target hosts lacking xattr support
+(e.g. some Synology NAS), while the send-stream contains "lsetxattr"
+commands. Another case is targets failing to set otime, complaining
+with "ERROR: attribute 12 requested but not present".
++
+Note that there is *no guarantee that backups created with this
+option enabled can be restored at all*.
+
+If you want to set this option for local or remote hosts only, set
+*compat_local* or *compat_remote* (e.g. "compat_remote busybox").
+--
+
+*cache_dir* <directory>::
+    If set, cache extent maps for the "btrbk extents" command.
+
+
+=== Btrfs Specific Options
+
+*incremental_prefs* <list-spec>[:<amount>]...::
+    Specify the preferences to determine the best common (correlated)
+    parent and clone sources for incremental backups, by choosing from
+    predefined candidate lists.
++
+--
+The 'list-spec' defines from what candidate list the next
+parent/clone-src should be appended to a result list; 'amount' defines
+how many (e.g. "sro:1 sro:1" is identical to "sro:2"), or all if
+omitted. Any candidate which is already in the results is dropped.
+
+The resulting list of subvolumes is then used as parameters for the
+btrfs-send(8) command: the first for "-p <parent>", all others for
+"-c <clone-src>".
+
+Available 'list-spec' (candidate lists = filtered subsets of
+correlated subvolumes):
+
+*sro*,*srn*:: All from 'snapshot_dir' matching 'snapshot_name', with
+ parent_uuid relationship, sorted by btrbk timestamp (o=older
+ n=newer).
+
+*sao*,*san*:: All from 'snapshot_dir' matching 'snapshot_name', sorted
+ by btrbk timestamp (o=older n=newer).
+
+*aro*,*arn*:: All from 'incremental_resolve', with parent_uuid
+ relationship, sorted by cgen (o=older n=newer).
+
+Defaults to "sro:1 srn:1 sao:1 san:1 aro:1 arn:1". Note that for
+most operations the default resolves a single parent, as there usually
+are no newer snapshots, and all "sro:1 sao:1 aro:1" resolve to the
+same snapshot.
+
+Example: "defaults,sao,san,aro,arn" takes the defaults, and adds clone
+sources for all (!) known candidates on the filesystem.
+--
+
+*incremental_clones* yes|no::
+    If enabled, btrbk adds "-c <clone-src>" to the btrfs-send(8)
+    command for all correlated subvolumes resolved by
+    'incremental_prefs'. If disabled, only "-p <parent>" is
+    used. Defaults to ``yes''.
+
+*incremental_resolve* mountpoint|directory::
+    Specifies where to search for the best common parent for
+    incremental backups. If set to ``mountpoint'', use parents in the
+    filesystem tree below the mount point of the snapshot and target
+    directory. If set to ``directory'', use parents strictly below
+    snapshot/target directories. Set this to ``directory'' if you get
+    access problems (when not running btrbk as root). Defaults to
+    ``mountpoint''.
+
+*btrfs_commit_delete* yes|no::
+    If set, wait for the transaction commit at the end of each
+    snapshot or backup deletion (sets '--commit-each' option for
+    "btrfs subvolume delete"). Defaults to ``no''.
+
+*send_protocol* <number>|no  _*experimental*_::
+    Use btrfs send protocol version N. If enabled on 'target', btrbk
+    adds "--proto <number>" to the btrfs-send(8) command. Defaults to
+    ``no'' (btrfs default).
+
+*send_compressed_data* yes|no  _*experimental*_::
+    Send data that is compressed on the filesystem directly without
+    decompressing it. This requires protocol version 2 or higher
+    (btrfs-progs >= 5.19), and implies "send_protocol 2". If enabled
+    on 'target', btrbk adds "--compressed-data" to the btrfs-send(8)
+    command. Defaults to ``no'' (btrfs default).
+
+*snapshot_qgroup_destroy* yes|no  _*experimental*_:: {blank}
+*target_qgroup_destroy* yes|no  _*experimental*_:: {blank}
+*archive_qgroup_destroy* yes|no  _*experimental*_::
+    Whenever a subvolume is deleted, also destroy corresponding
+    default qgroup "+0/<subvol-id>+". Only useful if you have enabled
+    btrfs quota support. See also:
+    <https://bugzilla.kernel.org/show_bug.cgi?id=91751>
+
+
+=== Informative Options
+
+*warn_unknown_targets* yes|no::
+    If set, prints a warning if btrbk encounters a target subvolume at
+    a unknown location (i.e. not following btrbk naming scheme, or
+    outside the target directory). Defaults to ``no''.
+
+
+RETENTION POLICY
+----------------
+
+Retention policies are defined individually for snapshots, backups and
+archives (summarized as "backups" in the following text), using a
+combination of:
+
+**_preserve_min* all|latest|no|<number>{h,d,w,m,y}::
+    Amount of time (duration) in which all backups are preserved.
+
+**_preserve* no|<retention_policy>::
+    Schedule (single points in time) for which individual backups are
+    preserved.
+
+Note that if "preserve_min" is set to ``all'' (the default), any
+setting of "preserve" obviously has no effect.
+
+The format for '<retention_policy>' is:
+
+    [<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]
+
+*hourly*::
+    Defines how many hours back hourly backups should be
+    preserved. The first backup of an hour is considered an hourly
+    backup.
+
+*daily*::
+    Defines how many days back daily backups should be preserved. The
+    first backup of a day (starting at 'preserve_hour_of_day') is
+    considered a daily backup.
+
+*weekly*::
+    Defines how many weeks back weekly backups should be
+    preserved. The first daily backup created at
+    'preserve_day_of_week' (or the first backup in this week if none
+    was made on the exact day) is considered as a weekly backup.
+
+*monthly*::
+    Defines how many months back monthly backups should be
+    preserved. Every first weekly backup in a month is considered a
+    monthly backup.
+
+*yearly*::
+    Defines for how many years back yearly backups should be
+    preserved. Every first monthly backup in a year is considered a
+    yearly backup.
+
+Use an asterisk for ``all'' (e.g. "target_preserve 60d *m"
+states: "preserve daily backups for 60 days back, and all monthly
+backups").
+
+Hint: Run btrbk with the '-S', '--print-schedule' option to get a
+comprehensive output of the scheduler results.
+
+
+=== Reference Time
+
+The local time on the host running btrbk defines the reference time
+for all date/time calculations, especially for "beginning of a day",
+and as a consequence for the first daily, weekly, monthly or yearly
+backups. The local time on remote hosts (ssh source/target) is never
+used.
+
+Unless "timestamp_format long-iso" is set, daily backups are preserved
+at "preserve_hour_of_day" (defaults to midnight) of the respective
+time zone (and not for "00:00 UTC", which would be "14:00" in
+Honolulu). This becomes relevant for setups with multiple btrbk
+instances, e.g. many snapshot-only instances (spread around the
+world), and a fetch-only instance on the backup server.
+
+Caveat:
+
+* If "timestamp_format long-iso" is set, each btrbk instance on has a
+  different interpretation of "first in day". Make sure to run btrbk
+  with the same time zone on every host, e.g. by setting the TZ
+  environment variable (see tzset(3)).
+
+
+TARGET TYPES
+------------
+
+*send-receive*::
+    Backup to a btrfs filesystem, using "btrfs send/receive". This is
+    the recommended (standard) target type. The '<target-directory>'
+    must be an absolute path and point to a subvolume or directory
+    within a btrfs file system. See btrfs-send(8), btrfs-receive(8).
+
+*raw*  _*experimental*_::
+    Backup to a raw (filesystem independent) file from the output of
+    btrfs-send(8), with optional compression and encryption.
++
+--
+Raw backups consist of two files: the main data file containing the
+btrfs send stream, and a sidecar file ".info" containing metadata:
+
+  <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg]
+  <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg].info
+
+For 'incremental' backups ("incremental yes"), please note that:
+
+* As soon as a single 'incremental' backup file is lost or corrupted,
+  all later incremental backups become invalid, as there is no common
+  parent for the subsequent incremental images anymore. This might be
+  a good compromise for a vacation backup plan, but for the long term
+  make sure that a non-incremental backup is triggered from time to
+  time.
+
+* The scheduler will never delete dependent parents of backups
+  preserved by the retention policy (run btrbk with the '-S',
+  '--print-schedule' option to get a comprehensive output of the
+  scheduler results).
+
+* There is currently no support for rotation of incremental backups:
+  if 'incremental' is set, a full backup must be triggered manually
+  from time to time in order to be able to delete old backups.
+
+
+Additional options for raw targets:
+
+*raw_target_compress* <compress_command>|no::
+    Compression algorithm to use for raw backup target. Supported
+    '<compress_command>': gzip, pigz, bzip2, pbzip2, bzip3, xz, lzo,
+    lz4, zstd.
+*raw_target_compress_level* default|<number>::
+    Compression level for the specified <compress_command>.
+*raw_target_compress_long* default|<number>::
+    Enable long distance matching for the specified
+    '<compress_command>'.
+*raw_target_compress_threads* default|<number>::
+    Number of threads to use for <compress_command>.
+*raw_target_split* <size>|no::
+    Split the raw backup file into pieces of size '<size>'.
+*raw_target_block_size* <number>::
+    Block size to use for writing the raw backup file. Defaults to
+    ``128K''.
+*raw_target_encrypt* gpg|openssl_enc|no::
+    If enabled, encrypt the target raw file using gpg or openssl_enc.
+
+
+Additional options for "raw_target_encrypt gpg":
+
+*gpg_keyring* <file>::
+    Keyring to use for gpg, e.g. "`/etc/btrbk/gpg/pubring.kbx`".
+*gpg_recipient* <name>...::
+    Encrypt for user id '<name>' (email address).
+
+
+Additional options for "raw_target_encrypt openssl_enc" ('very
+experimental'):
+
+*openssl_ciphername*{nbsp}<name>::
+    Defaults to ``aes-256-cbc''.
+*openssl_iv_size* <size-in-bytes>|no::
+    Depends on selected cipher.
+*openssl_keyfile* <file>|no::
+    Point to a key file in hex (absolute path). Example key file
+    creation (256bit key):
++
+------------
+  # dd if=/dev/urandom bs=1 count=32 \
+    | od -x -A n \
+    | tr -d "[:space:]" > /path/to/keyfile
+------------
+
+*kdf_backend* <file>|no::
+    KDF backend to be executed,
+    e.g. "`/usr/share/btrbk/scripts/kdf_pbkdf2.py`".
+*kdf_keysize* <size-in-bytes>::
+    Defaults to ``32''.
+*kdf_keygen* once|each::
+    Defaults to ``once''.
+--
+
+
+AVAILABILITY
+------------
+
+Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
+for further details.
+
+
+SEE ALSO
+--------
+
+*btrbk*(1)
+
+
+AUTHOR
+------
+Axel Burri <axel@tty0.ch>

doc/install.md

@@ -0,0 +1,68 @@
+Installation
+============
+
+Btrbk is a single perl script, and does not require any special
+installation procedures or libraries. In order to install the btrbk
+executable along with the documentation and an example configuration
+file, choose one of the following methods:
+
+
+### Generic Linux System
+
+Install [asciidoctor] or [asciidoc] if you want to build the
+documentation.
+
+Download and unpack the latest [btrbk source tarball] and type:
+
+    sudo make install
+
+#### Try latest master from Github:
+
+    wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
+    chmod +x btrbk
+    sudo ./btrbk ls /
+
+
+### Debian Based Distros
+
+btrbk is in debian stable (utils): https://packages.debian.org/stable/utils/btrbk
+
+Packages are also available via NeuroDebian: http://neuro.debian.net/pkgs/btrbk.html
+
+
+### Fedora Linux
+
+btrbk is in the official Fedora repos: https://src.fedoraproject.org/rpms/btrbk
+
+    sudo dnf install btrbk
+
+
+### Arch Linux
+
+btrbk is in AUR: https://aur.archlinux.org/packages/btrbk/
+
+
+### Alpine Linux
+
+btrbk is in the community repository
+
+    apk add btrbk
+
+
+### Gentoo Linux
+
+btrbk is in portage:
+
+    emerge app-backup/btrbk
+
+
+### Void Linux
+
+btrbk is in Void's `current` repository
+
+    xbps-install -S btrbk
+
+
+  [btrbk source tarball]: https://digint.ch/download/btrbk/releases/
+  [asciidoctor]: https://asciidoctor.org
+  [asciidoc]: https://asciidoc.org

doc/lsbtr.1.asciidoc

@@ -0,0 +1,126 @@
+lsbtr(1)
+========
+:date: 2023-03-25
+:release-version: 0.32.6
+:man manual: Btrbk Manual
+:man source: Btrbk {release-version}
+
+
+NAME
+----
+
+lsbtr - list btrfs subvolumes
+
+
+SYNOPSIS
+--------
+
+[verse]
+lsbtr [-h|--help] [--version]
+      [-l|--long] [-u|--uuid] [-1|--single-column] [--raw]
+      [--format <output-format>]
+      [-v|--verbose]
+      [-c|--config <file>]
+      [--override <config_option>=<value>]
+      [[--] <path>|<url>...]
+
+
+DESCRIPTION
+-----------
+
+List btrfs subvolumes and their mount points visible by the file
+system below '<path>'.
+
+*lsbtr* is part of *btrbk* (basically a shortcut for "btrbk ls"), and
+takes some global configuration options from btrbk.conf(5) if present.
+
+*lsbtr* requires 'root privileges' to run correctly. Alternatively,
+consider using "btrfs-progs-sudo" or "btrfs-progs-btrbk" backends,
+both of which allows you to run lsbtr as a regular user. Refer to
+configuration option 'backend' in btrbk.conf(5) for more details.
+
+
+OPTIONS
+-------
+
+-h, --help::
+    Prints the synopsis and a list of the commands.
+
+--version::
+    Prints the btrbk version.
+
+-l, --long::
+    Print output in long table format (additionally print subvolume
+    path).
+
+-u, --uuid::
+    Print UUID table (parent/received relations).
+
+-1, --single-column::
+    Print path column only (delimited by newline).
+
+--raw::
+    Print space-separated key="value" pairs (machine readable).
+
+--format table|long|raw|col:[h:]<columns>::
+    Print output in specified format. If set to "raw", prints
+    space-separated key="value" pairs (machine readable).
++
+If set to "col:", prints only the <columns> specified (comma-separated
+list). Header lines are omitted if the "h:" modifier is present.
+Columns prefixed with "-" are collapsed if empty. Columns postfixed
+with ":RALIGN" are right-aligned.
+
+-v, --verbose::
+    Increase the level of verbosity.
+
+-c, --config <file>::
+    Read the configuration from <file>.
+
+--override <config_option>=<value>::
+    Override a configuration option <config_option> with <value>.
+
+
+FILES
+-----
+
++/etc/btrbk.conf+::
++/etc/btrbk/btrbk.conf+::
+    Default configuration file. The file format and configuration
+    options are described in *btrbk.conf*(5).
+
+
+EXIT STATUS
+-----------
+
+*lsbtr* returns the following error codes:
+
+ifndef::backend-docbook,backend-manpage[]
+[horizontal]
+endif::backend-docbook,backend-manpage[]
+0::   No problems occurred.
+1::   Generic error code.
+2::   Parse error: when parsing command-line options or configuration
+      file.
+255:: Script error.
+
+
+AVAILABILITY
+------------
+
+Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
+for further details.
+
+
+SEE ALSO
+--------
+
+*btrbk*(1),
+*btrbk.conf*(5),
+*btrfs*(8)
+
+
+AUTHOR
+------
+
+Axel Burri <axel@tty0.ch>

doc/ssh_filter_btrbk.1.asciidoc

@@ -0,0 +1,117 @@
+ssh_filter_btrbk(1)
+===================
+:date: 2023-03-25
+:release-version: 0.32.6
+:man manual: Btrbk Manual
+:man source: Btrbk {release-version}
+
+
+NAME
+----
+
+ssh_filter_btrbk - ssh command filter script for btrbk
+
+
+SYNOPSIS
+--------
+
+[verse]
+ssh_filter_btrbk.sh [-s|--source] [-t|--target] [-d|--delete]
+    [-i|--info] [--snapshot] [--send] [--receive]
+    [-p|--restrict-path <path>] [-l|--log] [--sudo]
+
+
+DESCRIPTION
+-----------
+
+*ssh_filter_btrbk.sh* restricts SSH commands to commands used by
+'btrbk'. It examines the SSH_ORIGINAL_COMMAND environment variable
+(set by sshd) and executes it only if it contains commands used by
+'btrbk'.
+
+The accepted commands are specified by the "--source", "--target",
+"--delete" and "--info" options.
+
+The following commands are always allowed:
+
+ - "btrfs subvolume show" (not affected by "--restrict-path")
+ - "btrfs subvolume list" (not affected by "--restrict-path")
+ - "readlink"
+ - "test -d" (only if "compat busybox" configuration option is set)
+ - "cat /proc/self/mountinfo"
+ - pipes through "gzip", "pigz", "bzip2", "pbzip2", "bzip3", "xz",
+   "lzop", "lz4", "zstd" (stream_compress)
+ - pipes through "mbuffer" (stream_buffer, rate_limit)
+
+Example line in /root/.ssh/authorized_keys on a backup target host:
+
+    command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup",restrict ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@example.org
+
+
+OPTIONS
+-------
+
+-s, --source::
+    Allow commands for backup source: "btrfs subvolume snapshot",
+    "btrfs send". Equivalent to "--snapshot --send".
+
+-t, --target::
+    Allow commands for backup and archive target: "btrfs receive",
+    "mkdir".
+
+-d, --delete::
+    Allow commands for subvolume deletion: "btrfs subvolume
+    delete". This is used for backup source if
+    'snapshot_preserve_daily' is not set to ``all'', and for backup
+    targets if 'target_preserve_daily' is not set to ``all''.
+
+-i, --info::
+    Allow informative commands: "btrfs subvolume find-new", "btrfs
+    filesystem usage". This is used by btrbk 'info' and 'diff'
+    commands.
+
+--snapshot::
+    Allow btrfs snapshot command: "btrfs subvolume snapshot".
+
+--send::
+    Allow btrfs send command: "btrfs send".
+
+--receive::
+    Allow btrfs receive command: "btrfs receive".
+
+-p, --restrict-path <path>::
+    Restrict commands to <path>. Note that "btrfs subvolume show",
+    "btrfs subvolume list" are NOT affected by this option.
++
+It is not possible to restrict commands to exact subvolume names, as
+btrfs-receive(8) takes a <path> as argument (directory, not including
+the subvolume file name to be created, this is encoded in the
+send-stream).
+
+-l, --log::
+    Log ACCEPT and REJECT messages to the system log.
+
+--sudo::
+    Allow btrfs commands to be called via sudo. Enable this if you
+    have "backend btrfs-progs-sudo" in your btrbk configuration file.
+
+
+AVAILABILITY
+------------
+
+Please refer to the btrbk project page *<https://digint.ch/btrbk/>*
+for further details.
+
+
+SEE ALSO
+--------
+
+*btrbk*(1),
+*btrbk.conf*(5),
+*btrfs*(8)
+
+
+AUTHOR
+------
+
+Axel Burri <axel@tty0.ch>

lsbtr

@@ -0,0 +1 @@
+btrbk

ssh_filter_btrbk.sh

@@ -1,45 +1,219 @@
 #!/bin/sh
 
-export PATH=/sbin:/bin:/usr/sbin:/usr/bin
+# initialise and sanitise the shell execution environment
+unset -v IFS
+export LC_ALL=C
+export PATH='/sbin:/bin:/usr/sbin:/usr/bin'
 
-if [ "$1" = "-l" ]; then
-    enable_log=1
-fi
+set -e -u
+
+enable_log=
+restrict_path_list=
+allow_list=
+allow_exact_list=
+allow_rate_limit=1
+allow_stream_buffer=1
+allow_compress=1
+compress_list='gzip|pigz|bzip2|pbzip2|bzip3|xz|lzop|lz4|zstd'
+
+# note that the backslash is NOT a metacharacter in a POSIX bracket expression!
+option_match='-[a-zA-Z0-9=-]+'   # matches short as well as long options
+file_match_sane='/[0-9a-zA-Z_@+./-]*' # matches file path (equal to ${file_match} in btrbk < 0.32.0)
+file_match="/[^']*" # btrbk >= 0.32.0 quotes file arguments: match all but single quote
+file_arg_match="('${file_match}'|${file_match_sane})" # support btrbk < 0.32.0
+
+log_cmd()
+{
+    local priority="$1"
+    local authorisation_decision="$2"
+    local reason="${3-}"
+
+    if [ -n "${enable_log}" ]; then
+        logger -p "${priority}" -t ssh_filter_btrbk.sh "${authorisation_decision} (Name: ${LOGNAME:-<unknown>}; Connection: ${SSH_CONNECTION:-<unknown>})${reason:+: ${reason}}: ${SSH_ORIGINAL_COMMAND}"
+    fi
+}
+
+allow_cmd()
+{
+    local cmd="$1"
+
+    allow_list="${allow_list}|${cmd}"
+}
+
+allow_exact_cmd()
+{
+    local cmd="$1"
+
+    allow_exact_list="${allow_exact_list}|${cmd}"
+}
 
 reject_and_die()
 {
-    if [ -n "$enable_log" ]; then
-        /usr/bin/logger -p auth.err -t ssh_filter_btrbk.sh "$LOGNAME $SSH_CLIENT REJECT: $SSH_ORIGINAL_COMMAND"
-    fi
-    /bin/echo "ERROR: ssh command rejected" 1>&2;
-    exit 1;
+    local reason="$1"
+
+    log_cmd 'auth.err' 'btrbk REJECT' "${reason}"
+    printf 'ERROR: ssh_filter_btrbk.sh: ssh command rejected: %s: %s\n' "${reason}" "${SSH_ORIGINAL_COMMAND}" >&2
+    exit 255
 }
 
 run_cmd()
 {
-    if [ -n "$enable_log" ]; then
-        /usr/bin/logger -p auth.info -t ssh_filter_btrbk.sh "$LOGNAME $SSH_CLIENT ALLOW: $SSH_ORIGINAL_COMMAND"
-    fi
-    $SSH_ORIGINAL_COMMAND
+    log_cmd 'auth.info' 'btrbk ACCEPT'
+    eval " ${SSH_ORIGINAL_COMMAND}"
 }
 
-case "$SSH_ORIGINAL_COMMAND" in
-    *\$*) reject_and_die ;;
-    *\&*) reject_and_die ;;
-    *\(*) reject_and_die ;;
-    *\{*) reject_and_die ;;
-    *\;*) reject_and_die ;;
-    *\<*) reject_and_die ;;
-    *\>*) reject_and_die ;;
-    *\`*) reject_and_die ;;
-    *\|*) reject_and_die ;;
-    btrfs\ subvolume\ show\ *)      run_cmd ;;   # mandatory
-    btrfs\ subvolume\ list\ *)      run_cmd ;;   # mandatory
-    btrfs\ subvolume\ snapshot\ *)  run_cmd ;;   # mandatory if this host is backup source
-    btrfs\ send\ *)                 run_cmd ;;   # mandatory if this host is backup source
-    btrfs\ receive\ *)              run_cmd ;;   # mandatory if this host is backup target
-    btrfs\ subvolume\ delete\ *)    run_cmd ;;   # mandatory if scheduling is active
-    btrfs\ subvolume\ find-new\ *)  run_cmd ;;   # needed for "btrbk diff"
-    btrfs\ filesystem\ usage\ *)    run_cmd ;;   # needed for "btrbk info"
-    *) reject_and_die ;;
+reject_filtered_cmd()
+{
+    if [ -n "${restrict_path_list}" ]; then
+	# match any of restrict_path_list,
+	# or any file/directory (matching file_match) below restrict_path
+	path_match="'(${restrict_path_list})(${file_match})?'"
+	path_match_legacy="(${restrict_path_list})(${file_match_sane})?"
+    else
+	# match any absolute file/directory (matching file_match)
+	path_match="'${file_match}'"
+	path_match_legacy="${file_match_sane}"
+    fi
+    # btrbk >= 0.32.0 quotes files, allow both (legacy)
+    path_match="(${path_match}|${path_match_legacy})"
+
+    if [ -n "${allow_compress}" ]; then
+        decompress_match="(${compress_list}) -d -c( -[pT][0-9]+)?"
+        compress_match="(${compress_list}) -c( -[0-9])?( -[pT][0-9]+)?"
+    else
+        decompress_match=
+        compress_match=
+    fi
+
+    # rate_limit_remote and stream_buffer_remote use combined
+    # "mbuffer" as of btrbk-0.29.0
+    if [ -n "${allow_stream_buffer}" ] || [ -n "${allow_rate_limit}" ]; then
+        mbuffer_match='mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?'
+    else
+        mbuffer_match=
+    fi
+
+    # allow multiple paths (e.g. "btrfs subvolume snapshot <src> <dst>")
+    allow_cmd_match="(${allow_list})( ${option_match})*( ${path_match})+"
+    stream_in_match="(${decompress_match} \| )?(${mbuffer_match} \| )?"
+    stream_out_match="( \| ${mbuffer_match})?( \| ${compress_match}$)?"
+
+    # `grep`’s `-q`-option is not used as it may cause an exit status of `0` even
+    # when an error occurred.
+
+    allow_stream_match="^${stream_in_match}${allow_cmd_match}${stream_out_match}"
+    if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${allow_stream_match}" >/dev/null 2>/dev/null; then
+        return 0
+    fi
+
+    exact_cmd_match="^(${allow_exact_list})$";
+    if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${exact_cmd_match}" >/dev/null 2>/dev/null; then
+        return 0
+    fi
+
+    local formatted_restrict_path_list="$(printf '%s' "${restrict_path_list}" | sed 's/|/", "/g')"
+    reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${formatted_restrict_path_list}\")}"
+}
+
+
+# check for "--sudo" option before processing other options
+sudo_prefix=
+for key in "$@"; do
+    if [ "${key}" = '--sudo' ]; then
+        sudo_prefix='sudo -n '
+    fi
+    if [ "${key}" = '--doas' ]; then
+        sudo_prefix='doas -n '
+    fi
+done
+
+while [ "$#" -ge 1 ]; do
+    key="$1"
+
+    case "${key}" in
+      -l|--log)
+          enable_log=1
+          ;;
+
+      --sudo|--doas)
+          # already processed above
+          ;;
+
+      -p|--restrict-path)
+          restrict_path_list="${restrict_path_list}|${2%/}"  # add to list while removing trailing slash
+          shift # past argument
+          ;;
+
+      -s|--source)
+          allow_cmd "${sudo_prefix}btrfs subvolume snapshot"
+          allow_cmd "${sudo_prefix}btrfs send"
+          ;;
+
+      -t|--target)
+          allow_cmd "${sudo_prefix}btrfs receive"
+          allow_cmd "${sudo_prefix}mkdir"
+          ;;
+
+      -c|--compress)
+          # deprecated option, compression is always allowed
+          ;;
+
+      -d|--delete)
+          allow_cmd "${sudo_prefix}btrfs subvolume delete"
+          ;;
+
+      -i|--info)
+          allow_cmd "${sudo_prefix}btrfs subvolume find-new"
+          allow_cmd "${sudo_prefix}btrfs filesystem usage"
+          ;;
+
+      --snapshot)
+          allow_cmd "${sudo_prefix}btrfs subvolume snapshot"
+          ;;
+
+      --send)
+          allow_cmd "${sudo_prefix}btrfs send"
+          ;;
+
+      --receive)
+          allow_cmd "${sudo_prefix}btrfs receive"
+          ;;
+
+      *)
+          printf 'ERROR: ssh_filter_btrbk.sh: failed to parse command line option: %s\n' "${key}" >&2
+          exit 255
+          ;;
+    esac
+    shift
+done
+
+# NOTE: subvolume queries are NOT affected by "--restrict-path":
+# btrbk also calls show/list on the mount point of the subvolume
+allow_exact_cmd "${sudo_prefix}btrfs subvolume (show|list)( ${option_match})* ${file_arg_match}";
+allow_cmd "${sudo_prefix}readlink"                    # resolve symlink
+allow_exact_cmd "${sudo_prefix}test -d ${file_arg_match}" # check directory (only for compat=busybox)
+allow_exact_cmd 'cat /proc/self/mountinfo'            # resolve mountpoints
+allow_exact_cmd 'cat /proc/self/mounts'               # legacy, for btrbk < 0.27.0
+
+# remove leading "|" on alternation lists
+allow_list="${allow_list#\|}"
+allow_exact_list="${allow_exact_list#\|}"
+restrict_path_list="${restrict_path_list#\|}"
+
+case "${SSH_ORIGINAL_COMMAND}" in
+    *\.\./*)  reject_and_die 'directory traversal'  ;;
+    *'
+'*)           reject_and_die 'unsafe character LF'  ;;
+    *\$*)     reject_and_die 'unsafe character "$"' ;;
+    *\&*)     reject_and_die 'unsafe character "&"' ;;
+    *\(*)     reject_and_die 'unsafe character "("' ;;
+    *\{*)     reject_and_die 'unsafe character "{"' ;;
+    *\;*)     reject_and_die 'unsafe character ";"' ;;
+    *\<*)     reject_and_die 'unsafe character "<"' ;;
+    *\>*)     reject_and_die 'unsafe character ">"' ;;
+    *\`*)     reject_and_die 'unsafe character "`"' ;;
+    *\|*)     [ -n "${allow_compress}" ] || [ -n "${allow_rate_limit}" ] || [ -n "${allow_stream_buffer}" ] || reject_and_die 'unsafe character "|"' ;;
 esac
+
+reject_filtered_cmd
+run_cmd