Compare commits

..

No commits in common. "master" and "v0.31.2" have entirely different histories.

22 changed files with 2115 additions and 2326 deletions

3
.github/FUNDING.yml vendored
View File

@ -1,3 +0,0 @@
# These are supported funding model platforms
custom: ['https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S', 'https://btc.com/19DYtoEepxBmn9ZPspJGZrhCtySKCxPcP1']

View File

@ -1,26 +0,0 @@
---
name: Codespell
on:
push:
branches: [master]
pull_request:
branches: [master]
permissions:
contents: read
jobs:
codespell:
name: Check for spelling errors
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Codespell
uses: codespell-project/actions-codespell@v2
with:
check_filenames: true
skip: ".git,*.pdf,*.svg"
ignore_words_list: uptodate

View File

@ -1,87 +1,9 @@
btrbk-0.32.6
* Fix backup of unrelated (by parent_uuid) snapshots (close #339).
* Remove echo -e for portability (close #506).
* Support btrfs send protocol v2 (send_protocol and
send_compressed_data config options).
* Add bzip3 support.
* Convert ssh_filter_btrbk.sh to POSIX sh, and harden it.
* Slight change in ssh_filter_btrbk.sh logging output.
* Minor bugfixes, stability and documentation improvements.
btrbk-0.32.5
* Correct handling of zero-size raw info file (close #491).
btrbk-0.32.4
* Fix regression: wrong deprecation warnings in some cases.
btrbk-0.32.3
* Fix deletion of many subvolumes at once (close #476).
* Allow disabling ssh_identity and ssh_user options.
* Minor bugfixes and documentation improvements.
btrbk-0.32.2
* Fix regression: archive on missing target directories.
* Fix action "config print".
* Print version and help message to stdout (close #452).
* Check results of filesystem usage (close #469).
* Add "compat ignore_receive_errors" option (close #383).
* Support multiple gpg recipients (close #471).
* Fix changelog: remove "compat missing_otime" item in v0.32.1.
* Fix changelog: correct "warn_unknown_targets" item in v0.31.3.
* Minor bugfixes and framework improvements.
btrbk-0.32.1
* Fix regression: correctly read mountinfo (close #445).
* Fix regression: filter paths for "lsbtr" (action "ls").
* Add backend btrfs-progs-doas (close #444).
* Allow directory traversal for local command line arguments.
* Display all source subvolumes in "usage" action.
* Handle errors from btrfs filesystem usage.
btrbk-0.32.0
* MIGRATION
- If timestamp_format is not configured, explicitly set
"timestamp_format short" to revert old behavior.
- Update ssh_filter_btrbk.sh on remote hosts.
* Change default for timestamp_format to "long".
* Optional "volume" config section.
* Use "volume" section only for grouping and relative paths.
* Allow absolute path for "subvolume" and "snapshot_dir"
(close #407, #121).
* Support subvolume names with UTF-8 characters (close #392, #213,
#120). Add "safe_commands" option for paranoid people.
* Add "incremental_prefs" configuration option (close #387).
* Change incremental prefs policy, giving snapshots created by btrbk
higher preference than the global ones resolved by parent-uuid.
* Change "incremental_clones" option to boolean.
* Drop support of deprecated raw file format (btrbk < 0.26.0).
* Drop support of deprecated "-r, --resume-only" cmdline option.
* Fix ambiguous naming in --format=raw output of "list" actions.
* Accept quoted values in config.
* Use single quotes for raw table output.
btrbk-0.31.3
* Add "warn_unknown_targets" configuration option (close #393).
* Add -1,--single-column command-line option for listing actions.
* Allow relative path command line argument for all actions.
* Add support for zstd adaptive compression (close #397).
* Add contrib/tools/btrbk_restore_raw.py (close #401).
* Minor bugfixes and framework improvements.
btrbk-0.31.2 btrbk-0.31.2
* MIGRATION * MIGRATION
- Update ssh_filter_btrbk.sh on remote hosts. - Update ssh_filter_btrbk.sh on remote hosts.
* ssh_filter_btrbk.sh: Fix security vulnerability. * ssh_filter_btrbk.sh: Fix security vulnerability.
Specially crafted commands may be executed without being properly Specialy crafted commands may be executed without being propely
checked. Applies to remote hosts filtering ssh commands using checked. Applies to remote hosts filtering ssh commands using
ssh_filter_btrbk.sh in authorized_keys. ssh_filter_btrbk.sh in authorized_keys.
* Warn if no subvolume defined in config (close #378). * Warn if no subvolume defined in config (close #378).
@ -248,7 +170,7 @@ btrbk-0.27.0
- Allow snapshot_dir to be a mountpoint. - Allow snapshot_dir to be a mountpoint.
- Search complete target tree for correlated subvolumes. - Search complete target tree for correlated subvolumes.
- Include snapshots from all mountpoints as candidates (disabled - Include snapshots from all mountpoints as candidates (disabled
due to upstream bug: github.com/kdave/btrfs-progs/issues/96). due to uptream bug: github.com/kdave/btrfs-progs/issues/96).
- Read /proc/self/mountinfo instead of /proc/self/mounts. - Read /proc/self/mountinfo instead of /proc/self/mounts.
- Always read /proc/self/mountinfo. - Always read /proc/self/mountinfo.
- Resolve realpath using readlink(1). - Resolve realpath using readlink(1).
@ -319,7 +241,7 @@ btrbk-0.25.0
* Allow trailing comments in btrbk.conf (close #129). * Allow trailing comments in btrbk.conf (close #129).
* Bugfix: rate limiting must be done after compression (close #134). * Bugfix: rate limiting must be done after compression (close #134).
* raw_target_encrypt: Always set "gpg --no-random-seed-file": * raw_target_encrypt: Always set "gpg --no-random-seed-file":
prevents creation of "~/.gnupg/random_seed" with slight performance prevents creation of "~/.gnupg/random_seed" with slight perfomance
penalty. penalty.
btrbk-0.24.0 btrbk-0.24.0

View File

@ -17,8 +17,7 @@ SCRIPTS = ssh_filter_btrbk.sh \
contrib/cron/btrbk-mail \ contrib/cron/btrbk-mail \
contrib/cron/btrbk-verify \ contrib/cron/btrbk-verify \
contrib/migration/raw_suffix2sidecar \ contrib/migration/raw_suffix2sidecar \
contrib/crypt/kdf_pbkdf2.py \ contrib/crypt/kdf_pbkdf2.py
contrib/tools/btrbk_restore_raw.py
PN = btrbk PN = btrbk
PREFIX ?= /usr PREFIX ?= /usr

114
README.md
View File

@ -9,7 +9,7 @@ The source and target locations are specified in a config file, which
allows to easily configure simple scenarios like "laptop with locally allows to easily configure simple scenarios like "laptop with locally
attached backup disks", as well as more complex ones, e.g. "server attached backup disks", as well as more complex ones, e.g. "server
receiving backups from several hosts via ssh, with different retention receiving backups from several hosts via ssh, with different retention
policies". policy".
Key Features: Key Features:
@ -18,8 +18,7 @@ Key Features:
* Flexible retention policy * Flexible retention policy
* Backups to multiple destinations * Backups to multiple destinations
* Transfer via ssh * Transfer via ssh
* Robust recovery from interrupted backups (for removable and mobile * Resume backups (for removable and mobile devices)
devices)
* Archive to offline storage * Archive to offline storage
* Encrypted backups to non-btrfs storage * Encrypted backups to non-btrfs storage
* Wildcard subvolumes (useful for docker and lxc containers) * Wildcard subvolumes (useful for docker and lxc containers)
@ -60,7 +59,7 @@ For more information, read the [installation documentation].
[btrfs-progs]: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/ [btrfs-progs]: https://www.kernel.org/pub/linux/kernel/people/kdave/btrfs-progs/
[Perl interpreter]: https://www.perl.org [Perl interpreter]: https://www.perl.org
[OpenSSH]: https://www.openssh.com [OpenSSH]: https://www.openssh.org
[mbuffer]: https://www.maier-komor.de/mbuffer.html [mbuffer]: https://www.maier-komor.de/mbuffer.html
@ -73,8 +72,8 @@ a full description of the command line options.
[btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html [btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
Configuration Configuration File
============= ==================
Before running `btrbk`, you will need to create a configuration Before running `btrbk`, you will need to create a configuration
file. You might want to take a look at `btrbk.conf.example` provided file. You might want to take a look at `btrbk.conf.example` provided
@ -82,37 +81,21 @@ with this package. For a detailed description, please consult the
[btrbk.conf(5)] man-page. [btrbk.conf(5)] man-page.
After a configuration change, it is highly recommended to check it by After a configuration change, it is highly recommended to check it by
running btrbk with the `-n,--dry-run` option: running btrbk with the `-n,--dryrun` option:
# btrbk -c /path/to/myconfig -v -n run btrbk -c /path/to/myconfig -v -n run
This will read all btrfs information on the source/target filesystems This will read all btrfs information on the source/target filesystems
and show what actions would be performed (without writing anything to and show what actions would be performed (without writing anything to
the disks). the disks).
The examples below assume that the btrfs subvolume containing `home`
and `rootfs` is mounted at `/mnt/btr_pool`. This is usually the btrfs
root subvolume, which always has `subvolid=5`.
Mounting `subvolid=5` is *recommended* (mandatory for btrbk < v0.32.0)
if you want to backup your root filesystem `/`.
/etc/fstab:
/dev/sda1 /mnt/btr_pool btrfs subvolid=5,noatime 0 0
Note that some default btrfs installations (e.g. Ubuntu) use subvolume
names `@` for rootfs (mounted at `/`) and `@home` for `/home`, as a
naming convention. If this is the case on your file system, replace
the `subvolume` declarations in the examples accordingly.
[btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html [btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html
Example: Local Regular Snapshots (time-machine) Example: Local Regular Snapshots (time-machine)
----------------------------------------------- -----------------------------------------------
The simplest use case is to only create snapshots of your data. This The simpliest use case is to only create snapshots of your data. This
will obviously not protect it against hardware failure, but can be will obviously not protect it against hardware failure, but can be
useful for: useful for:
@ -138,32 +121,15 @@ manage snapshots located on the same volume in `snapshot_dir`. Btrbk
does not create subdirs by default, the snapshot directory must first does not create subdirs by default, the snapshot directory must first
be created manually: be created manually:
# mkdir /mnt/btr_pool/btrbk_snapshots sudo mkdir /mnt/btr_pool/btrbk_snapshots
The "volume" section is merely used as a specifier for a base Start a dry run:
directory, and can be skipped if you prefer to configure everything
using absolute paths. The above configuration can also be written as:
snapshot_dir /mnt/btr_pool/btrbk_snapshots sudo btrbk run -n
subvolume /mnt/btr_pool/home
If you don't want to mount the btrfs root filesystem to
`/mnt/btr_pool`, you might as well configure it like this:
snapshot_dir /btrbk_snapshots
subvolume /home
Start a dry run (-n, --dry-run):
# btrbk run -n
Create the first snapshot: Create the first snapshot:
# btrbk run sudo btrbk run
Print schedule (-S, --print-schedule):
# btrbk run -n -S
If it works as expected, configure a cron job to run btrbk hourly: If it works as expected, configure a cron job to run btrbk hourly:
@ -172,10 +138,13 @@ If it works as expected, configure a cron job to run btrbk hourly:
#!/bin/sh #!/bin/sh
exec /usr/bin/btrbk -q run exec /usr/bin/btrbk -q run
Snapshots will now be created every hour. All snapshots are preserved for at Snapshots will now be created every hour, kept for 48h
least 18 hours (`snapshot_preserve_min`), whether they are created by the cron (`snapshot_preserve`), then automatically removed.
job or manually by calling `sudo btrbk run` on the command line. Additionally,
48 hourly snapshots are preserved (`snapshot_preserve`). With this setup, the snapshots will be kept at least for 18 hours
(`snapshot_preserve_min`). This can be useful to create manual
snapshots by calling `sudo btrbk run` on the command line and keep
them around for a while, in addition to the regular snapshots.
Example: Backups to USB Disk Example: Backups to USB Disk
@ -208,9 +177,6 @@ Retention policy:
snapshot_preserve_min 2d snapshot_preserve_min 2d
snapshot_preserve 14d snapshot_preserve 14d
# Create snapshots only if the backup disk is attached
#snapshot_create ondemand
target_preserve_min no target_preserve_min no
target_preserve 20d 10w *m target_preserve 20d 10w *m
@ -236,11 +202,10 @@ Retention policy:
* `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD` * `/mnt/btr_backup/mylaptop/rootfs.YYYYMMDD`
* `/mnt/btr_backup/mylaptop/home.YYYYMMDD` * `/mnt/btr_backup/mylaptop/home.YYYYMMDD`
If you prefer triggering the backups manually, change the cron command If you want the snapshots to be created only if the backup disk is
to run the `snapshot` action instead of `run`. Start the backups attached, simply add the following line to the config:
manually by running:
# btrbk resume snapshot_create ondemand
For a quick additional snapshot of your home, run: For a quick additional snapshot of your home, run:
@ -250,8 +215,8 @@ For a quick additional snapshot of your home, run:
Example: Host-initiated Backup on Fileserver Example: Host-initiated Backup on Fileserver
-------------------------------------------- --------------------------------------------
Let's say you have a fileserver at "myserver.example.org" where you Let's say you have a fileserver at "myserver.mydomain.com" where you
want to create backups of your laptop disk. The config could look like want to create backups of your laptop disk, the config would look like
this: this:
ssh_identity /etc/btrbk/ssh/id_rsa ssh_identity /etc/btrbk/ssh/id_rsa
@ -259,11 +224,11 @@ this:
volume /mnt/btr_pool volume /mnt/btr_pool
subvolume rootfs subvolume rootfs
target /mnt/btr_backup/mylaptop target /mnt/btr_backup/mylaptop
target ssh://myserver.example.org/mnt/btr_backup/mylaptop target ssh://myserver.mydomain.com/mnt/btr_backup/mylaptop
In addition to the backups on your local usb-disk mounted at In addition to the backups on your local usb-disk mounted at
`/mnt/btr_backup/mylaptop`, incremental backups would also be pushed `/mnt/btr_backup/mylaptop`, incremental backups would also be pushed
to `myserver.example.org`. to `myserver.mydomain.com`.
Example: Fileserver-initiated Backups from Several Hosts Example: Fileserver-initiated Backups from Several Hosts
@ -274,17 +239,17 @@ fileserver, the config would be something like:
ssh_identity /etc/btrbk/ssh/id_rsa ssh_identity /etc/btrbk/ssh/id_rsa
volume ssh://alpha.example.org/mnt/btr_pool volume ssh://alpha.mydomain.com/mnt/btr_pool
target /mnt/btr_backup/alpha target /mnt/btr_backup/alpha
subvolume rootfs subvolume rootfs
subvolume home subvolume home
volume ssh://beta.example.org/mnt/btr_pool volume ssh://beta.mydomain.com/mnt/btr_pool
target /mnt/btr_backup/beta target /mnt/btr_backup/beta
subvolume rootfs subvolume rootfs
subvolume dbdata subvolume dbdata
This will pull backups from alpha/beta.example.org and locally This will pull backups from alpha/beta.mydomain.com and locally
create: create:
* `/mnt/btr_backup/alpha/rootfs.YYYYMMDD` * `/mnt/btr_backup/alpha/rootfs.YYYYMMDD`
@ -385,7 +350,7 @@ running btrbk. Something like:
rsync -az --delete \ rsync -az --delete \
--inplace --numeric-ids --acls --xattrs \ --inplace --numeric-ids --acls --xattrs \
-e 'ssh -i /etc/btrbk/ssh/id_rsa' \ -e 'ssh -i /etc/btrbk/ssh/id_rsa' \
myhost.example.org:/data/ \ myhost.mydomain.com:/data/ \
/mnt/btr_backup/myhost_sync/ /mnt/btr_backup/myhost_sync/
exec /usr/bin/btrbk -q run exec /usr/bin/btrbk -q run
@ -412,7 +377,7 @@ compressed and piped through GnuPG.
raw_target_compress xz raw_target_compress xz
raw_target_encrypt gpg raw_target_encrypt gpg
gpg_keyring /etc/btrbk/gpg/pubring.gpg gpg_keyring /etc/btrbk/gpg/pubring.gpg
gpg_recipient btrbk@example.org gpg_recipient btrbk@mydomain.com
volume /mnt/btr_pool volume /mnt/btr_pool
subvolume home subvolume home
@ -428,7 +393,7 @@ host. For each backup, two files are created:
* `/backup/home.YYYYMMDD.btrfs.xz.gpg.info`: sidecar file containing * `/backup/home.YYYYMMDD.btrfs.xz.gpg.info`: sidecar file containing
metadata used by btrbk. metadata used by btrbk.
If you are using raw _incremental_ backups, please make sure you I you are using raw _incremental_ backups, please make sure you
understand the implications (see [btrbk.conf(5)], TARGET TYPES). understand the implications (see [btrbk.conf(5)], TARGET TYPES).
@ -449,7 +414,7 @@ will need the `btrfs` executable from the [btrfs-progs] package.
On the client side, create a ssh key dedicated to btrbk, without On the client side, create a ssh key dedicated to btrbk, without
password protection: password protection:
# ssh-keygen -t rsa -b 4096 -f /etc/btrbk/ssh/id_rsa -C btrbk@example.org -N "" ssh-keygen -t rsa -b 4096 -f /etc/btrbk/ssh/id_rsa -C btrbk@mydomain.com -N ""
The content of the public key (/etc/btrbk/ssh/id_rsa.pub) is used for The content of the public key (/etc/btrbk/ssh/id_rsa.pub) is used for
authentication in "authorized_keys" on the server side (see [sshd(8)] authentication in "authorized_keys" on the server side (see [sshd(8)]
@ -555,14 +520,14 @@ to run it whenever the key is used for authentication. Example
"/root/.ssh/authorized_keys": "/root/.ssh/authorized_keys":
# example backup source (also allowing deletion of old snapshots) # example backup source (also allowing deletion of old snapshots)
command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete",restrict <pubkey>... command="/backup/scripts/ssh_filter_btrbk.sh -l --source --delete" <pubkey>...
# example backup target (also allowing deletion of old snapshots) # example backup target (also allowing deletion of old snapshots)
command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete",restrict <pubkey>... command="/backup/scripts/ssh_filter_btrbk.sh -l --target --delete" <pubkey>...
# example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no), # example fetch-only backup source (snapshot_preserve_min=all, snapshot_create=no),
# restricted to subvolumes within /home or /data # restricted to subvolumes within /home or /data
command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data",restrict <pubkey>... command="/backup/scripts/ssh_filter_btrbk.sh -l --send -p /home -p /data" <pubkey>...
[ssh_filter_btrbk(1)]: https://digint.ch/btrbk/doc/ssh_filter_btrbk.1.html [ssh_filter_btrbk(1)]: https://digint.ch/btrbk/doc/ssh_filter_btrbk.1.html
@ -680,8 +645,8 @@ Donate
So btrbk saved your day? So btrbk saved your day?
I will definitively continue to develop btrbk for free. If you want to I will definitively continue developing btrbk for free, but if you
support my hard work with a donation, you are welcome to do so! want to support me you can do so:
[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S) [![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=WFQSSCD9GNM4S)
@ -712,7 +677,7 @@ If you would like to contribute or have found bugs:
* Visit the [btrbk project page on GitHub] and use the * Visit the [btrbk project page on GitHub] and use the
[issues tracker] there. [issues tracker] there.
* Talk to us on [Libera.Chat] in `#btrbk`. * Talk to us on Freenode in `#btrbk`.
* Contact the author via email (the email address can be found in * Contact the author via email (the email address can be found in
the sources). the sources).
@ -720,7 +685,6 @@ Any feedback is appreciated!
[btrbk project page on GitHub]: https://github.com/digint/btrbk [btrbk project page on GitHub]: https://github.com/digint/btrbk
[issues tracker]: https://github.com/digint/btrbk/issues [issues tracker]: https://github.com/digint/btrbk/issues
[Libera.Chat]: https://libera.chat
License License

2815
btrbk

File diff suppressed because it is too large Load Diff

View File

@ -17,13 +17,6 @@
# Enable transaction log # Enable transaction log
transaction_log /var/log/btrbk.log transaction_log /var/log/btrbk.log
# Specify SSH private key for remote connections
ssh_identity /etc/btrbk/ssh/id_ed25519
ssh_user root
# Use sudo if btrbk or lsbtr is run by regular user
backend_local_user btrfs-progs-sudo
# Enable stream buffer. Adding a buffer between the sending and # Enable stream buffer. Adding a buffer between the sending and
# receiving side is generally a good idea. # receiving side is generally a good idea.
# NOTE: If enabled, make sure to install the "mbuffer" package! # NOTE: If enabled, make sure to install the "mbuffer" package!
@ -36,7 +29,7 @@ stream_buffer 256m
# If you want to set a custom name for the snapshot (and backups), # If you want to set a custom name for the snapshot (and backups),
# use the "snapshot_name" option within the subvolume section. # use the "snapshot_name" option within the subvolume section.
# #
# NOTE: btrbk does not automatically create this directory, and the # NOTE: btrbk does not autmatically create this directory, and the
# snapshot creation will fail if it is not present. # snapshot creation will fail if it is not present.
# #
snapshot_dir _btrbk_snap snapshot_dir _btrbk_snap
@ -74,6 +67,12 @@ snapshot_dir _btrbk_snap
#archive_preserve_min no #archive_preserve_min no
#archive_preserve <NN>h <NN>d <NN>w <NN>m <NN>y #archive_preserve <NN>h <NN>d <NN>w <NN>m <NN>y
# Specify SSH private key for "ssh://" volumes / targets:
#ssh_identity /etc/btrbk/ssh/id_ed25519
#ssh_user root
#ssh_compression no
#ssh_cipher_spec default
# Enable compression for remote btrfs send/receive operations: # Enable compression for remote btrfs send/receive operations:
#stream_compress no #stream_compress no
#stream_compress_level default #stream_compress_level default
@ -83,19 +82,19 @@ snapshot_dir _btrbk_snap
# can be run at a time. # can be run at a time.
#lockfile /var/lock/btrbk.lock #lockfile /var/lock/btrbk.lock
# Don't wait for transaction commit on deletion. Enable this to make # Don't wait for transaction commit on deletion. Set this to "after"
# sure the deletion of subvolumes is committed to disk when btrbk # or "each" to make sure the deletion of subvolumes is committed to
# terminates. # disk when btrbk terminates.
#btrfs_commit_delete no #btrfs_commit_delete no
# #
# Volume section (optional): "volume <volume-directory>" # Volume section: "volume <volume-directory>"
# #
# <volume-directory> Base path within a btrfs filesystem # <volume-directory> Directory of a btrfs volume (or subvolume)
# containing the subvolumes to be backuped # containing the subvolume to be backuped
# (usually the mount-point of a btrfs filesystem # (usually the mount-point of a btrfs filesystem
# mounted with subvolid=5 option). # mounted with subvolid=5 option)
# #
# Subvolume section: "subvolume <subvolume-name>" # Subvolume section: "subvolume <subvolume-name>"
# #
@ -105,19 +104,19 @@ snapshot_dir _btrbk_snap
# Target section: "target <type> <volume-directory>" # Target section: "target <type> <volume-directory>"
# #
# <type> (optional) type, defaults to "send-receive". # <type> (optional) type, defaults to "send-receive".
# <volume-directory> Directory within a btrfs filesystem # <volume-directory> Directory of a btrfs volume (or subvolume)
# receiving the backups. # receiving the backups.
# #
# NOTE: The parser does not care about indentation, this is only for # NOTE: The parser does not care about indentation, this is only for
# human readability. All options apply to the last section # human readability. The options always apply to the last section
# encountered, overriding the corresponding option of the upper # encountered, overriding the corresponding option of the upper
# section. This means that the global options must be set on top, # section. This means that the global options must be set before any
# before any "volume", "subvolume" or "target section. # "volume" section.
# #
# #
# Example retention policy: # Example configuration:
# #
snapshot_preserve_min 2d snapshot_preserve_min 2d
snapshot_preserve 14d snapshot_preserve 14d
@ -125,49 +124,29 @@ snapshot_preserve 14d
target_preserve_min no target_preserve_min no
target_preserve 20d 10w *m target_preserve 20d 10w *m
archive_preserve_min latest
archive_preserve 12m 10y
#
# Simple setup: Backup root and home to external disk
#
snapshot_dir /btrbk_snapshots
target /mnt/btr_backup
subvolume /
subvolume /home
#
# Complex setup
#
# In order to keep things organized, it is recommended to use "volume"
# sections and mount the top-level subvolume (subvolid=5):
#
# $ mount -o subvolid=5 /dev/sda1 /mnt/btr_pool
#
# Backup to external disk mounted on /mnt/btr_backup # Backup to external disk mounted on /mnt/btr_backup
volume /mnt/btr_pool volume /mnt/btr_pool
# Create snapshots in /mnt/btr_pool/btrbk_snapshots # no action if external disk is not attached
snapshot_dir btrbk_snapshots snapshot_create ondemand
# Target for all subvolume sections: # propagates to all subvolume sections:
target /mnt/btr_backup target /mnt/btr_backup/_btrbk
# Some default btrfs installations (e.g. Ubuntu) use "@" for rootfs subvolume root_gentoo
# (mounted at "/") and "@home" (mounted at "/home"). Note that this
# is only a naming convention.
#subvolume @
subvolume root
subvolume home
subvolume kvm subvolume kvm
# Use different retention policy for kvm backups: # use different retention policy for kvm backups
target_preserve 7d 4w target_preserve 7d 4w
# Backup data to external disk as well as remote host # Backup to external disk as well as some remote host
volume /mnt/btr_data volume /mnt/btr_data
subvolume data subvolume home
# Always create snapshot, even if targets are unreachable # always create snapshot, even if targets are unreachable
snapshot_create always snapshot_create always
target /mnt/btr_backup target /mnt/btr_backup/_btrbk
target ssh://backup.my-remote-host.com/mnt/btr_backup target ssh://backup.my-remote-host.com/mnt/btr_backup
@ -176,22 +155,7 @@ volume ssh://my-remote-host.com/mnt/btr_pool
subvolume data_0 subvolume data_0
snapshot_dir snapshots/btrbk snapshot_dir snapshots/btrbk
snapshot_name data_main snapshot_name data_main
target /mnt/btr_backup/my-remote-host.com target /mnt/btr_backup/_btrbk/my-remote-host.com
# Backup on demand (noauto) to remote host running busybox, login as
# regular user using ssh-agent with current user name (ssh_user no)
# and default credentials (ssh_identity no).
volume /home
noauto yes
compat busybox
backend_remote btrfs-progs-sudo
ssh_user no
ssh_identity no
target ssh://my-user-host.com/mnt/btr_backup/home
subvolume alice
subvolume bob
# Resume backups from remote host which runs its own btrbk instance # Resume backups from remote host which runs its own btrbk instance
@ -201,4 +165,4 @@ volume ssh://my-remote-host.com/mnt/btr_pool
snapshot_create no snapshot_create no
snapshot_preserve_min all snapshot_preserve_min all
subvolume home subvolume home
target /mnt/btr_backup/my-remote-host.com target /mnt/btr_backup/_btrbk/my-remote-host.com

View File

@ -4,7 +4,7 @@ _btrbk_init_cmds()
# #
# for example, for this command: # for example, for this command:
# #
# btrbk -v --override warn_unknown_targets=yes list config --long # btrbk -v list config --long
# #
# then $cmds is: # then $cmds is:
# #
@ -14,11 +14,6 @@ _btrbk_init_cmds()
local i local i
for ((i = 1; i < cword; i++)); do for ((i = 1; i < cword; i++)); do
case "${words[i-1]}" in
'-c' | '--config' | '--exclude' | '-l' | '--loglevel' | '--format' | '--lockfile' | '--override')
continue
;;
esac
[[ ${words[i]} != -* ]] && cmds+=(${words[i]}) [[ ${words[i]} != -* ]] && cmds+=(${words[i]})
done done
@ -34,28 +29,21 @@ _btrbk()
case "$prev" in case "$prev" in
'-c' | '--config') '-c' | '--config')
_filedir _filedir
return
;; ;;
'--exclude') '--exclude')
return
;; ;;
'-l' | '--loglevel') '-l' | '--loglevel')
COMPREPLY=($(compgen -W 'error warn info debug trace' -- "$cur")) COMPREPLY=($(compgen -W 'error warn info debug trace' -- "$cur"))
return
;; ;;
'--format') '--format')
COMPREPLY=($(compgen -W 'table long raw' -- "$cur")) COMPREPLY=($(compgen -W 'table long raw' -- "$cur"))
return
;; ;;
'--lockfile') '--lockfile')
_filedir _filedir
return
;; ;;
'--override') '--override')
return
;; ;;
esac esac
$split && return $split && return
if [[ $cur == -* ]]; then if [[ $cur == -* ]]; then

View File

@ -4,9 +4,6 @@
now=$(date +%Y%m%d) now=$(date +%Y%m%d)
declare -A rsync_src rsync_dst rsync_log rsync_rsh rsync_opt
declare -A sync_fs_onchange
##### start config section ##### ##### start config section #####
# Email recipients, separated by whitespace: # Email recipients, separated by whitespace:
@ -33,9 +30,7 @@ rsync_opt[example_data]="-az --delete --inplace --numeric-ids --acls --xattrs"
# If set, add "rsync_dst" to "sync_fs" (see below) if rsync reports files transferred # If set, add "rsync_dst" to "sync_fs" (see below) if rsync reports files transferred
#sync_fs_onchange[example_data]=yes #sync_fs_onchange[example_data]=yes
# Enable all rsync declarations (all indices of rsync_src array) # Enabled rsync declarations (whitespace-separated list)
#rsync_enable=${!rsync_src[@]}
# Explicitly enable rsync declarations (whitespace-separated list)
#rsync_enable="example_data" #rsync_enable="example_data"
rsync_enable= rsync_enable=
@ -60,8 +55,6 @@ btrbk_opts="-c /etc/btrbk/btrbk.conf"
#mail_cmd_block_prefix='\\u200B' # zero-width whitespace #mail_cmd_block_prefix='\\u200B' # zero-width whitespace
#mail_cmd_block_prefix=". " #mail_cmd_block_prefix=". "
# Newline character
BR=$'\n'
##### end config section ##### ##### end config section #####
@ -90,47 +83,47 @@ send_mail()
body+="$info" body+="$info"
fi fi
if [[ -n "$detail" ]] && [[ -n "$has_errors" ]] || [[ "${mail_detail:-no}" = "yes" ]]; then if [[ -n "$detail" ]] && [[ -n "$has_errors" ]] || [[ "${mail_detail:-no}" = "yes" ]]; then
[[ -n "$body" ]] && body+="${BR}${BR}DETAIL:${BR}" [[ -n "$body" ]] && body+="\n\nDETAIL:\n"
body+="$detail" body+="$detail"
fi fi
# skip sending mail on empty body # skip sending mail on empty body
if [[ -z "$body" ]] && [[ -n "$has_errors" ]]; then if [[ -z "$body" ]] && [[ -n "$has_errors" ]]; then
body+="FATAL: something went wrong (errors present but empty mail body)${BR}" body+="FATAL: something went wrong (errors present but empty mail body)\n"
fi fi
[[ -z "$body" ]] && exit 0 [[ -z "$body" ]] && exit 0
# send mail # send mail
echo "$body" | mail -s "$subject" $mailto echo -e "$body" | mail -s "$subject" $mailto
if [[ $? -ne 0 ]]; then if [[ $? -ne 0 ]]; then
echo "$0: Failed to send btrbk mail to \"$mailto\", dumping mail:${BR}" 1>&2 echo -e "$0: Failed to send btrbk mail to \"$mailto\", dumping mail:\n" 1>&2
echo "<mail_subject>$subject</mail_subject>${BR}<mail_body>${BR}$body</mail_body>" 1>&2 echo -e "<mail_subject>$subject</mail_subject>\n<mail_body>\n$body</mail_body>" 1>&2
fi fi
} }
einfo() einfo()
{ {
info+="$1${BR}" info+="$1\n"
} }
ebegin() ebegin()
{ {
ebtext=$1 ebtext=$1
detail+="${BR}### $1${BR}" detail+="\n### $1\n"
} }
eend() eend()
{ {
if [[ $1 -eq 0 ]]; then if [[ $1 -eq 0 ]]; then
eetext=${3-success} eetext=${3-success}
detail+="${BR}" detail+="\n"
else else
has_errors=1 has_errors=1
eetext="ERROR (code=$1)" eetext="ERROR (code=$1)"
[[ -n "$2" ]] && eetext+=": $2" [[ -n "$2" ]] && eetext+=": $2"
detail+="${BR}### $eetext${BR}" detail+="\n### $eetext\n"
fi fi
info+="$ebtext: $eetext${BR}" info+="$ebtext: $eetext\n"
return $1 return $1
} }
@ -146,10 +139,10 @@ run_cmd()
{ {
cmd_out=$("$@" 2>&1) cmd_out=$("$@" 2>&1)
local ret=$? local ret=$?
detail+="++ ${@@Q}${BR}" detail+="++ ${@@Q}\n"
if [[ -n "${mail_cmd_block_prefix:-}" ]] && [[ -n "$cmd_out" ]]; then if [[ -n "${mail_cmd_block_prefix:-}" ]] && [[ -n "$cmd_out" ]]; then
detail+=$(echo -n "$cmd_out" | sed "s/^/${mail_cmd_block_prefix}/") detail+=$(echo -n "$cmd_out" | sed "s/^/${mail_cmd_block_prefix}/")
detail+="${BR}" detail+="\n"
else else
detail+=$cmd_out detail+=$cmd_out
fi fi
@ -166,7 +159,7 @@ mount_all()
if [[ $? -eq 0 ]]; then if [[ $? -eq 0 ]]; then
eend -1 "already mounted" eend -1 "already mounted"
else else
detail+="${BR}" detail+="\n"
run_cmd mount --target $mountpoint run_cmd mount --target $mountpoint
eend $? && mounted+=" $mountpoint" eend $? && mounted+=" $mountpoint"
fi fi

View File

@ -31,7 +31,7 @@
# NOTE: Depending on your setup (hardware, btrfs mount options), # NOTE: Depending on your setup (hardware, btrfs mount options),
# btrbk-verify may eat all your CPU power and use high bandwidth! # btrbk-verify may eat all your CPU power and use high bandwidth!
# Consider nice(1), ionice(1). # Consider nice(1), ionice(1).
#
# Incomplete resource eater list: # Incomplete resource eater list:
# - rsync: checksums, heavy disk I/O # - rsync: checksums, heavy disk I/O
# - btrfs: decompression, encryption # - btrfs: decompression, encryption
@ -77,7 +77,7 @@ set -u
set -e set -e
set -o pipefail set -o pipefail
btrbk_version_min='0.32.0' btrbk_version_min='0.28.0'
# defaults: ignore subvol dirs and root folder timestamp change # defaults: ignore subvol dirs and root folder timestamp change
ignore_nested_subvolume_dir=1 ignore_nested_subvolume_dir=1
@ -107,8 +107,8 @@ options:
--ignore-xattrs ignore xattrs when verifying subvolumes --ignore-xattrs ignore xattrs when verifying subvolumes
--ssh-identity FILE override ssh_identity from btrbk.conf(5) with FILE, --ssh-identity FILE override ssh_identity from btrbk.conf(5) with FILE,
and clear all other ssh_* options (use with --ssh-user) and clear all other ssh_* options (use with --ssh-user)
--ssh-user USER override ssh_user from btrbk.conf(5) with USER, and --ssh-user USER override ssh_user from btrbk.conf(5) with USER
clear all other ssh_* options(use with --ssh-identity) (only in conjunction with --ssh-identity)
--ssh-agent start ssh-agent(1) and add identity --ssh-agent start ssh-agent(1) and add identity
commands: commands:
@ -157,6 +157,7 @@ while [[ "$#" -ge 1 ]]; do
;; ;;
--ssh-identity) --ssh-identity)
# use different ssh identity (-i option) for rsync rsh. # use different ssh identity (-i option) for rsync rsh.
# if set, ssh_user defaults to root.
# NOTE: this overrides all btrbk ssh_* options # NOTE: this overrides all btrbk ssh_* options
ssh_identity="$2" ssh_identity="$2"
shift shift
@ -185,8 +186,6 @@ while [[ "$#" -ge 1 ]]; do
shift shift
done done
BR=$'\n'
log_line() log_line()
{ {
echo "$@" 1>&2 echo "$@" 1>&2
@ -209,13 +208,13 @@ tlog()
[[ -n "$dryrun" ]] && [[ "$status" == "starting" ]] && status="dryrun_starting" [[ -n "$dryrun" ]] && [[ "$status" == "starting" ]] && status="dryrun_starting"
local line="$(date --iso-8601=seconds) verify-rsync ${status} ${target} ${source} - -" local line="$(date --iso-8601=seconds) verify-rsync ${status} ${target} ${source} - -"
[[ -n "$comment" ]] && line="$line # $comment"; [[ -n "$comment" ]] && line="$line # $comment";
tlog_text+="$line${BR}" tlog_text+="$line\n"
log_debug "$line" log_debug "$line"
} }
tlog_print() tlog_print()
{ {
# tlog goes to stdout # tlog goes to stdout
echo "${BR}TRANSACTION LOG${BR}---------------${BR}${tlog_text:-}" echo -e "\nTRANSACTION LOG\n---------------\n${tlog_text:-}"
} }
# parse "rsync -i,--itemize-changes" output. # parse "rsync -i,--itemize-changes" output.
@ -268,14 +267,11 @@ rsync_rsh()
local rsh_match="(.*) ([a-z0-9_-]+)@([a-zA-Z0-9.-]+)$" local rsh_match="(.*) ([a-z0-9_-]+)@([a-zA-Z0-9.-]+)$"
if [[ -z "$rsh" ]]; then if [[ -z "$rsh" ]]; then
return echo
elif [[ -n "$ssh_user" ]] || [[ -n "$ssh_identity" ]]; then elif [[ -n "$ssh_identity" ]]; then
# override btrbk.conf from command line arguments # override btrbk.conf from command line arguments
log_debug "Overriding all ssh_* options from btrbk.conf" log_debug "Overriding all ssh_* options from btrbk.conf"
local cmd="ssh -q" echo "ssh -q -i $ssh_identity -l $ssh_user"
[[ -n "$ssh_identity" ]] && cmd="$cmd -i '$ssh_identity'"
[[ -n "$ssh_user" ]] && cmd="$cmd -l '$ssh_user'"
echo "$cmd"
elif [[ $rsh =~ $rsh_match ]]; then elif [[ $rsh =~ $rsh_match ]]; then
echo "${BASH_REMATCH[1]} -l ${BASH_REMATCH[2]}" echo "${BASH_REMATCH[1]} -l ${BASH_REMATCH[2]}"
else else
@ -306,7 +302,7 @@ start_ssh_agent()
eval_btrbk_resolved_line() eval_btrbk_resolved_line()
{ {
local line=" $1" local line=$1
local prefix=$2 local prefix=$2
local required_keys=$3 local required_keys=$3
# reset all variables first # reset all variables first
@ -315,13 +311,25 @@ eval_btrbk_resolved_line()
done done
for vv in $required_keys; do for vv in $required_keys; do
# basic input validation, set prefixed variable (eval) # basic input validation, set prefixed variable (eval)
local match=" ${vv}='([^']*('\\\\''[^']*)*)'" local var_assignment=$(echo "$line" | grep -Eo "${vv}"'="[^"]*"')
if [[ $line =~ $match ]] ; then if [[ $? -ne 0 ]] || [[ -z "$var_assignment" ]]; then
eval "${prefix}${vv}='${BASH_REMATCH[1]}'" || return 1 if [[ $vv == "target_type" ]]; then
else # hack for btrbk <= 0.27.2 bug: missing "target_type" in $table_formats{resolved}
log_line "btrbk-verify: ERROR: Missing variable \"${vv}\"" # hardcode send-receive type, this results in rsync failing on raw targets with:
# rsync: ERROR: cannot stat destination "xxx.btrfs.xz.gpg/": Not a directory (20)
eval "${prefix}target_type=\"send-receive\"" || return 1
continue
fi
if [[ $vv == "target_rsh" ]] && [[ -z "$R_target_host" ]]; then
# hack for btrbk <= 0.27.2 bug: missing "target_rsh" in $table_formats{resolved}
# note that target_rsh is not needed if target_host is not set.
# variable has already been reset above, continue
continue
fi
log_line "btrbk-verify: missing required variable \"${vv}\" in btrbk --format=raw line"
return 1 return 1
fi fi
eval "${prefix}${var_assignment}" || return 1
done done
} }
@ -331,6 +339,10 @@ exit_trap_action()
[[ $verbose -gt 0 ]] && tlog_print [[ $verbose -gt 0 ]] && tlog_print
} }
# restrictions from rsync_rsh():
[[ -z "$ssh_identity" ]] && [[ -n "$ssh_user" ]] && print_usage 2
[[ -n "$ssh_identity" ]] && [[ -z "$ssh_user" ]] && print_usage 2
# start ssh-agent(1) # start ssh-agent(1)
[[ -n "$ssh_start_agent" ]] && start_ssh_agent [[ -n "$ssh_start_agent" ]] && start_ssh_agent
@ -359,22 +371,20 @@ while read -r btrbk_list_line; do
log_debug "Evaluating [btrbk list] line: $btrbk_list_line" log_debug "Evaluating [btrbk list] line: $btrbk_list_line"
[[ -z "$btrbk_list_line" ]] && continue [[ -z "$btrbk_list_line" ]] && continue
if ! eval_btrbk_resolved_line "$btrbk_list_line" \ if ! eval_btrbk_resolved_line "$btrbk_list_line" \
"R_" "snapshot_subvolume target_subvolume source_host target_host target_type source_rsh target_rsh" "R_" "snapshot_path target_path source_host target_host target_type source_rsh target_rsh"
then then
log_line "btrbk-verify: ERROR: Parse error of command output: ${btrbk_cmd[@]}" log_line "btrbk-verify: WARNING: Skipping task (parse error). Make sure to have >=btrbk-${btrbk_version_min} installed!"
log_line "Make sure to have >=btrbk-${btrbk_version_min} installed!" continue
exitstatus=1
break
fi fi
source="${R_snapshot_subvolume}/" source="${R_snapshot_path}"
target="${R_target_subvolume}/" target="${R_target_path}"
[[ -n "$R_source_host" ]] && source="${R_source_host}:${source}" [[ -n "$R_source_host" ]] && source="${R_source_host}:${source}"
[[ -n "$R_target_host" ]] && target="${R_target_host}:${target}" [[ -n "$R_target_host" ]] && target="${R_target_host}:${target}"
if [[ -z "$R_snapshot_subvolume" ]]; then if [[ -z "$R_snapshot_path" ]]; then
log_line "WARNING: Skipping task (missing snapshot): target=$target" log_line "WARNING: Skipping task (missing snapshot): target=$target"
elif [[ -z "$R_target_subvolume" ]]; then elif [[ -z "$R_target_path" ]]; then
log_line "Skipping task (no target): source=$source" log_line "Skipping task (no target): source=$source"
elif [[ "$R_target_type" != "send-receive" ]]; then elif [[ "$R_target_type" != "send-receive" ]]; then
log_line "Skipping task (target_type=$R_target_type): source=$source, target=$target" log_line "Skipping task (target_type=$R_target_type): source=$source, target=$target"
@ -386,11 +396,10 @@ while read -r btrbk_list_line; do
# rsync rsh is either source_rsh or target_rsh or empty # rsync rsh is either source_rsh or target_rsh or empty
eff_rsh="$R_source_rsh" eff_rsh="$R_source_rsh"
[[ -z "$eff_rsh" ]] && eff_rsh="$R_target_rsh" [[ -z "$eff_rsh" ]] && eff_rsh="$R_target_rsh"
eff_rsh=$(rsync_rsh "$eff_rsh")
rsync_cmd=("rsync" "${rsync_args[@]}") rsync_cmd=("rsync" "${rsync_args[@]}")
[[ -n "$eff_rsh" ]] && rsync_cmd+=("-e" "$eff_rsh") [[ -n "$eff_rsh" ]] && rsync_cmd+=(-e "$(rsync_rsh "$eff_rsh")")
rsync_cmd+=("${source}" "${target}") rsync_cmd+=("${source}/" "${target}/")
log_cmd "${rsync_cmd[@]}" log_cmd "${rsync_cmd[@]}"
[[ -n "$dryrun" ]] && rsync_cmd=("cat" "/dev/null") [[ -n "$dryrun" ]] && rsync_cmd=("cat" "/dev/null")

View File

@ -54,7 +54,7 @@ salt_hex = "".join(["{:02x}".format(x) for x in salt])
dk_hex = "".join(["{:02x}".format(x) for x in dk]) dk_hex = "".join(["{:02x}".format(x) for x in dk])
print("KEY=" + dk_hex); print("KEY=" + dk_hex);
print("algorithm=pbkdf2_hmac"); print("algoritm=pbkdf2_hmac");
print("hash_name=" + hash_name); print("hash_name=" + hash_name);
print("salt=" + salt_hex); print("salt=" + salt_hex);
print("iterations=" + str(iterations)); print("iterations=" + str(iterations));

View File

@ -42,7 +42,7 @@ my $compress_format_alt = 'gz|bz2|xz|lzo|lz4';
my $file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/; # note: ubuntu uses '@' in the subvolume layout: <https://help.ubuntu.com/community/btrfs> my $file_match = qr/[0-9a-zA-Z_@\+\-\.\/]+/; # note: ubuntu uses '@' in the subvolume layout: <https://help.ubuntu.com/community/btrfs>
my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/; my $uuid_match = qr/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/;
my $timestamp_postfix_match = qr/\.(?<YYYY>[0-9]{4})(?<MM>[0-9]{2})(?<DD>[0-9]{2})(T(?<hh>[0-9]{2})(?<mm>[0-9]{2})((?<ss>[0-9]{2})(?<zz>(Z|[+-][0-9]{4})))?)?(_(?<NN>[0-9]+))?/; # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]" my $timestamp_postfix_match = qr/\.(?<YYYY>[0-9]{4})(?<MM>[0-9]{2})(?<DD>[0-9]{2})(T(?<hh>[0-9]{2})(?<mm>[0-9]{2})((?<ss>[0-9]{2})(?<zz>(Z|[+-][0-9]{4})))?)?(_(?<NN>[0-9]+))?/; # matches "YYYYMMDD[Thhmm[ss+0000]][_NN]"
my $raw_postfix_match = qr/--(?<received_uuid>$uuid_match)(\@(?<parent_uuid>$uuid_match))?\.btrfs?(\.(?<compress>($compress_format_alt)))?(\.(?<encrypt>gpg))?(\.(?<split>split_aa))?(\.(?<incomplete>part))?/; # matches ".btrfs_<received_uuid>[@<parent_uuid>][.gz|.bz2|.xz|...][.gpg][.split_aa][.part]" my $raw_postfix_match = qr/--(?<received_uuid>$uuid_match)(\@(?<parent_uuid>$uuid_match))?\.btrfs?(\.(?<compress>($compress_format_alt)))?(\.(?<encrypt>gpg))?(\.(?<split>split_aa))?(\.(?<incomplete>part))?/; # matches ".btrfs_<received_uuid>[@<parent_uuid>][.gz|bz2|xz][.gpg][.split_aa][.part]"
my $dryrun; my $dryrun;

View File

@ -1,217 +0,0 @@
#!/usr/bin/env python3
import os
import logging
import subprocess
import argparse
logger = logging.getLogger(__name__)
class TransformProcess:
def run(self, bfile, options, **kw):
return subprocess.Popen(self.get_cmd(bfile, options), **kw)
def get_cmd(self, bfile, options):
raise NotImplementedError()
@classmethod
def add_parser_options(cls, parser):
pass
class TransformOpensslDecrypt(TransformProcess):
@staticmethod
def get_cmd(bfile, options):
return [
'openssl', 'enc', '-d', '-' + bfile.info['cipher'], '-K',
open(options.openssl_keyfile, 'r').read(), '-iv', bfile.info['iv']
]
@staticmethod
def add_parser_options(parser):
parser.add_argument('--openssl-keyfile', help="path to private encryption key file")
class TransformDecompress(TransformProcess):
def __init__(self, program):
self.p = program
def get_cmd(self, bfile, options):
return [self.p, '-d']
class TransformBtrfsReceive(TransformProcess):
@classmethod
def run(cls, bfile, options, **kw):
return subprocess.Popen(cls.get_cmd(bfile, options), **kw)
@staticmethod
def get_cmd(bfile, options):
return ['btrfs', 'receive', options.restore_dir]
TRANSFORMERS = (
TransformOpensslDecrypt, TransformDecompress, TransformBtrfsReceive
)
class BtrfsPipeline:
def __init__(self, bfile):
self.bfile = bfile
self.processors = []
def append(self, transformer):
self.processors.append(transformer)
def run(self, options):
processes = []
with open(self.bfile.data_file, 'rb') as next_input:
for transformer in self.processors:
process = transformer.run(
self.bfile, options,
stdin=next_input, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
next_input = process.stdout
processes.append(process)
btrfs_process = TransformBtrfsReceive.run(
self.bfile, options, stdin=next_input,
stderr=subprocess.PIPE, stdout=subprocess.DEVNULL)
processes.append(btrfs_process)
# warning: the code below is pretty ugly and hacky
terminated = 0
while terminated < len(processes):
for p in processes:
if p.returncode is not None:
continue
msg = None
try:
p.wait(timeout=1)
except subprocess.TimeoutExpired as e:
pass
except Exception as e:
msg = e
else:
msg = p.stderr.read().decode('utf-8').strip()
finally:
if p.returncode is not None:
terminated += 1
if p.returncode != 0:
for p_other in processes:
p_other.terminate()
terminated += 1
if msg:
logger.error(f"error running {p.args}: {msg}")
def get_cmd(self, options):
command_pipe = [['cat', self.bfile.data_file]]
for transformer in self.processors:
command_pipe.append(transformer.get_cmd(self.bfile, options))
command_pipe.append(TransformBtrfsReceive.get_cmd(self.bfile, options))
return ' | '.join(' '.join(x) for x in command_pipe)
class BackupFile:
def __init__(self, path):
assert path.endswith('.info')
self.info_file = path
self.info = self._parse_info()
self.uuid = self.info['RECEIVED_UUID']
self.data_file = os.path.join(os.path.dirname(path), self.info['FILE'])
self.parent = self.info.get('RECEIVED_PARENT_UUID')
self.is_restored = False
def _parse_info(self):
config = {}
with open(self.info_file, 'r') as fh:
# skip command option line
for line in fh.readlines():
if '=' not in line:
continue
key, val = line.strip().split('=', maxsplit=1)
config[key] = val
return config
def get_transformers(self):
if 'encrypt' in self.info:
if self.info['encrypt'] == 'gpg':
raise NotImplementedError('gpg encryption')
elif self.info['encrypt'] == 'openssl_enc':
yield TransformOpensslDecrypt()
else:
raise Exception(f'unknown encryption type: "{self.info["encrypt"]}"')
if 'compress' in self.info:
yield TransformDecompress(self.info['compress'])
def restore_file(self, options):
assert self.info.get('TYPE') == 'raw'
assert not self.info.get('INCOMPLETE')
pipeline = BtrfsPipeline(self)
for transformer in self.get_transformers():
pipeline.append(transformer)
if options.dry_run:
print(pipeline.get_cmd(options))
else:
logger.info(f"restoring backup {os.path.basename(self.data_file)}")
pipeline.run(options)
self.is_restored = True
def restore_from_path(backup, options):
path = os.path.dirname(backup)
info_files = {}
backup_file = BackupFile(backup + '.info')
restored_files = set()
for entry in os.scandir(path):
if entry.is_file() and entry.name.endswith('.info'):
info = BackupFile(entry.path)
info_files[info.uuid] = info
restored_files.update(restore_backup(backup_file, info_files, options))
logger.info(f"finished; restored {len(restored_files)} backup files")
def restore_backup(bfile, parents, options):
if bfile.is_restored:
return
if bfile.parent:
parent = parents.get(bfile.parent)
if not parent:
msg = (f"missing parent {bfile.parent} for"
f"'{os.path.basename(bfile.info_file)}'")
if options.ignore_missing:
logger.warning(msg)
else:
raise Exception(msg)
else:
yield from restore_backup(parent, parents, options)
bfile.restore_file(options)
yield bfile.uuid
def main():
parser = argparse.ArgumentParser(description="restore btrbk raw backup")
parser.add_argument('backup', help="backup file to restore; for incremental"
" backups the parent files must be in the same directory")
parser.add_argument('restore_dir', help="target directory for restored subvolumes"
" (path argument for \"btrfs receive\")")
parser.add_argument('-n', '--dry-run', action='store_true',
help="print commands that would be executed")
parser.add_argument('--ignore-missing', action='store_true',
help="do not fail on missing parent snapshots")
for transformer in TRANSFORMERS:
transformer.add_parser_options(parser)
args = parser.parse_args()
if args.dry_run:
logger.setLevel('ERROR')
restore_from_path(args.backup, args)
if __name__ == '__main__':
logger.setLevel('INFO')
logging.basicConfig(format='%(asctime)s %(levelname)s - %(message)s')
main()

View File

@ -110,8 +110,8 @@ have to create a run-time (rw) snapshot before booting into it:
How do I convert '/' (subvolid=5) into a subvolume? How do I convert '/' (subvolid=5) into a subvolume?
--------------------------------------------------- ---------------------------------------------------
There's several ways to achieve this, the solution described below There's several ways to achieve this, the solution described below is
guarantees not to create new files (extents) on disk. that it guarantees not to create new files (extents) on disk.
### Step 1: make a snapshot of your root filesystem ### Step 1: make a snapshot of your root filesystem
@ -191,7 +191,7 @@ What is the most efficient way to clone btrfs storage?
------------------------------------------------------ ------------------------------------------------------
It is very common (and avisable!) to keep backups on a separate It is very common (and avisable!) to keep backups on a separate
location. In some situations, it is also required to transport the location. In some situations, is is also required to transport the
data physically, either to the datacenter or to your safe in the data physically, either to the datacenter or to your safe in the
basement. basement.

View File

@ -1,4 +1,5 @@
DOCS = FAQ.md DOCS = FAQ.md \
upgrade_to_v0.23.0.md
MAN_MAN1 = btrbk.1 \ MAN_MAN1 = btrbk.1 \
lsbtr.1 \ lsbtr.1 \
ssh_filter_btrbk.1 ssh_filter_btrbk.1
@ -17,19 +18,8 @@ ifeq ($(COMPRESS), yes)
endif endif
# convert using "asciidoctor": <https://asciidoctor.org> # convert using "asciidoctor": <https://asciidoctor.org>
# fallback to "a2x" from asciidoc package: <http://asciidoc.org> ASCIIDOCTOR_MANPAGE = asciidoctor -d manpage -b manpage
ifneq (, $(shell command -v asciidoctor 2> /dev/null)) ASCIIDOCTOR_HTML = asciidoctor -b html5 -d article
ASCIIDOC_MANPAGE = asciidoctor -d manpage -b manpage
ASCIIDOC_HTML = asciidoctor -b html5 -d article
else ifneq (, $(shell command -v a2x 2> /dev/null))
# NOTE: using -L (--no-xmllint), as xmllint is a separate package on many distros.
ASCIIDOC_MANPAGE = a2x -L -d manpage -f manpage
ASCIIDOC_HTML = asciidoc -b html -d article
else
ASCIIDOC_ERR = $(error "please install either asciidoc or asciidoctor")
ASCIIDOC_MANPAGE = $(ASCIIDOC_ERR)
ASCIIDOC_HTML = $(ASCIIDOC_ERR)
endif
# reproducible builds: reference date is ":date:" attribute from asciidoc source # reproducible builds: reference date is ":date:" attribute from asciidoc source
date_attr = $(shell sed -rn 's/:date:\s*//p' $(1)) date_attr = $(shell sed -rn 's/:date:\s*//p' $(1))
@ -60,10 +50,10 @@ clean:
gzip -9 -n -c $< > $@ gzip -9 -n -c $< > $@
%.1 : %.1.asciidoc %.1 : %.1.asciidoc
SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $< SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_MANPAGE) -o $@ $<
%.5 : %.5.asciidoc %.5 : %.5.asciidoc
SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_MANPAGE) $< SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_MANPAGE) -o $@ $<
%.html : %.asciidoc %.html : %.asciidoc
SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOC_HTML) -o $@ $< SOURCE_DATE_EPOCH=$(call source_date_epoch,$<) $(ASCIIDOCTOR_HTML) -o $@ $<

View File

@ -1,7 +1,7 @@
btrbk(1) btrbk(1)
======== ========
:date: 2023-03-25 :date: 2021-03-21
:release-version: 0.32.6 :release-version: 0.31.2
:man manual: Btrbk Manual :man manual: Btrbk Manual
:man source: Btrbk {release-version} :man source: Btrbk {release-version}
@ -20,12 +20,11 @@ btrbk [-h|--help] [--version]
[-c|--config <file>] [-n|--dry-run] [--exclude <filter>] [-c|--config <file>] [-n|--dry-run] [--exclude <filter>]
[-p|--preserve] [--preserve-snapshots] [--preserve-backups] [-p|--preserve] [--preserve-snapshots] [--preserve-backups]
[-v|--verbose] [-q|--quiet] [-l|--loglevel <level>] [-v|--verbose] [-q|--quiet] [-l|--loglevel <level>]
[-t|--table] [-L|--long] [-1|--single-column] [-t|--table] [-L|--long] [--format <output-format>] [--pretty]
[--format <output-format>] [--pretty]
[-S|--print-schedule] [--progress] [-S|--print-schedule] [--progress]
[--lockfile <file>] [--lockfile <file>]
[--override <config_option>=<value>] [--override <config_option>=<value>]
<command> [[--] <filter>...] <command> [<args>]
DESCRIPTION DESCRIPTION
@ -45,27 +44,26 @@ btrbk.conf(5) for more details.
=== Snapshots and Backups === Snapshots and Backups
Snapshots as well as backup subvolumes are created in the form: Snapshots as well as backup subvolumes are created in form:
<snapshot-name>.<timestamp>[_N] <snapshot-name>.<timestamp>[_N]
Where '<snapshot-name>' is identical to the source subvolume name, Where '<snapshot-name>' is identical to the source subvolume name,
unless the configuration option 'snapshot_name' is set. '<timestamp>' unless the configuration option 'snapshot_name' is set. The
is a timestamp describing the creation time (local time of the host <timestamp> is either "YYYYMMDD" or "YYYYMMDDThhmm" (dependent of the
running btrbk) of the snapshot/backup. The format can be configured 'timestamp_format' configuration option), where "YYYY" is the year,
using the 'timestamp_format' option, refer to btrbk.conf(5) for "MM" is the month, "DD" is the day, "hh" is the hour and "mm" is the
details. If multiple snapshots/backups are created on the same minute of the creation time (local time of the host running btrbk). If
date/time, 'N' will be incremented on each snapshot, starting at 1. multiple snapshots/backups are created on the same date/time, N will
be incremented on each snapshot, starting at 1.
If a snapshot or backup does not match the naming scheme above If a snapshot or backup does not match the naming scheme above
(e.g. if it has been renamed manually), btrbk will leave it untouched. (i.e. if it has been renamed manually), btrbk will leave it untouched.
Note that in btrfs terminology, a 'snapshot' is a ``subvolume with Note that 'snapshot' is a btrfs terminology for a ``read-only
a given initial content of the original subvolume'' (showing a subvolume'' (showing a parent-uuid, see btrfs-subvolume(8)); 'backup'
parent-uuid, see btrfs-subvolume(8)), and they can be read-write is a btrbk terminology for a ``read-only subvolume created with
(default) or read-only. In btrbk terminology, 'snapshot' means send/receive'' (showing a received-uuid).
``read-only btrfs snapshot'', and 'backup' means ``read-only subvolume
created with send/receive'' (showing a received-uuid).
OPTIONS OPTIONS
@ -88,15 +86,8 @@ OPTIONS
commands that would be executed. commands that would be executed.
--exclude <filter>:: --exclude <filter>::
Exclude configured sections matching '<filter>' (see Exclude configured sections matching '<filter>'. See
<<_filter_statements,FILTER STATEMENTS>> below), or any specific <<_filter_statements,FILTER STATEMENTS>> below.
snapshot from being backuped or deleted, or any specific backup
from being deleted.
+
Note that excluding specific snapshots from being backuped has impact
on scheduling: e.g. if the "first snapshot of the day" is excluded,
the "second snapshot of the day" shifts to "first", creating a backup
as "first backup of the day".
-p, --preserve:: -p, --preserve::
Preserve all snapshots and backups. Skips deletion of any Preserve all snapshots and backups. Skips deletion of any
@ -112,7 +103,7 @@ as "first backup of the day".
specified in the configuration file. specified in the configuration file.
--wipe:: --wipe::
Ignore configured snapshot retention policy, delete all but the latest Ignore configured snapshot retention policy, delete all but latest
snapshots instead. All snapshots needed for incremental backup snapshots instead. All snapshots needed for incremental backup
(latest common) are also preserved. Useful if you are getting low (latest common) are also preserved. Useful if you are getting low
on disk space (ENOSPC). on disk space (ENOSPC).
@ -135,15 +126,12 @@ as "first backup of the day".
-L, --long:: -L, --long::
Print output in long table format (shortcut for "--format=long"). Print output in long table format (shortcut for "--format=long").
-1, --single-column::
Print output as single column (not available for all commands).
--format table|long|raw|col:[h:]<columns>:: --format table|long|raw|col:[h:]<columns>::
Print output in specified format. If set to "raw", prints Print output in specified format. If set to "raw", prints
space-separated, quoted key=value pairs (machine readable). space-separated key="value" pairs (machine readable).
+ +
If set to "col:", prints only the <columns> specified (comma-separated If set to "col:", prints only the <columns> specified (comma-separated
list). Header lines are omitted if the "h:" modifier is present. list). Header lines are ommitted if the "h:" modifier is present.
Columns prefixed with "-" are collapsed if empty. Columns postfixed Columns prefixed with "-" are collapsed if empty. Columns postfixed
with ":RALIGN" are right-aligned. with ":RALIGN" are right-aligned.
@ -161,9 +149,9 @@ with ":RALIGN" are right-aligned.
command (version >= 20180505) installed on the host running btrbk. command (version >= 20180505) installed on the host running btrbk.
--lockfile <file>:: --lockfile <file>::
Place an exclusive lock on <file> during program execution, using Create lockfile <file> on startup; checks lockfile before running
flock(2). If the lock is held by another process, exit before any btrfs commands (using perl "flock"), and exits if the lock is
running any actions. Overrides configuration option held by another btrbk instance. Overrides configuration option
"lockfile". Ignored on dryrun ('-n', '--dry-run'). "lockfile". Ignored on dryrun ('-n', '--dry-run').
--override <config_option>=<value>:: --override <config_option>=<value>::
@ -222,9 +210,8 @@ configuring the retention policy.
snapshots that are not preserved by their configured retention snapshots that are not preserved by their configured retention
policy will be deleted. Note that the latest snapshot (the one policy will be deleted. Note that the latest snapshot (the one
created in step 1) as well as the latest snapshot/backup pair are created in step 1) as well as the latest snapshot/backup pair are
always preserved, regardless of the retention policy. If any always preserved, regardless of the retention policy.
target is unreachable or has errors, all snapshots are preserved
in order not to break the incremental chain.
*dryrun* [filter...]:: *dryrun* [filter...]::
Don't run any btrfs commands that would alter the filesystem, just Don't run any btrfs commands that would alter the filesystem, just
@ -429,7 +416,8 @@ execution times and high ram usage. Consider setting 'cache_dir'.
lsbtr(1). lsbtr(1).
*config* print|print-all:: *config* print|print-all::
Prints the parsed configuration file. Prints the parsed configuration file. Use the '--format' command
line option to switch between different output formats.
FILTER STATEMENTS FILTER STATEMENTS
@ -480,9 +468,6 @@ Note that for *run* and *snapshot* commands, a filter matching a
surrounding 'subvolume' section. If this is not desired, consider surrounding 'subvolume' section. If this is not desired, consider
running *snapshot* and *resume* commands separately. running *snapshot* and *resume* commands separately.
Filter statements can match multiple times (e.g. on group as well as
host name). In such a case, all matches are processed.
FILES FILES
----- -----

View File

@ -1,7 +1,7 @@
btrbk.conf(5) btrbk.conf(5)
============= =============
:date: 2023-03-25 :date: 2021-03-21
:release-version: 0.32.6 :release-version: 0.31.2
:man manual: Btrbk Manual :man manual: Btrbk Manual
:man source: Btrbk {release-version} :man source: Btrbk {release-version}
@ -36,21 +36,34 @@ global options must be set before any sections are defined.
Blank lines are ignored. A hash character (#) starts a comment Blank lines are ignored. A hash character (#) starts a comment
extending until end of line. extending until end of line.
Whitespace or unicode characters are not allowed for file
names. Allowed characters are:
[0-9] [a-z] [A-Z] and "._+-@"
This is for sanity/safety/security reasons, we apologize for the
inconvenience.
SECTIONS SECTIONS
-------- --------
*volume* <volume-directory>|<url> (optional):: *volume* <volume-directory>|<url>::
Absolute path pointing to a btrfs file system containing the Directory of a btrfs volume containing the source subvolume(s) to
source subvolume(s) to be backed up. Usually the mount point of a be backed up. '<volume-directory>' must be an absolute path and
btrfs filesystem mounted with the 'subvolid=5' option. point to a btrfs volume (or subvolume). Usually the mount point of
a btrfs filesystem mounted with the 'subvolid=5' option.
*subvolume* <subvolume-name>:: *subvolume* <subvolume-name>::
Subvolume to be backed up, relative to the '<volume-directory>' of Subvolume to be backed up, relative to the '<volume-directory>'
the 'volume' section, or absolute if the 'volume' section is specified in the 'volume' section. Multiple 'subvolume' sections
omitted. Accepts wildcard character "*". are allowed within 'volume' sections. Accepts wildcard character
"*".
+ +
-- --
If set to ".", the subvolume at '<volume-directory>' is used as backup
source, and the snapshots will be created within the source subvolume
itself (see 'snapshot_dir' option below), which is not recommended.
Note that if this subvolume is btrfs root (id=5), it needs to have a Note that if this subvolume is btrfs root (id=5), it needs to have a
valid UUID, which is not the case for file systems created with valid UUID, which is not the case for file systems created with
btrfs-progs < 4.16. btrfs-progs < 4.16.
@ -96,7 +109,7 @@ otherwise.
*timestamp_format* short|long|long-iso:: *timestamp_format* short|long|long-iso::
Timestamp format used as postfix for new snapshot subvolume Timestamp format used as postfix for new snapshot subvolume
names. Defaults to ``long''. names. Defaults to ``short''.
+ +
-- --
ifndef::backend-docbook,backend-manpage[] ifndef::backend-docbook,backend-manpage[]
@ -116,13 +129,12 @@ snapshots are created during a daylight saving time clock
change). change).
+ +
Note that using ``long-iso'' has implications on the scheduling, see Note that using ``long-iso'' has implications on the scheduling, see
<<_reference_time,Reference Time>> below. <<_retention_policy,RETENTION POLICY>> (caveats) below.
*snapshot_dir* <directory>:: *snapshot_dir* <directory>::
Directory in which the btrfs snapshots are created, relative to Directory in which the btrfs snapshots are created, relative to
'<volume-directory>' of the 'volume' section, or absolute if the '<volume-directory>' of the 'volume' section. Note that btrbk does
'volume' section is omitted. Note that btrbk does not not automatically create this directory, and the snapshot creation
automatically create this directory, and the snapshot creation
will fail if it is not present. will fail if it is not present.
*snapshot_name* <basename>:: *snapshot_name* <basename>::
@ -185,9 +197,10 @@ not be certain about this, such operations are disallowed in
Defines after what time (in full hours since midnight) a Defines after what time (in full hours since midnight) a
snapshot/backup is considered to be a "daily" backup. Daily, snapshot/backup is considered to be a "daily" backup. Daily,
weekly, monthly and yearly backups are preserved on this hour (see weekly, monthly and yearly backups are preserved on this hour (see
<<_retention_policy,RETENTION POLICY>> below). Ignored on <<_retention_policy,RETENTION POLICY>> below). If you set this
snapshots or backups without time information ('timestamp_format option, make sure to also set 'timestamp_format' to ``long'' or
short'). Defaults to ``0''. ``long-iso'' (backups and snapshots having no time information
will ignore this option). Defaults to ``0''.
*snapshot_preserve* no|<retention_policy>:: *snapshot_preserve* no|<retention_policy>::
Set retention policy for snapshots (see Set retention policy for snapshots (see
@ -242,29 +255,32 @@ set to ``all'' (the default).
=== SSH Options === SSH Options
*ssh_identity* <file>|no:: *ssh_identity* <file>::
Absolute path to a ssh identity file (private key). If not set, Absolute path to a ssh identity file (private key). Note that if
the ssh default is used (see ssh(1), "-i identity_file"). Note the private key is password protected, btrbk will prompt for user
that if the identity key is password protected and no input, which is usually not desired.
authentication agent is used, btrbk will prompt for user input on
every connection attempt.
*ssh_user* <username>|no:: *ssh_user* <username>::
Remote username for ssh. Defaults to ``root''. Make sure the Remote username for ssh. Defaults to ``root''. Make sure the
remote user is able to run "btrfs" with root privileges (see remote user is able to run "btrfs" with root privileges (see
option 'backend' for details). If set to ``no'', the ssh default option 'backend' for details).
is used.
*ssh_compression* yes|no:: *ssh_compression* yes|no::
Enables or disables the compression of ssh connections. Defaults Enables or disables the compression of ssh connections. Defaults
to ``no''. Note that if *stream_compress* is enabled, ssh to ``no''. Note that if *stream_compress* is enabled, ssh
compression will always be disabled for send/receive operations. compression will always be disabled for send/receive operations.
*ssh_cipher_spec* default|<cipher_spec>:: *ssh_cipher_spec* <cipher_spec>::
Selects the cipher specification for encrypting the session Selects the cipher specification for encrypting the session
(comma-separated list of ciphers in order of preference). See the (comma-separated list of ciphers in order of preference). See the
"-c cipher_spec" option in ssh(1) for more information. Defaults "-c cipher_spec" option in ssh(1) for more information. Defaults
to ``default'' (the ciphers specified in ssh_config(5)). to ``default'' (the ciphers specified in 'ssh_config').
Previous versions btrbk allowed you to set a *ssh_port* option, this
has been dropped in favor of the `ssh://hostname:port` notation in the
'volume' and 'target' <<_sections,sections>>. If you want to set a
global port for all SSH connections to remote hosts, set the ``Port''
option in ssh_config(5).
=== Data Stream Options === Data Stream Options
@ -274,7 +290,7 @@ set to ``all'' (the default).
remote locations. Defaults to ``no''. If enabled, make sure that remote locations. Defaults to ``no''. If enabled, make sure that
'<compress_command>' is available on the source and target '<compress_command>' is available on the source and target
hosts. Supported '<compress_command>': gzip, pigz, bzip2, pbzip2, hosts. Supported '<compress_command>': gzip, pigz, bzip2, pbzip2,
bzip3, xz, lzo, lz4, zstd. xz, lzo, lz4, zstd.
*stream_compress_level* default|<number>:: *stream_compress_level* default|<number>::
Compression level for the specified '<compress_command>'. Refer to Compression level for the specified '<compress_command>'. Refer to
@ -289,11 +305,7 @@ set to ``all'' (the default).
*stream_compress_threads* default|<number>:: *stream_compress_threads* default|<number>::
Number of threads to use for <compress_command>. Only supported Number of threads to use for <compress_command>. Only supported
for "pigz", "pbzip2", "bzip3", "zstd" and recent versions of "xz". for "pigz", "pbzip2", "zstd" and recent versions of "xz".
*stream_compress_adapt* yes|no::
Enable adaptive compression for <compress_command>. Only supported
for "zstd" (version >= 1.3.6). Defaults to ``no''.
*stream_buffer* <size>|no:: *stream_buffer* <size>|no::
Add a buffer to the btrfs send stream (locally, on uncompressed Add a buffer to the btrfs send stream (locally, on uncompressed
@ -359,14 +371,14 @@ constraints.
daemon, auth, lpr, news, cron, authpriv, local0..local7. daemon, auth, lpr, news, cron, authpriv, local0..local7.
*lockfile* <file>|no:: *lockfile* <file>|no::
Place an exclusive lock on <file> during program execution, using Create lockfile <file> on startup; checks lockfile before running
flock(2). If the lock is held by another process, exit before any btrfs commands (using perl "flock"), and exits if the lock is
running any actions. Ignored on dryrun ('-n', '--dry-run'). See held by another btrbk instance. Ignored on dryrun ('-n',
also '--lockfile' command-line option. '--dry-run'). See also '--lockfile' command-line option.
*backend* <backend>:: *backend* btrfs-progs|btrfs-progs-btrbk|btrfs-progs-sudo::
Backend filesystem utilities to be used for btrfs specific Backend filesystem utilities to be used for btrfs specific
operations. Available backends: operations. Defaults to ``btrfs-progs''.
+ +
-- --
*btrfs-progs*:: *btrfs-progs*::
@ -386,9 +398,6 @@ constraints.
have appropriate (root) permissions for the "btrfs" command groups have appropriate (root) permissions for the "btrfs" command groups
as well as the "readlink" and "test" commands in /etc/sudoers. as well as the "readlink" and "test" commands in /etc/sudoers.
*btrfs-progs-doas*::
Similar to btrfs-progs-sudo, using prefix "doas -n".
If you want to set this option for local or remote hosts only, set If you want to set this option for local or remote hosts only, set
*backend_local* or *backend_remote* (e.g. "backend_remote *backend_local* or *backend_remote* (e.g. "backend_remote
btrfs-progs-btrbk"). btrfs-progs-btrbk").
@ -397,29 +406,12 @@ If you want to set this option for regular (non-root) user only, set
*backend_local_user*. *backend_local_user*.
-- --
*compat* <compat-option>...:: *compat* busybox|no::
Enable compatibility options. Available 'compat-option': If set to ``busybox'', use busybox compatible commands. Defaults
+ to ``no''.
-- If you want to set this option for local or remote hosts only, you
*busybox*:: can set *compat_local* or *compat_remote*
Use busybox compatible commands, at the expense of slight overhead (e.g. "compat_remote busybox").
while reading filesystem information.
*ignore_receive_errors* _*experimental*_::
Tell btrfs-receive(8) to not terminate on errors by setting
"--max-errors=0" option. Print warnings instead.
+
A known use case for this are target hosts lacking xattr support
(e.g. some Synology NAS), while the send-stream contains "lsetxattr"
commands. Another case is targets failing to set otime, complaining
with "ERROR: attribute 12 requested but not present".
+
Note that there is *no guarantee that backups created with this
option enabled can be restored at all*.
If you want to set this option for local or remote hosts only, set
*compat_local* or *compat_remote* (e.g. "compat_remote busybox").
--
*cache_dir* <directory>:: *cache_dir* <directory>::
If set, cache extent maps for the "btrbk extents" command. If set, cache extent maps for the "btrbk extents" command.
@ -427,74 +419,28 @@ If you want to set this option for local or remote hosts only, set
=== Btrfs Specific Options === Btrfs Specific Options
*incremental_prefs* <list-spec>[:<amount>]...:: *btrfs_commit_delete* after|each|no::
Specify the preferences to determine the best common (correlated) If set, make sure the deletion of snapshot and backup subvolumes
parent and clone sources for incremental backups, by choosing from are committed to disk when btrbk terminates. Defaults to ``no''.
predefined candidate lists.
+
--
The 'list-spec' defines from what candidate list the next
parent/clone-src should be appended to a result list; 'amount' defines
how many (e.g. "sro:1 sro:1" is identical to "sro:2"), or all if
omitted. Any candidate which is already in the results is dropped.
The resulting list of subvolumes is then used as parameters for the *incremental_clones* <number>::
btrfs-send(8) command: the first for "-p <parent>", all others for Maximum number of clone sources allowed for incremental send. If
"-c <clone-src>". set, btrbk adds "-c <clone-src>" to the btrfs-send(8) command for
all present snapshot/backup pairs (correlated subvolumes matching
Available 'list-spec' (candidate lists = filtered subsets of matching 'received_uuid', printed by "btrbk stats"). Set this to a
correlated subvolumes): high number if you want to make sure that no common data is missed
on incremental backups, in expense of btrfs-send
*sro*,*srn*:: All from 'snapshot_dir' matching 'snapshot_name', with performance. Defaults to 0.
parent_uuid relationship, sorted by btrbk timestamp (o=older
n=newer).
*sao*,*san*:: All from 'snapshot_dir' matching 'snapshot_name', sorted
by btrbk timestamp (o=older n=newer).
*aro*,*arn*:: All from 'incremental_resolve', with parent_uuid
relationship, sorted by cgen (o=older n=newer).
Defaults to "sro:1 srn:1 sao:1 san:1 aro:1 arn:1". Note that for
most operations the default resolves a single parent, as there usually
are no newer snapshots, and all "sro:1 sao:1 aro:1" resolve to the
same snapshot.
Example: "defaults,sao,san,aro,arn" takes the defaults, and adds clone
sources for all (!) known candidates on the filesystem.
--
*incremental_clones* yes|no::
If enabled, btrbk adds "-c <clone-src>" to the btrfs-send(8)
command for all correlated subvolumes resolved by
'incremental_prefs'. If disabled, only "-p <parent>" is
used. Defaults to ``yes''.
*incremental_resolve* mountpoint|directory:: *incremental_resolve* mountpoint|directory::
Specifies where to search for the best common parent for Specifies where to search for the best common parent for
incremental backups. If set to ``mountpoint'', use parents in the incremental backups. If set to ``mountpoint'', use parents in the
filesystem tree below the mount point of the snapshot and target filesystem tree below mount points of source
directory. If set to ``directory'', use parents strictly below "<volume-directory>/<snapshot-dir>" and target
snapshot/target directories. Set this to ``directory'' if you get "<target-directory>". If set to ``directory'', use parents
access problems (when not running btrbk as root). Defaults to strictly below source/target directories. Set this to
``mountpoint''. ``directory'' if you get access problems (when not running btrbk
as root). Defaults to ``mountpoint''.
*btrfs_commit_delete* yes|no::
If set, wait for the transaction commit at the end of each
snapshot or backup deletion (sets '--commit-each' option for
"btrfs subvolume delete"). Defaults to ``no''.
*send_protocol* <number>|no _*experimental*_::
Use btrfs send protocol version N. If enabled on 'target', btrbk
adds "--proto <number>" to the btrfs-send(8) command. Defaults to
``no'' (btrfs default).
*send_compressed_data* yes|no _*experimental*_::
Send data that is compressed on the filesystem directly without
decompressing it. This requires protocol version 2 or higher
(btrfs-progs >= 5.19), and implies "send_protocol 2". If enabled
on 'target', btrbk adds "--compressed-data" to the btrfs-send(8)
command. Defaults to ``no'' (btrfs default).
*snapshot_qgroup_destroy* yes|no _*experimental*_:: {blank} *snapshot_qgroup_destroy* yes|no _*experimental*_:: {blank}
*target_qgroup_destroy* yes|no _*experimental*_:: {blank} *target_qgroup_destroy* yes|no _*experimental*_:: {blank}
@ -505,39 +451,30 @@ sources for all (!) known candidates on the filesystem.
<https://bugzilla.kernel.org/show_bug.cgi?id=91751> <https://bugzilla.kernel.org/show_bug.cgi?id=91751>
=== Informative Options
*warn_unknown_targets* yes|no::
If set, prints a warning if btrbk encounters a target subvolume at
a unknown location (i.e. not following btrbk naming scheme, or
outside the target directory). Defaults to ``no''.
RETENTION POLICY RETENTION POLICY
---------------- ----------------
Retention policies are defined individually for snapshots, backups and btrbk uses separate retention policies for snapshots and backups,
archives (summarized as "backups" in the following text), using a which are defined by the 'snapshot_preserve_min', 'snapshot_preserve',
combination of: 'target_preserve_min', 'target_preserve', 'preserve_day_of_week' and
'preserve_hour_of_day' configuration options.
**_preserve_min* all|latest|no|<number>{h,d,w,m,y}:: Within this section, any statement about "backups" is always valid for
Amount of time (duration) in which all backups are preserved. backups as well as snapshots, referring to 'target_preserve' or
'snapshot_preserve' respectively.
**_preserve* no|<retention_policy>::
Schedule (single points in time) for which individual backups are
preserved.
Note that if "preserve_min" is set to ``all'' (the default), any
setting of "preserve" obviously has no effect.
The format for '<retention_policy>' is: The format for '<retention_policy>' is:
[<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y] [<hourly>h] [<daily>d] [<weekly>w] [<monthly>m] [<yearly>y]
With the following semantics:
*hourly*:: *hourly*::
Defines how many hours back hourly backups should be Defines how many hours back hourly backups should be
preserved. The first backup of an hour is considered an hourly preserved. The first backup of an hour is considered an hourly
backup. backup. Note that if you use <hourly> scheduling, make sure to
also set 'timestamp_format' to ``long'' or ``long-iso'', or the
scheduler will interpret the time as "00:00" (midnight).
*daily*:: *daily*::
Defines how many days back daily backups should be preserved. The Defines how many days back daily backups should be preserved. The
@ -564,31 +501,29 @@ Use an asterisk for ``all'' (e.g. "target_preserve 60d *m"
states: "preserve daily backups for 60 days back, and all monthly states: "preserve daily backups for 60 days back, and all monthly
backups"). backups").
The reference time (which defines the beginning of a day, week, month
or year) for all date/time calculations is the local time of the host
running btrbk.
Hint: Run btrbk with the '-S', '--print-schedule' option to get a Hint: Run btrbk with the '-S', '--print-schedule' option to get a
comprehensive output of the scheduler results. comprehensive output of the scheduler results.
Caveats:
=== Reference Time * If you run a setup with several btrbk instances (e.g. one
snapshot-only instance per remote client, and a separate fetch-only
instance on the backup server), it makes perfectly sense to run
btrbk with different local time on the clients, in order to make
sure the backups from all the remote hosts are preserved for
"midnight", and not at "00:00 UTC" (which would be "14:00" in
Honolulu). If you want this behavior, do NOT use "timestamp_format
long-iso".
The local time on the host running btrbk defines the reference time * If "timestamp_format long-iso" is set, running btrbk from different
for all date/time calculations, especially for "beginning of a day", time zones leads to different interpretation of "first in day, week,
and as a consequence for the first daily, weekly, monthly or yearly month, or year". Make sure to run btrbk with the same time zone on
backups. The local time on remote hosts (ssh source/target) is never every host, e.g. by setting the TZ environment variable (see
used. tzset(3)).
Unless "timestamp_format long-iso" is set, daily backups are preserved
at "preserve_hour_of_day" (defaults to midnight) of the respective
time zone (and not for "00:00 UTC", which would be "14:00" in
Honolulu). This becomes relevant for setups with multiple btrbk
instances, e.g. many snapshot-only instances (spread around the
world), and a fetch-only instance on the backup server.
Caveat:
* If "timestamp_format long-iso" is set, each btrbk instance on has a
different interpretation of "first in day". Make sure to run btrbk
with the same time zone on every host, e.g. by setting the TZ
environment variable (see tzset(3)).
TARGET TYPES TARGET TYPES
@ -597,19 +532,24 @@ TARGET TYPES
*send-receive*:: *send-receive*::
Backup to a btrfs filesystem, using "btrfs send/receive". This is Backup to a btrfs filesystem, using "btrfs send/receive". This is
the recommended (standard) target type. The '<target-directory>' the recommended (standard) target type. The '<target-directory>'
must be an absolute path and point to a subvolume or directory must be an absolute path and point to a btrfs volume (or
within a btrfs file system. See btrfs-send(8), btrfs-receive(8). subvolume), or to a directory within a subvolume. See
btrfs-send(8), btrfs-receive(8).
*raw* _*experimental*_:: *raw* _*experimental*_::
Backup to a raw (filesystem independent) file from the output of Backup to a raw (filesystem independent) file from the output of
btrfs-send(8), with optional compression and encryption. btrfs-send(8), with optional compression and encryption.
+ +
-- --
Note that the target preserve mechanism is currently disabled for
incremental raw backups (btrbk does not delete any incremental raw
files)!
Raw backups consist of two files: the main data file containing the Raw backups consist of two files: the main data file containing the
btrfs send stream, and a sidecar file ".info" containing metadata: btrfs send stream, and a sidecar file ".info" containing metadata:
<snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg] <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|.xz][.gpg]
<snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|...][.gpg].info <snapshot-name>.<timestamp>[_N].btrfs[.gz|.bz2|.xz][.gpg].info
For 'incremental' backups ("incremental yes"), please note that: For 'incremental' backups ("incremental yes"), please note that:
@ -620,11 +560,6 @@ For 'incremental' backups ("incremental yes"), please note that:
make sure that a non-incremental backup is triggered from time to make sure that a non-incremental backup is triggered from time to
time. time.
* The scheduler will never delete dependent parents of backups
preserved by the retention policy (run btrbk with the '-S',
'--print-schedule' option to get a comprehensive output of the
scheduler results).
* There is currently no support for rotation of incremental backups: * There is currently no support for rotation of incremental backups:
if 'incremental' is set, a full backup must be triggered manually if 'incremental' is set, a full backup must be triggered manually
from time to time in order to be able to delete old backups. from time to time in order to be able to delete old backups.
@ -634,8 +569,8 @@ Additional options for raw targets:
*raw_target_compress* <compress_command>|no:: *raw_target_compress* <compress_command>|no::
Compression algorithm to use for raw backup target. Supported Compression algorithm to use for raw backup target. Supported
'<compress_command>': gzip, pigz, bzip2, pbzip2, bzip3, xz, lzo, '<compress_command>': gzip, pigz, bzip2, pbzip2, xz, lzo, lz4,
lz4, zstd. zstd.
*raw_target_compress_level* default|<number>:: *raw_target_compress_level* default|<number>::
Compression level for the specified <compress_command>. Compression level for the specified <compress_command>.
*raw_target_compress_long* default|<number>:: *raw_target_compress_long* default|<number>::
@ -656,7 +591,7 @@ Additional options for "raw_target_encrypt gpg":
*gpg_keyring* <file>:: *gpg_keyring* <file>::
Keyring to use for gpg, e.g. "`/etc/btrbk/gpg/pubring.kbx`". Keyring to use for gpg, e.g. "`/etc/btrbk/gpg/pubring.kbx`".
*gpg_recipient* <name>...:: *gpg_recipient* <name>::
Encrypt for user id '<name>' (email address). Encrypt for user id '<name>' (email address).

View File

@ -9,18 +9,16 @@ file, choose one of the following methods:
### Generic Linux System ### Generic Linux System
Install [asciidoctor] or [asciidoc] if you want to build the
documentation.
Download and unpack the latest [btrbk source tarball] and type: Download and unpack the latest [btrbk source tarball] and type:
sudo make install sudo make install
#### Try latest master from Github:
wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk ### Gentoo Linux
chmod +x btrbk
sudo ./btrbk ls / btrbk is in portage:
emerge app-backup/btrbk
### Debian Based Distros ### Debian Based Distros
@ -32,7 +30,7 @@ Packages are also available via NeuroDebian: http://neuro.debian.net/pkgs/btrbk.
### Fedora Linux ### Fedora Linux
btrbk is in the official Fedora repos: https://src.fedoraproject.org/rpms/btrbk btrbk is in the official Fedora repos: https://apps.fedoraproject.org/packages/btrbk
sudo dnf install btrbk sudo dnf install btrbk
@ -49,13 +47,6 @@ btrbk is in the community repository
apk add btrbk apk add btrbk
### Gentoo Linux
btrbk is in portage:
emerge app-backup/btrbk
### Void Linux ### Void Linux
btrbk is in Void's `current` repository btrbk is in Void's `current` repository
@ -64,5 +55,3 @@ btrbk is in Void's `current` repository
[btrbk source tarball]: https://digint.ch/download/btrbk/releases/ [btrbk source tarball]: https://digint.ch/download/btrbk/releases/
[asciidoctor]: https://asciidoctor.org
[asciidoc]: https://asciidoc.org

View File

@ -1,7 +1,7 @@
lsbtr(1) lsbtr(1)
======== ========
:date: 2023-03-25 :date: 2021-03-21
:release-version: 0.32.6 :release-version: 0.31.2
:man manual: Btrbk Manual :man manual: Btrbk Manual
:man source: Btrbk {release-version} :man source: Btrbk {release-version}
@ -17,12 +17,11 @@ SYNOPSIS
[verse] [verse]
lsbtr [-h|--help] [--version] lsbtr [-h|--help] [--version]
[-l|--long] [-u|--uuid] [-1|--single-column] [--raw] [-l|--long] [-u|--uuid] [--raw]
[--format <output-format>]
[-v|--verbose] [-v|--verbose]
[-c|--config <file>] [-c|--config <file>]
[--override <config_option>=<value>] [--override <config_option>=<value>]
[[--] <path>|<url>...] [<path>|<url>...]
DESCRIPTION DESCRIPTION
@ -56,21 +55,9 @@ OPTIONS
-u, --uuid:: -u, --uuid::
Print UUID table (parent/received relations). Print UUID table (parent/received relations).
-1, --single-column::
Print path column only (delimited by newline).
--raw:: --raw::
Print space-separated key="value" pairs (machine readable). Print space-separated key="value" pairs (machine readable).
--format table|long|raw|col:[h:]<columns>::
Print output in specified format. If set to "raw", prints
space-separated key="value" pairs (machine readable).
+
If set to "col:", prints only the <columns> specified (comma-separated
list). Header lines are omitted if the "h:" modifier is present.
Columns prefixed with "-" are collapsed if empty. Columns postfixed
with ":RALIGN" are right-aligned.
-v, --verbose:: -v, --verbose::
Increase the level of verbosity. Increase the level of verbosity.

View File

@ -1,7 +1,7 @@
ssh_filter_btrbk(1) ssh_filter_btrbk(1)
=================== ===================
:date: 2023-03-25 :date: 2021-03-21
:release-version: 0.32.6 :release-version: 0.31.2
:man manual: Btrbk Manual :man manual: Btrbk Manual
:man source: Btrbk {release-version} :man source: Btrbk {release-version}
@ -39,13 +39,13 @@ The following commands are always allowed:
- "readlink" - "readlink"
- "test -d" (only if "compat busybox" configuration option is set) - "test -d" (only if "compat busybox" configuration option is set)
- "cat /proc/self/mountinfo" - "cat /proc/self/mountinfo"
- pipes through "gzip", "pigz", "bzip2", "pbzip2", "bzip3", "xz", - pipes through "gzip", "pigz", "bzip2", "pbzip2", "xz", "lzop",
"lzop", "lz4", "zstd" (stream_compress) "lz4", "zstd" (stream_compress)
- pipes through "mbuffer" (stream_buffer, rate_limit) - pipes through "mbuffer" (stream_buffer, rate_limit)
Example line in /root/.ssh/authorized_keys on a backup target host: Example line in /root/.ssh/authorized_keys on a backup target host:
command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup",restrict ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@example.org command="ssh_filter_btrbk.sh --target --delete --restrict-path /mnt/btr_backup" ssh-rsa AAAAB3NzaC1...hwumXFRQBL btrbk@mydomain.com
OPTIONS OPTIONS
@ -82,11 +82,6 @@ OPTIONS
-p, --restrict-path <path>:: -p, --restrict-path <path>::
Restrict commands to <path>. Note that "btrfs subvolume show", Restrict commands to <path>. Note that "btrfs subvolume show",
"btrfs subvolume list" are NOT affected by this option. "btrfs subvolume list" are NOT affected by this option.
+
It is not possible to restrict commands to exact subvolume names, as
btrfs-receive(8) takes a <path> as argument (directory, not including
the subvolume file name to be created, this is encoded in the
send-stream).
-l, --log:: -l, --log::
Log ACCEPT and REJECT messages to the system log. Log ACCEPT and REJECT messages to the system log.

92
doc/upgrade_to_v0.23.0.md Normal file
View File

@ -0,0 +1,92 @@
Upgrading to btrbk-v0.23.0
==========================
In order to keep btrbk simple and intuitive while adding new features,
it became inevitable to change the semantics of the "retention policy"
related configuration options.
What has changed?
-----------------
### Preserve *first* instead of *last* snapshot/backup
btrbk used to *always* transfer the latest snapshot to the target
location, while considering the *last* snapshot/backup of a day as a
daily backup (and also the last weekly as a monthly). This made it
very cumbersome when running btrbk in a cron job as well as manually,
because the last manually created snapshot was immediately transferred
on every run, and used as the daily backup (instead of the one created
periodically by the cron job).
The new semantics are to consider the *first* (instead of *last*)
snapshot of a hour/day/week/month as the one to be preserved, while
only transferring the snapshots needed to satisfy the target retention
policy.
### Preserve snapshots for a minimum amount of time
In order to specify a minimum amount of time in which *all* snapshots
should be preserved, the new "snapshot_preserve_min" and
"target_preserve_min" configuration options were introduced. This was
previously covered by "snapshot_preserve_daily", which caused a lot of
confusion among users.
Upgrading the configuration file: /etc/btrbk/btrbk.conf
-------------------------------------------------------
Please read the description of the "run" command in [btrbk(1)], as
well as the "RETENTION POLICY" section in [btrbk.conf(5)] for a
detailed description. Make sure to understand the new concept, and run
`btrbk --print-schedule dryrun` after updating the configuration.
### Upgrade retention policy
If you want the same behaviour as before:
# replace this:
snapshot_preserve_daily <daily>
snapshot_preserve_weekly <weekly>
snapshot_preserve_monthly <monthly>
# with:
snapshot_preserve_min <daily>d
snapshot_preserve <weekly>w <monthly>m
# ... do the same with "target_preserve_*" options
But what you probably want is something like:
snapshot_preserve_min 5d
snapshot_preserve <daily>d <weekly>w <monthly>m
target_preserve_min no
target_preserve <daily>d <weekly>w <monthly>m *y
This states:
* Keep all snapshots for five days (no matter how many there are)
* Transfer only the first snapshot of a day to the target
* Keep all "first snapshots of a day" for `<daily>` days, etc.
### Upgrade "resume_missing"
If you have a line: "resume_missing yes" somwhere in your config,
simply remove it. btrbk always resumes missing backups.
If you have "resume_missing no", you can imitate this behaviour by
setting:
target_preserve_min latest
target_preserve no
This states: "always transfer the latest snapshot to the target".
[btrbk(1)]: https://digint.ch/btrbk/doc/btrbk.1.html
[btrbk.conf(5)]: https://digint.ch/btrbk/doc/btrbk.conf.5.html

View File

@ -1,11 +1,9 @@
#!/bin/sh #!/bin/bash
# initialise and sanitise the shell execution environment set -e
unset -v IFS set -u
export LC_ALL=C
export PATH='/sbin:/bin:/usr/sbin:/usr/bin'
set -e -u export PATH=/sbin:/bin:/usr/sbin:/usr/bin
enable_log= enable_log=
restrict_path_list= restrict_path_list=
@ -14,70 +12,55 @@ allow_exact_list=
allow_rate_limit=1 allow_rate_limit=1
allow_stream_buffer=1 allow_stream_buffer=1
allow_compress=1 allow_compress=1
compress_list='gzip|pigz|bzip2|pbzip2|bzip3|xz|lzop|lz4|zstd' compress_list="gzip|pigz|bzip2|pbzip2|xz|lzop|lz4|zstd"
# note that the backslash is NOT a metacharacter in a POSIX bracket expression! # note that the backslash is NOT a metacharacter in a POSIX bracket expression!
option_match='-[a-zA-Z0-9=-]+' # matches short as well as long options option_match='-[a-zA-Z0-9=-]+' # matches short as well as long options
file_match_sane='/[0-9a-zA-Z_@+./-]*' # matches file path (equal to ${file_match} in btrbk < 0.32.0) file_match='[0-9a-zA-Z_@+./-]*' # matches file path (equal to $file_match in btrbk)
file_match="/[^']*" # btrbk >= 0.32.0 quotes file arguments: match all but single quote
file_arg_match="('${file_match}'|${file_match_sane})" # support btrbk < 0.32.0
log_cmd() log_cmd()
{ {
local priority="$1" if [[ -n "$enable_log" ]]; then
local authorisation_decision="$2" logger -p $1 -t ssh_filter_btrbk.sh "$2 (Name: ${LOGNAME:-<unknown>}; Remote: ${SSH_CLIENT:-<unknown>})${3:+: $3}: $SSH_ORIGINAL_COMMAND"
local reason="${3-}"
if [ -n "${enable_log}" ]; then
logger -p "${priority}" -t ssh_filter_btrbk.sh "${authorisation_decision} (Name: ${LOGNAME:-<unknown>}; Connection: ${SSH_CONNECTION:-<unknown>})${reason:+: ${reason}}: ${SSH_ORIGINAL_COMMAND}"
fi fi
} }
allow_cmd() allow_cmd()
{ {
local cmd="$1" allow_list="${allow_list}|$1"
allow_list="${allow_list}|${cmd}"
} }
allow_exact_cmd() allow_exact_cmd()
{ {
local cmd="$1" allow_exact_list="${allow_exact_list}|$1"
allow_exact_list="${allow_exact_list}|${cmd}"
} }
reject_and_die() reject_and_die()
{ {
local reason="$1" local reason=$1
log_cmd "auth.err" "btrbk REJECT" "$reason"
log_cmd 'auth.err' 'btrbk REJECT' "${reason}" echo "ERROR: ssh_filter_btrbk.sh: ssh command rejected: $reason: $SSH_ORIGINAL_COMMAND" 1>&2
printf 'ERROR: ssh_filter_btrbk.sh: ssh command rejected: %s: %s\n' "${reason}" "${SSH_ORIGINAL_COMMAND}" >&2
exit 255 exit 255
} }
run_cmd() run_cmd()
{ {
log_cmd 'auth.info' 'btrbk ACCEPT' log_cmd "auth.info" "btrbk ACCEPT"
eval " ${SSH_ORIGINAL_COMMAND}" eval " $SSH_ORIGINAL_COMMAND"
} }
reject_filtered_cmd() reject_filtered_cmd()
{ {
if [ -n "${restrict_path_list}" ]; then if [[ -n "$restrict_path_list" ]]; then
# match any of restrict_path_list, # match any of restrict_path_list with or without trailing slash,
# or any file/directory (matching file_match) below restrict_path # or any file/directory (matching file_match) below restrict_path
path_match="'(${restrict_path_list})(${file_match})?'" path_match="(${restrict_path_list})(/${file_match})?"
path_match_legacy="(${restrict_path_list})(${file_match_sane})?"
else else
# match any absolute file/directory (matching file_match) # match any absolute file/directory (matching file_match)
path_match="'${file_match}'" path_match="/${file_match}"
path_match_legacy="${file_match_sane}"
fi fi
# btrbk >= 0.32.0 quotes files, allow both (legacy)
path_match="(${path_match}|${path_match_legacy})"
if [ -n "${allow_compress}" ]; then if [[ -n "$allow_compress" ]]; then
decompress_match="(${compress_list}) -d -c( -[pT][0-9]+)?" decompress_match="(${compress_list}) -d -c( -[pT][0-9]+)?"
compress_match="(${compress_list}) -c( -[0-9])?( -[pT][0-9]+)?" compress_match="(${compress_list}) -c( -[0-9])?( -[pT][0-9]+)?"
else else
@ -87,8 +70,8 @@ reject_filtered_cmd()
# rate_limit_remote and stream_buffer_remote use combined # rate_limit_remote and stream_buffer_remote use combined
# "mbuffer" as of btrbk-0.29.0 # "mbuffer" as of btrbk-0.29.0
if [ -n "${allow_stream_buffer}" ] || [ -n "${allow_rate_limit}" ]; then if [[ -n "$allow_stream_buffer" ]] || [[ -n "$allow_rate_limit" ]]; then
mbuffer_match='mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?' mbuffer_match="mbuffer -v 1 -q( -s [0-9]+[kmgKMG]?)?( -m [0-9]+[kmgKMG]?)?( -[rR] [0-9]+[kmgtKMGT]?)?"
else else
mbuffer_match= mbuffer_match=
fi fi
@ -98,44 +81,35 @@ reject_filtered_cmd()
stream_in_match="(${decompress_match} \| )?(${mbuffer_match} \| )?" stream_in_match="(${decompress_match} \| )?(${mbuffer_match} \| )?"
stream_out_match="( \| ${mbuffer_match})?( \| ${compress_match}$)?" stream_out_match="( \| ${mbuffer_match})?( \| ${compress_match}$)?"
# `grep`s `-q`-option is not used as it may cause an exit status of `0` even
# when an error occurred.
allow_stream_match="^${stream_in_match}${allow_cmd_match}${stream_out_match}" allow_stream_match="^${stream_in_match}${allow_cmd_match}${stream_out_match}"
if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${allow_stream_match}" >/dev/null 2>/dev/null; then if [[ $SSH_ORIGINAL_COMMAND =~ $allow_stream_match ]] ; then
return 0 return 0
fi fi
exact_cmd_match="^(${allow_exact_list})$"; exact_cmd_match="^(${allow_exact_list})$";
if printf '%s' "${SSH_ORIGINAL_COMMAND}" | grep -E "${exact_cmd_match}" >/dev/null 2>/dev/null; then if [[ $SSH_ORIGINAL_COMMAND =~ $exact_cmd_match ]] ; then
return 0 return 0
fi fi
local formatted_restrict_path_list="$(printf '%s' "${restrict_path_list}" | sed 's/|/", "/g')" reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${restrict_path_list//|/\", \"}\")}"
reject_and_die "disallowed command${restrict_path_list:+ (restrict-path: \"${formatted_restrict_path_list}\")}"
} }
# check for "--sudo" option before processing other options # check for "--sudo" option before processing other options
sudo_prefix= sudo_prefix=
for key in "$@"; do for key; do
if [ "${key}" = '--sudo' ]; then [[ "$key" == "--sudo" ]] && sudo_prefix="sudo -n "
sudo_prefix='sudo -n '
fi
if [ "${key}" = '--doas' ]; then
sudo_prefix='doas -n '
fi
done done
while [ "$#" -ge 1 ]; do while [[ "$#" -ge 1 ]]; do
key="$1" key="$1"
case "${key}" in case $key in
-l|--log) -l|--log)
enable_log=1 enable_log=1
;; ;;
--sudo|--doas) --sudo)
# already processed above # already processed above
;; ;;
@ -180,7 +154,7 @@ while [ "$#" -ge 1 ]; do
;; ;;
*) *)
printf 'ERROR: ssh_filter_btrbk.sh: failed to parse command line option: %s\n' "${key}" >&2 echo "ERROR: ssh_filter_btrbk.sh: failed to parse command line option: $key" 1>&2
exit 255 exit 255
;; ;;
esac esac
@ -189,21 +163,19 @@ done
# NOTE: subvolume queries are NOT affected by "--restrict-path": # NOTE: subvolume queries are NOT affected by "--restrict-path":
# btrbk also calls show/list on the mount point of the subvolume # btrbk also calls show/list on the mount point of the subvolume
allow_exact_cmd "${sudo_prefix}btrfs subvolume (show|list)( ${option_match})* ${file_arg_match}"; allow_exact_cmd "${sudo_prefix}btrfs subvolume (show|list)( ${option_match})* ${file_match}";
allow_cmd "${sudo_prefix}readlink" # resolve symlink allow_cmd "${sudo_prefix}readlink" # resolve symlink
allow_exact_cmd "${sudo_prefix}test -d ${file_arg_match}" # check directory (only for compat=busybox) allow_exact_cmd "${sudo_prefix}test -d ${file_match}" # check directory (only for compat=busybox)
allow_exact_cmd 'cat /proc/self/mountinfo' # resolve mountpoints allow_exact_cmd "cat /proc/self/mountinfo" # resolve mountpoints
allow_exact_cmd 'cat /proc/self/mounts' # legacy, for btrbk < 0.27.0 allow_exact_cmd "cat /proc/self/mounts" # legacy, for btrbk < 0.27.0
# remove leading "|" on alternation lists # remove leading "|" on alternation lists
allow_list="${allow_list#\|}" allow_list=${allow_list#\|}
allow_exact_list="${allow_exact_list#\|}" allow_exact_list=${allow_exact_list#\|}
restrict_path_list="${restrict_path_list#\|}" restrict_path_list=${restrict_path_list#\|}
case "${SSH_ORIGINAL_COMMAND}" in case "$SSH_ORIGINAL_COMMAND" in
*\.\./*) reject_and_die 'directory traversal' ;; *\.\./*) reject_and_die 'directory traversal' ;;
*'
'*) reject_and_die 'unsafe character LF' ;;
*\$*) reject_and_die 'unsafe character "$"' ;; *\$*) reject_and_die 'unsafe character "$"' ;;
*\&*) reject_and_die 'unsafe character "&"' ;; *\&*) reject_and_die 'unsafe character "&"' ;;
*\(*) reject_and_die 'unsafe character "("' ;; *\(*) reject_and_die 'unsafe character "("' ;;
@ -212,7 +184,7 @@ case "${SSH_ORIGINAL_COMMAND}" in
*\<*) reject_and_die 'unsafe character "<"' ;; *\<*) reject_and_die 'unsafe character "<"' ;;
*\>*) reject_and_die 'unsafe character ">"' ;; *\>*) reject_and_die 'unsafe character ">"' ;;
*\`*) reject_and_die 'unsafe character "`"' ;; *\`*) reject_and_die 'unsafe character "`"' ;;
*\|*) [ -n "${allow_compress}" ] || [ -n "${allow_rate_limit}" ] || [ -n "${allow_stream_buffer}" ] || reject_and_die 'unsafe character "|"' ;; *\|*) [[ -n "$allow_compress" ]] || [[ -n "$allow_rate_limit" ]] || [[ -n "$allow_stream_buffer" ]] || reject_and_die 'unsafe character "|"' ;;
esac esac
reject_filtered_cmd reject_filtered_cmd