Write to a file without rewriting constant parts

This little tool writes its standard input to a file.

It only writes to parts that actually change. This is very useful for keeping incremental backups small when the file in question is mostly-constant.
Example: Backing up database tables that are append-only or change seldom or never.
pull/604/head
Matthias Urlichs 2024-10-31 14:45:19 +01:00 committed by GitHub
parent 18ddc65979
commit 0fa156e240
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 47 additions and 0 deletions

47
contrib/tools/write_to.py Normal file
View File

@ -0,0 +1,47 @@
#!/usr/bin/python3
"""
This simple tool reads data from stdin and writes them to a file,
carefully *not* overwriting blocks that already have the desired content.
Usage example:
cd /mnt/backup/mysql
mysql -Ne "show databases;" | grep -v '_schema$' | while read db ; do
mysql -Ne "show tables;" "$db" | while read t ; do
f="$db/$t.db"
if ! test -f "$f" ; then
mkdir -p $db
touch "$f"
fi
echo "mysqldump '$db' '$t' | write_to '$f'"
done
done | parallel
The effect is that when your tables don't change/ are only appended to,
your files are not overwritten and thus your incremental backups stay
nice and small.
"""
import sys
if len(sys.argv) != 2:
raise RuntimeError(f"Usage: {sys.argv[0]} destfile")
fpos=0
bs=4096
fi = sys.stdin.buffer
with open(sys.argv[1], "rb+") as fo:
fo.seek(0)
while True:
od=fi.read(bs)
if not od:
fo.truncate(fpos)
break
nd=fo.read(bs)
if nd != od:
fo.seek(fpos)
fo.write(od)
fpos += len(od)