将 Linux 配置、脚本和文档备份到 Gmail

Win*_*nix 6 gmail cloud bash backup

在 gmail.com 上,我看到了这个:

已使用 15 GB 的 0.38 GB (2%)

我的 30 GB 分区中有 20 GB 装满了 Ubuntu 的东西,但可能只有 1 GB 的重要文件,那么为什么不使用这个免费的云存储进行备份呢?

是否已经有一个程序可以做到这一点?反正我可能会写一个,但在开始之前想问问。

我还想要文件压缩和分段,因为有些系统有 10 MB 的附件限制(比如工作中的那个)。

谢谢 :)

Win*_*nix 7

编辑 2021 年 5 月 7 日

下一节中的历史帖子

答案的原始部分和第一次主要编辑在接下来的部分中保持不变,以供有关反复试验的历史参考。

2021 年 4 月 29 日是旧脚本最后一次成功运行。开始于2021年4月30日谷歌开始扫描备份存档,看是否有文件名与结束.exe.js等等。这导致谷歌拒绝附件。

现在的解决方案是使用密码对备份存档文件进行加密。

更新脚本

请记住用您的真实单词替换下面的大写单词:

HomeDir="/home/USER_NAME"               # Required for cron compatibility
EmailAddr="EMAIL_ADDR@gmail.com"        #  where $HOME is not setup for us.
Run Code Online (Sandbox Code Playgroud)

每日备份.sh

#!/bin/bash

# NAME: daily-backup.sh
# PATH: /mnt/e/bin
# DESC: Backup scripts, documents and configuration files to .tar
# PARM: $1 = Filename for backup (without .tar extension)

# DATE: July 11, 2017. Modified May 7, 2021.

# NOTE: Requires zip package because pbzip2 doesn't provide encryption:
#           sudo apt install zip


HomeDir="/home/USER_NAME"               # Required for cron compatibility
EmailAddr="EMAIL_ADDR@gmail.com"        #  where $HOME is not setup for us.

# tar removes leading / to make restores painless, suppress this error for cron
exec 2> >(grep -v "Removing leading '/' from member names" >&2)


# NOTE: To recover backup, download .64 backup format from google and use:
#       base64 -di backup.tar.gz.64 > backup.tar.gz

# NOTE: To include MBR (Master Boot Record) in backup create an image using:
#       sudo dd if=/dev/sda of="$HOME/.mbr.sav" bs=512 count=1

# NOTE: CLONE CURRENT INSTALLATION TO NEW MACHINE
#       =========================================

#       To restore use Live USB to install Ubuntu alongside Windows 10
#       Connect to network with password xxxxxxxxx

#       Install Google Chrome
#       (https://askubuntu.com/questions/510056/how-to-install-google-chrome):

#           wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub 
#               | sudo apt-key add
#           echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/
#               stable main' | sudo tee /etc/apt/sources.list.d/google-chrome.list
#           sudo apt update
#           sudo apt install google-chrome-stable

#       Open gmail.com and download attachment `$1` which is usually called
#           Backup-yymmdd-DayOfWeekName.tar

#       Create home/ sub-directories which tar doesn't create automatically:
#           mkdir ~/bin
#           mkdir ~/eyesome
#           mkdir ~/python
#           mkdir ~/gmail
#           mkdir ~/roboto
#           mkdir ~/sony

#       Restore the daily backup using:
#           download .64 backup format from google and use:
#           base64 -di backup.tar.gz.64 > backup.tar.gz
#           sudo tar -xvf Backup-yymmdd-DayOfWeekName.tar -C /

#       Patch /etc/default/grub with new machine parameters, ie for nvme use:
#           acpiphp.disable=1

#       Use `sudo apt install aptitude-common`
#       Clone packages using `aptitude-create-state-bundle` on Source
#       Copy state-bundle.tar file from Source to Target machine
#       Restore packages using `aptitude-run-state-bundle` on Target

#       Manually copy ~/Pictures, ~/Videos, etc. not in daily backup.

#       sudo update-grub        # NVMe suspend/resume acpiphp.disable=1
#       sudo update-initramfs   # to get plymouth sunrise splash screen

if [[ $# -ne 1 ]]; then
    echo 'One argument required for file name, e.g. "Backup-2017-10-21-Saturday"'
    echo '.tar will automatically be added as a file extension'
    exit 1
fi

Filename="$1.tar"

cd $HomeDir || exit 1                   # Change to homedir, exit on failure

dpkg --get-selections > .packages       # List of installed applications

tar -cpf "$Filename" bin                # create .tar & add user scripts
# find all "$HOME/." files using "-maxdepth" to skip ".config" directory
# Fpr example this includes .roboto
find .* -maxdepth 0 -type f -exec tar -rpf "$Filename" {} +
tar -rpf "$Filename" .config/autostart  # autostart programs configuration
tar -rpf "$Filename" .config/mserve     # music library and location playlists
tar -rpf "$Filename" .local/share/nautilus/scripts
tar -rpf "$Filename" Desktop            # files and links on desktop
tar -rpf "$Filename" Documents/*.od*    # Libre Office: *.ods, *.odt, etc.
tar -rpf "$Filename" eyesome            # ~/eyesome - Development version
tar -rpf "$Filename" gmail/*.py         # gmail daily backup management scripts
tar -rpf "$Filename" gmail/*.sh         #  and configuration files. Excludes
tar -rpf "$Filename" gmail/go           #  message data files 
tar -rpf "$Filename" gmail/BackupSets   #  which exceed 8 MB and can be
tar -rpf "$Filename" gmail/BackupDays   #  downloaded if needed again.
tar -rpf "$Filename" python/*.py        # Python scripts
tar -rpf "$Filename" roboto/roboto      # Script only, no download files
tar -rpf "$Filename" sony               # Sony TV via REST API over HTTP
#July 20, 2018 - /boot/grub takes 5MB+
#tar -rpf "$Filename" /boot/grub        # Custom grub fonts and splash...
tar -rpf "$Filename" /etc/apt           # 3rd party keys, repositories, etc.
tar -rpf "$Filename" /etc/cron*         # crontab, cron.d, cron.daily, etc
tar -rpf "$Filename" /etc/default/grub  # bootstrap loader
tar -rpf "$Filename" /etc/environment   # PATH backup
tar -rpf "$Filename" /etc/fstab         # UUID partitions
tar -rpf "$Filename" /etc/grub.d        # 00_header, etc. changes
tar -rpf "$Filename" /etc/hosts         # IP configuration
tar -rpf "$Filename" /etc/NetworkManager # .conf and .dispatcher.d
tar -rpf "$Filename" /etc/rc.local      # Startup script: calls zaprestore.
tar -rpf "$Filename" /etc/sudoers       # 120 minute sudo, stars in password
tar -rpf "$Filename" /etc/systemd       # systemd files: login.conf, etc.
tar -rpf "$Filename" /etc/udev/rules.d
tar -rpf "$Filename" /etc/X11/xorg.conf # /etc/X11/xorg.conf.d crashes
tar -rpf "$Filename" /lib/systemd/system-sleep
tar -rpf "$Filename" /lib/systemd/system/rc.local.service
tar -rpf "$Filename" /mnt/e/bin         # /mnt/e - shared WSL + Linux scripts
tar -rpf "$Filename" /mnt/e/Documents
tar -rpf "$Filename" /usr/local/bin     # add global root-based scripts
tar -rpf "$Filename" /usr/share/plymouth   # ... screen (plymouth)
tar -rpf "$Filename" /usr/share/grub/themes/Tuxkiller2/

# gsettings modified from default. To restore see answer at:
# https://askubuntu.com/questions/420527/how-to-dump-all-the-manully-altered-gsettings-keys
dconf dump / > dump.dconf
tar -rpf "$Filename" dump.dconf

# Get list of all filenames in backup archive (tar file).
tar -tvf "$Filename" > BackupLog    # list filenames and sizes
chmod a+w BackupLog                 # give user delete access

# From: https://internetlifeforum.com/gmail/2251-gmail-some-file-types-blocked-fix-how-go-around/
#   to create base64 file:
#       cat archive.tar.gz | base64 > file
#   then i sent the file via email:
#   then mail was delivered properly! Then when one need to get readable archive 
#   again, he need to decode it by base64. In my case i do it with:
#       cat file | base64 -d > decodedarchive.tar.gz

# NOTE: To recover backup, download .64 backup format from google and use:
#       base64 -di backup.tar.gz.64 > backup.tar.gz     # < May 6, 2021
#       base64 -di backup.tar.zip.64 > backup.tar.zip   # > May 6, 2021

# https://support.google.com/mail/answer/6590?hl=en#zippy=%2Cmessages-that-have-attachments
# May 6 2021 - Google has started scanning contents and rejecting zipped files
#              If they contain certain file extensions. It reports an error
#              without giving the filename. Solution is to use zip with 
#              password protection emcryption. Filename is backup.tar.zip
zip --password daily-backup --quiet "$Filename.zip" "$Filename"

# Attachment has to be base64 to avoid google errors: backup.tar.zip.64
# NOtE: prior to May 7, 2021 gzip was used and file extenion was '.gz.64'
Filename64="$Filename.zip.64"
cat "$Filename.zip" | base64 > "$Filename64"

#echo "Compare size to 24 MB"
size=$(stat --printf="%s" "$Filename64")
if [[ $size -gt 24000000 ]] ; then
    echo "=================== Backup > 24 MB aborting. ========================"
    ls -la "$Filename64"    # list backup file attributes
    tar -tvf "$Filename"    # list backup file contents with total size below
    tar -tvf "$Filename" | awk '{sum += $3} END {print sum}'
    exit 1                  # Abort backup
fi

# email the backup as base64 attachment with list of files in message body
mail -a "$Filename64" -s "$Filename64" "$EmailAddr" < BackupLog

rm "$Filename" "$Filename64" "$Filename.zip"

exit 0
Run Code Online (Sandbox Code Playgroud)

编辑 2019 年 7 月 9 日

下一节的历史帖子

答案的原始部分在下一节中保持不变,以供试错的历史参考

创建 .tar 文件的备份脚本

这是当前的备份脚本:

#!/bin/bash

# NAME: daily-backup.sh
# PATH: /mnt/e/bin
# DESC: Backup scripts, documents and configuration files to .tar

# DATE: July 11, 2017. Modified July 7, 2019.

HomeDir="/home/USER_NAME"                    # Required for cron compatibility
EmailAddr="EMAIL_NAME@gmail.com"

# PARM: 1=backup file name. Extension .tar.gz automatically appended.

# NOTE: To include MBR (Master Boot Record) in backup create an image using:
#       sudo dd if=/dev/sda of="$HOME/.mbr.sav" bs=512 count=1

# NOTE: CLONE CURRENT INSTALLATION TO NEW MACHINE
#       =========================================

#       To restore use Live USB to install Ubuntu alongside Windows 10
#       Connect to network with password xxxxxxxxx

#       Install Google Chrome
#       (https://askubuntu.com/questions/510056/how-to-install-google-chrome):

#           wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub 
#               | sudo apt-key add
#           echo 'deb [arch=amd64] http://dl.google.com/linux/chrome/deb/
#               stable main' | sudo tee /etc/apt/sources.list.d/google-chrome.list
#           sudo apt update
#           sudo apt install google-chrome-stable

#       Open gmail.com and download attachment `$1` which is usually called
#           Backup-yymmdd-DayOfWeekName.tar

#       Make missing home/bin directory which tar doesn't create automatically:
#           mkdir ~/bin

#       Restore the daily backup using:
#           sudo tar -xvf Backup-yymmdd-DayFfWeekName.tar -C /
#           yar -xvf Backup-yymmdd-DayFfWeekName.tar -C /

#       Patch /etc/default/grub with new machine parameters, ie for nvme use:
#           acpiphp.disable=1

#       Use `sudo apt install aptitude-common`
#       Clone packages using `aptitude-create-state-bundle` on Source
#       Copy state-bundle.tar file from Source to Target machine
#       Restore packages using `aptitude-run-state-bundle` on Target

#       Manually copy ~/Pictures, ~/Videos, etc. not in daily backup.

#       sudo update-grub        # NVMe suspend/resume acpiphp.disable=1
#       sudo update-initramfs   # to get plymouth sunrise splash screen

if [[ $# -ne 1 ]]; then
    echo 'One argument required for file name, e.g. "Backup-2017-10-21-Saturday"'
    echo '.tar will automatically be added as a file extension'
    exit 1
fi

Filename="$1.tar"

cd $HomeDir ||
    exit 1

dpkg --get-selections > .packages       # List of installed applications

tar -cvpf "$Filename" bin               # create .tar & add user scripts
tar -rvpf "$Filename" .config/autostart # autostart programs configuration
tar -rvpf "$Filename" /usr/local/bin    # add global root-based scripts
tar -rvpf "$Filename" /etc/cron*        # crontab, cron.d, cron.daily, etc
tar -rvpf "$Filename" /etc/system*      # systemd files: login.conf, etc.
tar -rvpf "$Filename" /lib/systemd/system-sleep
tar -rvpf "$Filename" /etc/rc.local     # Startup script: calls zaprestore.
tar -rvpf "$Filename" /etc/sudoers      # 120 minute sudo, stars in password
tar -rvpf "$Filename" /etc/environment  # PATH backup
tar -rvpf "$Filename" /etc/default/grub # bootstrap loader
#July 20, 2018 - /boot/grub takes 5MB+
#tar -rvpf "$Filename" /boot/grub        # Custom grub fonts and splash...
tar -rvpf  "$Filename" /usr/share/plymouth   # ... screen (plymouth)
#included above tar -rvpf "$Filename" /usr/share/plymouth/themes/earth-sunrise/
tar -rvpf "$Filename" /usr/share/grub/themes/Tuxkiller2/
tar -rvpf "$Filename" /etc/grub.d       # 00_header, etc. changes
tar -rvpf "$Filename" Desktop           # files and links on desktop
tar -rvpf "$Filename" Documents/*.od*   # Libre Office: *.ods, *.odt, etc.

# Trusted keys to install from third party PPAs
tar -rvpf "$Filename" /etc/apt/trusted.gpg
tar -rvpf "$Filename" /etc/apt/trusted.gpg.d

# Sources for repositories - 1) Main single file - 2) directory of files
tar -rvpf "$Filename" /etc/apt/sources.list
tar -rvpf "$Filename" /etc/apt/sources.list.d

# find all $HOME/.config files and add to .tar
find .* -maxdepth 0 -type f -exec tar -rvf "$Filename" {} +

# Nautilus custom scripts
tar -rvpf "$Filename" .local/share/nautilus/scripts

# /etc/udev rules
tar -rvpf "$Filename" /etc/udev/rules.d

# /etc/rc.local
tar -rvpf "$Filename" /etc/rc.local

# /etc/X11/xorg.conf.d
tar -rvpf "$Filename" /etc/X11/xorg.conf.d

# /mnt/e - shared WSL + Linux
tar -rvpf "$Filename" /mnt/e/bin
tar -rvpf "$Filename" /mnt/e/Documents

# ~/eyesome - Development version
tar -rvpf "$Filename" eyesome

# ~/gmail - Python and Bash scripts but NOT huge data files
tar -rvpf "$Filename" gmail/*.py
tar -rvpf "$Filename" gmail/*.sh
tar -rvpf "$Filename" gmail/go
tar -rvpf "$Filename" gmail/BackupSets
tar -rvpf "$Filename" gmail/BackupDays

echo "Complete file list with sizes..."
tar -tvf "$Filename" > BackupLog    # list filenames and sizes
chmod a+w BackupLog                 # give user delete access

echo "Compressing with gzip..."
gzip "$Filename"
Filename="$Filename.gz"

echo "Emailing: $EmailAddr"

# From: https://internetlifeforum.com/gmail/2251-gmail-some-file-types-blocked-fix-how-go-around/
# cat archive.tar.gz | base64 > file
# then i sent the file via email:
# echo "Base64 encoded file" | mutt -a file -s subject -- mymail@gmail.com
# then mail was delivered properly! Then when one need to get readable archive 
# again, he need to decode it by base64. In my case i do it via linux command line:
# cat file | base64 -d > decodedarchive.tar.gz

Filename64="$Filename.64"
cat "$Filename" | base64 > "$Filename64"
mail -a "$Filename64" -s "$Filename64" "$EmailAddr" < BackupLog

ls -la "$Filename" "$Filename64"
rm     "$Filename" "$Filename64"

exit 0
Run Code Online (Sandbox Code Playgroud)

USER_NAME用您的用户名替换上面的内容。替换EMAIL_NAME@gamil.com为您的实际 Gmail 地址。将目录更改为/mnt/e/bin存储 bash 脚本的目录。保存文件并退出。然后使用:

chmod a+x /mnt/e/bin/backup
Run Code Online (Sandbox Code Playgroud)

这使脚本可执行。

注意 MBR(主引导记录)是如何保存到备份的。如脚本注释中所述,需要单独的早期创建~/.mbr.savusing步骤sudo dd ...

注意dpkg --get-selections线路。这将创建备份所有已安装应用程序名称的列表。

自动发送电子邮件的最简单方法

使用 ssmtp 发送电子邮件警报中,我们找到了从终端或脚本自动发送电子邮件的最简单方法。安装步骤很简单:

sudo apt install ssmtp
sudo nano /etc/ssmtp/ssmtp.conf
# Change "MyEmailAddress" and "MyPassword" to your own.
Run Code Online (Sandbox Code Playgroud)

有一个步骤没有提到;Google 会向您发送一封电子邮件,确认您希望允许“不太安全”的应用程序使用您的帐户发送邮件:

gmail 为电子邮件启用安全性较低的应用

安装和配置后,还需要ssmpt一个软件包才能将 .tar 备份文件附加到电子邮件中:

sudo apt install sharutils
Run Code Online (Sandbox Code Playgroud)

该软件包包含uuencode需要转换二进制文件以进行传输的程序。

cron每天设置以调用备份脚本

创建/etc/cron.daily/daily-backup包含以下内容的文件:

#!/bin/sh
#
# NAME: daily-backup
# DESC: A .tar backup file is created, emailed and removed.
# DATE: Nov 25, 2017.
# CALL: WSL or Ubuntu calls from /etc/cron.daily/daily-backup
# PARM: No parameters but /etc/ssmtp/ssmtp.conf must be setup

# NOTE: Backup file name contains machine name + Distro
#       Same script for user with multiple dual boot laptops
#       Single machine should remove $HOSTNAME from name
#       Single distribution should remove $Distro

sleep 30 # Wait 30 seconds after boot

# Running under WSL (Windows Subsystem for Ubuntu)?
if cat /proc/version | grep Microsoft; then
    Distro="WSL"
else
    Distro="Ubuntu"
fi

today=$( date +%Y-%m-%d-%A )
/mnt/e/bin/daily-backup.sh Daily-$(hostname)-$Distro-backup-$today
Run Code Online (Sandbox Code Playgroud)

保存文件,退出并使用:

chmod a+x /etc/cron.daily/daily-backup
Run Code Online (Sandbox Code Playgroud)

这使脚本可执行。

每天早上什么 cron 给你发邮件

每天早上,/etc/cron.daily/daily-backup运行后都会cron向您发送两封电子邮件。一个是备份Backup-YYYY-MM-DD.tar文件,在我的情况下是 5.2 MB,我无法向您展示。另一个是列出该tar命令已报告给备份中的所有文件cron

Anacron <Me@gmail.com>
6:58 AM (1 hour ago)

to root, bcc: me 
/etc/cron.daily/daily-backup:
bin/
bin/.websync.new
bin/log-gsu-del
bin/now
  (... SNIP ...)
.xscreensaver
.xsession-errors
.xsession-errors.old
Run Code Online (Sandbox Code Playgroud)

概括

花了一个月的时间等待答案,然后花了一个月的时间写出答案,但是,现在该项目已完成。展望未来,只需将其他目录添加到备份脚本即可。

下一个项目将是完整备份,但它有 6 GB 大,并将复制到 gdrive (Google Drive),因为 gmail 限制为 25 MB。/usr/local/bin/full-backup如果您有兴趣,该脚本将被调用并包含在此处:

#!/bin/bash

# NAME: full-backup
# PATH: $HOME/bin
# DESC: Full system backup - must call with SUDO

# DATE: July 16, 2017. Modified July 26, 2017.

apt autoclean   # reduces size of /var/cache/apt/archives

cd /tmp         # tar must be created in directory not backed up.

time tar -cvpzf backup.tar.gz \
--exclude=/backup.tar.gz \
--exclude=/proc \
--exclude=/tmp \
--exclude=/mnt \
--exclude=/dev \
--exclude=/sys \
--exclude=/media \
--exclude=/usr/src/linux-headers* \
--exclude=/home/Me/.cache \
--exclude=/var/log \
--exclude=/var/run/ \
--exclude=/run \
--exclude=/var/cache/apt/archives /
Run Code Online (Sandbox Code Playgroud)

历史部分

随着探索可用选项,这将更像是“旅程”而不是答案。

首先备份对您最重要的内容

自 2016 年 8 月以来,我在两个目录中投入了大部分时间:

/home/rick/bin
/usr/local/bin
Run Code Online (Sandbox Code Playgroud)

当我第一次tar使用这两个目录创建一个文件(磁带存档)并尝试通过电子邮件将它们发送给我自己时,我收到了这个错误:

gmail 25MB 限制

gmail.com 不接受大于 25 MB 的文件

超过 10 个月编写的两个脚本目录怎么会大于 25 MB?经过仔细检查,它们 > 190 MB。什么?

结果是为测试目的创建的单个文件:

-rw-rw-r--  1 rick rick 191143744 Dec 23 17:27 log-gsu-gedit.tst
Run Code Online (Sandbox Code Playgroud)

所以删除这个测试文件并重新运行命令:

tar -cvf scripts-2017-06-05.tar /home/rick/bin
tar -rvf scripts-2017-06-05.tar /usr/local/bin
Run Code Online (Sandbox Code Playgroud)

第一个命令.tar使用脚本文件的一个目录创建文件,第二个命令使用脚本文件的第二.tar个目录附加到文件。

.tar文件现在更可观的大小为 1.3 MB:

-rw-rw-r-- 1 rick rick 1341440 Jun  5 17:27 scripts-2017-06-05.tar
Run Code Online (Sandbox Code Playgroud)

最简单的方法是通过电子邮件作为附件

现在.tar文件已创建,只需进入gmail.com并将文件作为附件通过电子邮件发送给您自己。在下一步中,我们需要一个cron每天创建文件并使用MTA(邮件传输代理)自动通过电子邮件发送的作业。需要在 gmail.com 中设置一个选项来删除所有这些超过 30 天的电子邮件。这样只会存储大约 400 MB 的总脚本备份。


编辑 2017 年 6 月 25 日

今晚我发现一些配置文件很难备份,直到我偶然发现了这个线程。有问题的文件在我的主目录中:

.bashrc
.conkyrc
.websync # one of my own databases
.bafman* # Another one of my own databases
Run Code Online (Sandbox Code Playgroud)

使用上面的链接,我创建了一个名为的脚本~/bin/backup