Storage snapshots for consistent backup of postgresql - different data and log volumes

Question

We are running many Linux VM's in a vmware/shared storage environment, each running its own instance of postgreSQL (a mix of 9.0 and 9.3). Currently, the entire VM sits on a single root partition/volume, and we've had great success (~8 years) using storage-based snapshots of the underlying VMFS volumes for backup/restore process (and replication to our DR site).

Due to the architecture of our storage, it would be advantageous to separate postgres WAL files to a non-cached, mostly-write volume to give us less cache churn on the storage side. With our storage (Nimble Storage), we can assign both volumes to a single protection/snapshot group, but I haven't been able to elicit from our vendor that the snapshots will happen at EXACTLY the same time across all volumes in the protection group - it likely will, but there's always that chance that its milliseconds apart.

To that end, we ran some experiments, all while writing data to the DB as fast as possible using pg_bench. After the experiments, we restored our snapshot'ed volumes and started the VM+postgres

• Snapshot both data and log volumes close to simultaneously - result: DB recovered
• Snapshot data volume first, log volume ~1 minute later - result: DB recovered
• Snapshot log volume first, data volume ~1 minute later - result: DB recovered
• Snapshot log volume first, data volume ~3 minutes later, after a WAL checkpoint wrote new data to datafiles: result: DB recovered

So testing seem to tell us as long as both snapshots are consistent at the volume level, and relatively close together, you get a consistent copy of the DB, based on the time of the WAL/Log volume snapshot.

My question: Is this safe? What are the corner cases we are missing in our testing, and what could go wrong?

Postgres' doc indicates this is not safe, but testing seems to indicate its pretty robust: http://www.postgresql.org/docs/9.1/static/backup-file.html

如果您的数据库分布在多个文件系统中，则可能无法获得所有卷的完全同时冻结的快照。例如，如果您的数据文件和 WAL 日志在不同的磁盘上，或者如果表空间在不同的文件系统上，则可能无法使用快照备份，因为快照必须同时进行。在这种情况下信任一致性快照技术之前，请仔细阅读您的文件系统文档。

注意：是的，我们知道确保它们一致的其他选项，例如将 PostgreSQL 置于热备份模式或使用我们存储的 VMware 集成来静默 VM 本身，但我们正在寻找仅存储解决方案以提高速度、便利性，并对我们的客户零影响。

Answer 1

您引用的文档说明了一切，但如果您想尝试验证供应商关于同时拍摄的快照的声明，我不会责怪您。也许发现问题的一种方法是对WAL 系统进行更具体的压力测试。

例如，除了基于 pgbench 的测试之外，尝试添加随机调用以pg_switch_xlog()强制日志轮换、更短和更长的检查点间隔（缩短和延长 和checkpoint_timeout）checkpoint_timeout，甚至使用小或大的 wal 文件大小。

除非我遗漏了什么，否则您的快照不是同时拍摄的，我会将您恢复的数据库归因于一些幸运的时机。在最后一种情况下，假设您在当前 xlog 位置为 时拍摄了日志快照0/A1C0FFEE。然后，系统上有 3 分钟的特别重的负载，这会导致 WAL 文件的完整循环，并且您的数据库现在处于0/DEADBEEF拍摄数据快照时。当您尝试恢复时，数据快照时写入的 WAL 文件早已消失，恢复将失败。