FreeNAS 作为虚拟文件服务器的风险

Question

关于 FreeNAS 是否可以作为虚拟机运行存在很多争论。

官方的立场是可以，但需要额外的配置。

如果我不能保证我可以遵循这些建议，那么与我运行带有 EXT4/XFS 的普通 Linux 系统或带有 UFS 的 FreeBSD 相比，我是否更容易出现故障——尤其是灾难性故障？

具体来说，假设我无法进行 PCI 直通，也无法禁用写入缓存。此外，我将只有一个虚拟磁盘用于存储（由硬件 RAID 支持的 VMDK），因此没有 RAIDZ。显然，会有备份。

编辑：为了澄清我为什么要这样做 - 我需要一个文件服务器，这是我必须使用的基础设施。如果我需要，我可以获得额外的虚拟磁盘来设置 RAIDZ，否则就是这样。我正在寻找一个好的文件服务器解决方案，而 FreeNAS 似乎符合要求。除了关于虚拟化 ZFS 以及如何丢失所有数据和损坏备份的所有这些可怕的警告之外。

我意识到在这个基础设施上部署 FreeNAS 是有风险的。我的问题是：它是否比替代方案风险更大？

EDIT2：我似乎无法传达我的意图。带有 ZFS 的 FreeNAS 是一个坚如磐石的 NAS 平台。但是，从我所读到的内容来看，如果您在标准 VM 配置上运行它，那么使 ZFS 作为裸机文件服务器更可靠的功能似乎实际上可能对您不利。如果是这样，那么在标准 VM 设置上使用不同的文件系统是更好的选择（即没有直接 IO，启用写缓存）。这是一个正确的评估吗？

Answer 1

General answer

If I cannot guarantee that I can follow these recommendations, am I more vulnerable to failure - especially catastrophic failure - than if I run a vanilla Linux system with EXT4/XFS, or FreeBSD with UFS?

The risks are different and not directly comparable.

• I would always prefer a ZFS system, even without redundant vdevs, if only for the knowledge of data integrity (even if I have to restore from backup, I like to know that I have to restore from backup, instead of silent corruption I am not even aware of). Also features like send/recv or snapshots make your live much easier and have nothing to do with integrity.
• Speaking of catastrophic failure, only backups will prevent you from that, and you need them even if your normal system is highly reliable, so it makes sense to first start with the backups (as you already did) to get that out of the way and then think about what other quality of service you require and with which downsides you can live with.
• In theory, more complex systems are more error-prone, but as all mentioned file systems are over 10 years old, actively used and maintained, I would say the majority of bugs has been ironed out already (which does not mean there are none left, of course).
• One may argue that copy-on-write file systems are inherently safer because they never overwrite live data and therefore cannot corrupt it. I assume this risk is more theoretical and much more influenced by other things, like the actual implementation and the handling of metadata.

Specific to your case

If you look at the referenced recommendations and dissect them, you notice a few things:

1. If you are not using PCI passthrough (more on that below), then you must disable the scrub tasks in ZFS. The hardware can “lie” to ZFS so a scrub can do more damage than good, possibly even permanently destroying your zpool.

Scrub just reads every block of the underlying vdevs and verifies their checksums. If your virtual disk does not cope with this, it is garbage and you should be concerned about it, not about ZFS. On the other hand, if your virtual disks are already checksummed on the SAN, your additional scrub will do nothing except cause additional I/O (it is useless).

2. The second precaution is to disable any write caching that is happening on the SAN, NAS, or RAID controller itself. A write cache can easily confuse ZFS about what has or has not been written to disk. This confusion can result in catastrophic pool failures.

This is good advice if you don't trust the hardware. The downside is considerably lower performance, of course. You may also have no control ofer the SAN settings, so you need to treat it as a cheap disk you bought from ebay and slapped into your system - anything can happen, at least in theory.

3. Using a single disk leaves you vulnerable to pool metadata corruption which could cause the loss of the pool. To avoid this, you need a minimum of three vdevs, either striped or in a RAIDZ configuration. Since ZFS pool metadata is mirrored between three vdevs if they are available, using a minimum of three vdevs to build your pool is safer than a single vdev. Ideally vdevs that have their own redundancy are preferred.

This is okay as general advice, but a bit of a nitpick. Assuming your SAN is bad, this will help you in certain cases (with much luck at least). Assuming your SAN is good, this does nothing and just costs you space and performance. It is much better in my opinion to make sure that the chain from physical disks to SAN to network to VM host to VM guest is equally good, so you don't have to do everything again in each layer.

FreeNAS vs others

A word about the FreeNAS recommendations - they are certainly okay as recommendations, that is, guidelines or tips for the general audience. If you follow them, you will not be worse off then otherwise, and might even be better off. Then again, they are stern-worded, as seems to be the usual tone in the FreeNAS community (judging from certain forum posters at least). I guess they just want to be on the safe side with that. I have always preferred the ZFS Best Practices guide, because it is worded pretty neutral and just presents facts, leaving it up to you to decide.

It's also interesting that according to the FreeNAS docs and forums, you will die a gruesome death if you dare to run a ZFS system for file services with less than a pitiful 4GB of RAM, while on the mailing lists of OmniOS (or SmartOS or illumos or Nexenta, I don't remember at the moment) people tested systems with 512MB of RAM and shared their suggestions how to configure them. All in all, it was more about knowledge of details and the choice was left to each person, instead of establishing rules that thou shalt follow.

随着时间的推移，这个问题也将变得不那么重要，建议也会改变，因为越来越多的系统在普通桌面和服务器版本上切换到 ZFS。Ubuntu 已经做到了，其他人肯定会效仿。如果在两三年内 80% 的发行版使用 ZFS 或 btrfs，它们中的大多数将运行虚拟化，因此这是一个有争议的问题。