Hi all, so I have about 6 nvme drives in my server, I use them in pairs for download cache, appdata, and vm’s. Would there be a benefit or would it act worse if I combined all 6 drives into one ZFS pool to have the bitrot protection on my VM drives, and the increased storage space.
Hmm, I was thinking if they had 3x2NVMe drives, then they could have a total read/write bandwidth of 900MB/s (assuming a 300MB/s throughput, and assuming the 3 different raids/pools each are completely independent).
Where as if they make it one ZFS pool, it would be 300MB/s because all 3 “apps”, would be accessing the same “pool”.
Though, this may only be relevant to HDDs not NVMe’s as I havn’t had the privilege to have NVMe’s in a zfs/raid/etc.
If he had 6 x NVME drives and his controller and MB supported maximizing bandwidth it, it should be capable of 300MB/sec x 6 (1.8TB/sec), parity calculations nonwithstanding because the data is striped across all drives in the pool as it reads and writes to/from all drives in the pool at the same time.
For example, if a big 1TB file is split across 6 physical devices in ZFS with striping, the system will read pieces of that 1TB from all NVME drives in the pool at the same time, not from one drive at a time.
Transfer rate is determined by the physical devices (and the hardware attached), not a pool.
Here’s a little video about it - this isn’t using ZFS, but the idea about increasing transfer rates by raiding your NVMEs together is accurate. He’s getting speeds of over 21TB/sec for reads, and almost 17TB/sec for writes (though his example doesn’t have parity protection, which will boost his speeds)
https://www.youtube.com/watch?v=DXT1IXFIFAI
Oh wow! That’s great info thanks a heap!