All file systems have metadata which is good to
keep in memory. Building several 50+TB NAS boxes recently, it isn’t just ZFS either. And it isn’t some sort of linear performance penalties sometimes when you don’t have enough RAM. It can be kernel panics, exponential decay in performance, etc.
That’s a good point - I’ve seen free memory drop every time I’ve built the larger file systems (and not from just the cache), but I never tried to quantify it. And I don’t see any good stats or notes on it.
seems like no one is building these larger systems on boxes small enough for it to matter, or at least google isn’t finding it.
Another way of saying this is that RAM usage doesn't meaningfully scale with storage size in scope of storage systems encountered by actual non theoretical people because the minimum ram available on any system one encounters is sufficient to service the amount of storage that it is possible to use on said system.
The main case of deduplication that I know is hosting many virtual machines with the same OS. If that OS is Windows (explains why VM and not containers), there are dozens of GB of data that is duplicated per VM. It is not 'nobody', even if it is not common and not always worth.
All file systems have metadata which is good to keep in memory. Building several 50+TB NAS boxes recently, it isn’t just ZFS either. And it isn’t some sort of linear performance penalties sometimes when you don’t have enough RAM. It can be kernel panics, exponential decay in performance, etc.
reply