If the storage all belongs to one machine, yes. If it’s spread across multiple machines with similar setups that share a LAN, then you need to put in a little thought to make sure that there’s only one copy for all machines, but it’s still doable.
In this case, we’re talking millions of machines with different owners, OSs, network security setups, etc. that are only connected across the Internet. The logistics are enough to make a hardened sysadmin blanch.
Deduplication is trivial when applied at the block level, as long as the data is not encrypted, or is encrypted at rest by the storage system.
If the storage all belongs to one machine, yes. If it’s spread across multiple machines with similar setups that share a LAN, then you need to put in a little thought to make sure that there’s only one copy for all machines, but it’s still doable.
In this case, we’re talking millions of machines with different owners, OSs, network security setups, etc. that are only connected across the Internet. The logistics are enough to make a hardened sysadmin blanch.