Yep, most of tracks were already available on “various” sources, but this time they directly scraped the whole Spotify database.
It’s really nice from them to backup Spotify database on a distributed system, and for free ! This ensure Spotify business won’t be endanger in case of critical hardware failure.
300tb is a lot, but its kind of crazy to think this entire company only needs 300tb storage arrays to function. I wonder how they handle things internally. I would imagine at least 1 backup server ready to go in HA. I wonder if they have multiple regions across the country that also serves up the same setup.
Likely cloned Netflix’s “netflix in a box” design, where they drop a large 200TB+ NAS in thousands of different CDN datecenters with their most popular content cached so that total traffic is minimal across the internet at large.
Spotify mainly being music with very little video likely makes this even easier.
IIRC there’s still like 700TB of low popularity music missing, but it is only something like 0.4% of listens.
And they need a more storage overall because they have to set up datecenters around the world - doesn’t make sense to stream tens of millions of connections across the ocean. But that also gives all the backups one would need for “free”.
Is this new? Aren’t most tracks already available in torrents?
Yep, most of tracks were already available on “various” sources, but this time they directly scraped the whole Spotify database.
It’s really nice from them to backup Spotify database on a distributed system, and for free ! This ensure Spotify business won’t be endanger in case of critical hardware failure.
So nice of them to help with Spotify’s off-site backup.
It’s new insofar as this is one big scrape. About 300TB iirc.
300tb is a lot, but its kind of crazy to think this entire company only needs 300tb storage arrays to function. I wonder how they handle things internally. I would imagine at least 1 backup server ready to go in HA. I wonder if they have multiple regions across the country that also serves up the same setup.
Likely cloned Netflix’s “netflix in a box” design, where they drop a large 200TB+ NAS in thousands of different CDN datecenters with their most popular content cached so that total traffic is minimal across the internet at large.
Spotify mainly being music with very little video likely makes this even easier.
IIRC there’s still like 700TB of low popularity music missing, but it is only something like 0.4% of listens.
And they need a more storage overall because they have to set up datecenters around the world - doesn’t make sense to stream tens of millions of connections across the ocean. But that also gives all the backups one would need for “free”.
They need other 300TB to store all the ads.
Afaik 300 TB is just the most popular music and around a third of all tracks. The blog post on anna’s is quite entertaining tho.