Migrating Messenger storage to optimize performance

  • They say that they brought some new search features to mobile, though I would prefer if they fixed their search on desktop (or in general) first, as it only returns results from recent messages.

    Until sometime in 2016/2017 it was possible to look for specific things at whatever time (e.g. 2011) and now even things from <3y ago don't return anything.

  • Whatever you may think of social networking, Facebook takes huge amount of pains to ensure reliability. They moved all the data from spinning disks to flash, also migrating to a new db. No downtime at all.

    I'm a bit unclear as to how this process resulted in 90% decreased storage use.

  • Facebook has been using RocksDB more and more (now messenger, but also Instragram’s flavor of Cassandra). I wonder if Google has a similar penetration of LevelDB (and we just don’t hear about it because Google’s infrastructure work isn’t open source)

  • During the dual-write phase, what happens if one request succeeds while the other doesn't?

  • What about giving me an easy option to export my messages while you are at it?

  • TL;DR - Moved from HBase to MyRocks engine on MySQL sitting on top of NVMe storage via their Lightning Server [1], which is a JBOF (just a bunch of flash) setup using x16 PCIe.

    [1] https://code.facebook.com/posts/989638804458007/introducing-...

  • Great, now they can serve video ads in Messenger more easily.