Demystifying Site Outages
This is the most detailed server outage response I have ever seen. Looks like Etsy understand the importance of accountability and I really appreciate the way they wrote the post. A short version for non-technical and short attention span minded people and a longer version for the nerds who like to know the intricacies of big sites like Etsy.
Good on them for taking the time to write such a detailed report. Shows respect for their users.
Without diving too far into technical details, suffice it to say that in order to support the special characters that some languages require to be complete, we needed to do an upgrade of the database server software.
Oh, hi, I see you're using MySQL. MySQL's 1024 character (oh sorry, not characters, bytes, oopsie!) limit for indices was the bane of my existence. You have a mountain of code and tables using varchar(200) and an index spanning five such columns. Works with latin-1. Then somebody decides this new fangled utf-8 is the new hotness. hahahaha, no database index for you! Now you get to go back and decide if you can shrink said columns or get by with a one column index (mysql assumes 3 bytes, so one 200 char column is already more than half your allotment).
that was interesting, but it felt like two different articles rolled into one (a short non-technical and a long-but-shorter-than-that technical one).
anyway, so alter table modify on a primary key (int to bigint) requires that the index be rebuilt? is that true for all databases? or was the problem something else?
In 2004, when etsy.com was created, we all were pretty aware of the importance of specifying utf-8 when we setup systems.
How could a company, that now has 120+ engineers listed on the about page, get to 80+ databases without fixing this issue?