HIVE MIND for software developers: I’d like your help – and your stories!
I’m working on an article with a tentative title of “Tales of the Crypt: Horror stories where database performance caused a real problem.” It’s meant to be schadenfreude nostalgia, about your late nights coping with a performance issue (with, hopefully, a happy ending of “…and this is what we did to fix it”).
So, what happened? Tell me about it.
@estherschindler Oh, goodness, I have many of these, from an e-commerce site that needed 10 seconds to load a search page due to missing indexes and not using a proper in-memory store (thus resulting in e-commerce users going *back* to calling the call center, and getting them off it was the whole point!) to a website that randomly hung due to transaction isolation issues (and missing indexes), creating an impressive Heisenbug.
I'm open to describing any of these in more detail if you'd like :)
@lenaoflune I WOULD LOVE TO HEAR MORE! #enthusiasm
@estherschindler Back in the IBM System-36 days (1980s), a colleague in county government IT installed a new voter registration tracking system. On the first day, a user said "I wonder how many voters we have named Elsie". Unfortunately, the first name column was NOT indexed, and the whole system waited while that search churned away.
@Dashdrum Ha! I remember doing a SQL search on a PC in the 80s, and it ran for 5 hours before I finally rebooted the computer.
@Dashdrum A follow-up to an old question...
Was the solution simply telling the other person, "Never do that again?" or did you index the database just in case someone decided to try the same trick?
@estherschindler not sure, but since it was a vendor system, they probably just said don't do that.
@estherschindler - Mongo DB was shiny and new. We had just moved into a new building and the boss wanted am interactive "get to know the agency" game for all the customers at the party showing off our new digs. Our developer who was always chasing the hottest new technology decided to use mongo as the backend data source not realizing it had serious performance limitations when doing things like... Summing up and sorting scores. Halfway through the party he did an emergency refactor to MSSQL.
@estherschindler Don't let me forget to share deets! :D
@lenaoflune Please do -- and now is the time!
If you need an email ID, use my handle here plus gmail.com. (That isn't my primary ID but it'll get to me.)
I'm finally working on this story! So if you have anything to add I'd love to hear it.