And We're Back
For those of you that didn't notice, this site is hosted and published through Six Apart's TypePad service and was down for a day or two. TypePad suffered a major outage late last week when their new storage hardware failed. Because TypePad creates static pages they were able to get weblogs back up however recent posts went temporarily missing. Users also couldn't make new posts nor could comments or TrackBacks be made. The news and the cries of pain were heard everywhere. (Especially once the service started coming back up.)
This is really unfortunate because the company has been trying really hard to cope with their success. This outage was part of one of those measures they were taking to scale up their system. While user unrest to this outage is understandable, having a brand new (and surely expensive) piece of hardware fail when you are dealing with such hyper growth is a difficult thing to have accounted for. Major system migrations and growth management are difficult for any company. Some may recall eBay suffered a number of outages as it was growing expontentially. So while a system outage will never make me happy, I see this as a side-effect of using with a great online tool -- it can become a victim of its own success.
Incidentially, Om Malik reports several Web 2.0 companies had outages last week that includes Bloglines, Feedster and WordPress.com. Om doesn't mention the extended downtime del.icio.us has been suffering either. TypePad got the most press presumably because so many rely on it as an essential part of online activities.
In his post he later noted that he had a a healthy and civilized debate
with 37 Signals about scale. When the good people of 37 Signals preach to worry about scale until later, I understand and agree to a certain extent, however one must remember that later eventually becomes now if you are successful.
In their response to Om's arguement on scale David Heinemeier Hansson responds for 37 Signals in Don’t scale: 99.999% uptime is for Wal-Mart
where he writes:
Now what if Delicious, Feedster, or Technorati goes down for 30 minutes? How big is the inconvenience of not being able to get to your tagged bookmarks or do yet another ego-search with Feedster or Technorati for 30 minutes? Not that high. The world does not come to an end. Nobody gets fired.
First I'm not comfortable with saying the world does not come to an end when you are charging for a service and have downtime. It just seems like a poor attitude to take towards customer service.
I hope that no one gets fired at Six Apart unless it's the hardware manufacturer whose product failed. (Six Apart has done them a favor by not disclosing who they are or placing much blame on them.) The thing about David's statement is as these services grow in size recovery takes longer. Witness the TypePad outage which took nearly a day to restore and verify backups and return primary services. It took the weekend to restore everything (photos and other recent file uploads) back to normal. So having the scale and redundency to recover for a breakdown does eventually matter. My take: while scale probably should not be a high priority initially it cannot be a non-issue. A bit of thought to how smethings might scale is healthy -- just don't obcess over it. It will focus your energy on all the wrong things
as Heinemeier Hansson warns.
Time will tell if TypePad is toast
, but I am doubtful that the company will not recover from this event. Surely some will leave in disgust and perhaps will indeed be happier elsewhere. TypePad is not the right tool for evertybody. Ulitimately I believe most will stay. Recent events aside, Six Apart's product is still a very good tool for a lot of users.
This brings me to one last thing I noted. It's a point David Heinemeier Hansson touches upon in his post: Criticality. This is just something that occurred to me as I thought this over and its something to watch as these Web 2.0 companies grow and mature. I think the howls of frustration over demonstrate that TypePad has graduated to a higher level of criticality then others. While its of no help to its users, Six Apart may find some soltice that they are that successful as they work through this tough time.
