2009-11-02 #1

Why did Amazon take so long to deliver SQL in the Cloud?

The storage market has been complexifying, (Is that a word Ed.) over the last few years; I have for a while considered the databases to be just another software abstraction layer between the hardware and the application i.e. completely equivalent to a file system. Also more recently it is clear that the highly scalable solutions builders have moved beyond relational databases.

I was asked my opinion on Cloud Computing and Enterprise Storage late last week, and was pushed to do some reading, John Stanford, had previously pointed me at "The End of an Architectural Era (It's Time for a Complete Rewrite)", by Michael Stonebraker, his team and his collaborators. In this paper, prepared for VLDB 2007, they point at a previous paper, called "One Size Fits All : An Idea whose time has come and gone.". These both explore problems exhibited by the RDBMS market leaders and importantly examine areas where the RDBMS struggles to deliver. Storage is used to support the data involved in many applications classes, from commercial OLTP to Data Warehousing, from Scientific applications such as experimental physics as at CERN to time-series or stream processing solutions, and we're still not sure how best to store XML. SQL has become equivalent to Relational and its just not always the best answer.

In the earlier paper, the author's explore the limitations of SQL in a non-transactional world, such as feed streams, i.e. the streams never close, but once one challenges the relevance of the general purpose relational databases, a whole series of design issues come to the fore, design issues that applications architects in the Enterprise have been happy to leave to database designers & authors; these issues include,

• client server architecture vs. a unique address space
• shared nothing vs. shared everything
• ACID locking vs. pipes
• b-tree indices vs. bitmap vs none
• 3rd Normal Form vs. 4th Normal Form and column based databases
• the Write Ahead Log

Today's application designers and storage consumers are no longer always prepared to accept the compromises buying an RDBMS requires, its about Storage not SQL.