2009-10-16 #1

Some insights into managing the cloud

Dave Cliff, Professor of Computer Sci at Bristol spoke to Waters Power:09 in Canary Wharf yesterday. It is clear from many sources that IT is changing and he examined some of these changes. He woke me up by quoting Carlota Perez http://www.carlotaperez.org/ who argues that there a five transformational changes since the industrial revolution, Steam, Railways, Electricity, internal combustion and IT. She also argues that the adoption and maturity cycles are similar, and that "money's out of IT now". Her book is called "Technological Revolutions and Finance Capital:The Dynamics of Bubbles and Golden Ages", which gives on an idea of where she's coming from. Cliff also pointed his audiance at Nick Carr's "The Big Switch", another pundit that argues that IT is done!

Despite the maturation of IT, the new mega-systems, grids, or clusters are more complex than ever before, and this is why computer based automation is so important. Prof Cliff wrote one of the first trading algorithms and used it to trade on several banks trading floors. The reason he told us about this, is that some of his and other colleagues research shows that a trading solution works well within the distributed platform scheduling problem. He is quite funny, unless you're a socialist, in that he compares it to free market vs command economy solutions. A classic rigid programmatic scheduler is seen as the IT equivalent of the Stalinist 5 year plan. I think the original paper where these ideas were explored is http://www.hpl.hp.com/techreports/98/HPL-98-17.html © 1997 (My notes say this started in 1988, was that the IBM stuff.) The fact is that the 'Price Mechanism' is the largest distributed decision making process in the world. Why not apply it to the scheduling problem? http://tycoon.hpl.hp.com/ is an open source MBC implementation which I need to explore. Early grids were simple to schedule, but as we begin to build clusters of heterogeneous nodes and multiple operating system builds and run-time containers the management problem becomes more and more complex. The original ZIP had eight controls, because that's all that the user could cope with, but once you hand over control to the computers then you can deal with many more parameters which is why its been developed, and developed with computer assistance. This is also true of the next generation of computers and schedulers. I have looked, in unpublished work, at the possibility of a pub/sub model where systems advertised themselves as available to perform certain types of work which would be defined by attribute collections, with the simple attributes being things like high memory, persistent storage, multi-thread etc and jobs ask for a container meeting the various attributes. I also envisaged jobs estimating the system capability it required. This solves a slightly different problem, but it may be part of the same thing, with the various attribute collections offering a foreign exchange dimension to market based controls.

Dave also pointed at http://www.marketbasedcontrols.com, the site of an ESPRC funded project, and also at http://www.lscits.org, the home of the UK's national research and training initiative in the science and engineering of Large-Scale Complex IT Systems (LSCITS), of which he is the Director. He also offered some of his people to work beside the delegates on joint research projects.

All in all a very thought provoking session. Thanks Dave.