opensolaris, virtualbox, GNOME & RAM

Having been inspired at the Open Solaris usergroup in London on Wednesday, I installed an Open Solaris Virtual Box VM on my works laptop. I came across the guest additions bug in VB 3.0.10 and so upgraded to VB3.0.12 and chose the defaults for the VM. This includes setting the RAM at 768 Mb. (I think it checks how much RAM the host has and makes a recommendation based on that, I am running Windows Business 32 bit and can 'see' 3.45 Gb of RAM. ) Anyhow, it seems if the VM has under 1024 Mb of RAM the window manager won't start. Simple enough to solve. See this thread at http://forums.virtualbox.org

Reasons not to 'Port'

I was asked to join in the writing of Migrating to Solaris OS (c) 2003 due to some of the projects I had worked in. We argued that there were four basic techniques available,

• retirement,
• emulation
• source code engineering
• reverse engineering

Retirement is fairly obvious, but its unusual that you can retire the whole of an application. Emulation ranges from black box language emulators such as BASIC or PICK, or more modern and comprehensive interpretive solutions such as a DBMS, web container or even a complete COTS package. This technique relies on the guarantees of the vendor/author, but is an excellent and cost effective method of migrating between operating systems and hence between hardware platforms and architectures. The latter two techniques can be used to migrate the business logic at higher levels of abstraction, between, say two database implementations or 4GLs.

In my experience there are two huge inhibitors from moving.

• cost of labour intensive projects is high, particularly if using 3rd party people
• and the code/business logic is insufficiently worth preserving.

If one starts with a blank piece of paper, the opportunity to increase the business alignment of the application, increase the functionality, and reduce the maintenance cost of the application outweighs the costs of a migration. Businesses choose to rewrite the application. It also often aligns with the personal and team goals of the applications development/maintenance teams. (They stay in work & get new toys and skills).

Over the 12 years I worked at Sun, I was involved in two software migration projects, both of them from one operating system to another using emulation techniques, one of which acted as the basis for Chapter 11 of the book. I also worked on proposing two reverse engineering projects from one database vendor to another, both of which were rejected on cost grounds. (The customers stayed with their then current technology vendors). This inertia will get worse as hardware capital and maintenance costs fall both relatively and absolutley. Potential savings in the cost of platform also decrease. N.B. To most large companies, the marginal cost of licensing a database is zero, and having a support contract with a multi-billion dollar company very attractive. This trends mean that the cost of the personell cost of software increases as a proportion of the total cost and re-writing for maintenance becomes more and more attractive, since the best savings come from long term improvements in software engineering productivity.

Mingle

It seems I havn't documented how to install mingle on top of venus. I wonder if its on one of the home PCs.

Filesharing, Economics & Human Rights

I submitted evidence to the Department for Business, Enterprise and Regulatory Reform, (BRERR) on their proposals to regulate illegal file sharing, it said what you'd expect and when I get time to polish the evidence I'll publish it in White Paper form. It'll be no surprise that I argue

• using Copyright to charge for right to use is economically "inefficient" in the Macro sense and thus not in the public interest
• Downloading digital content should be legal irrespective of the copyright status of the content for personal non-commercial use.
• we don't know how much of the so called "lost" revenue funds distribution and/or foreign super-profits
• copyright duration should be significantly reduced in duration, the concept of abandonment should be introduced as exists in trademark law, if insufficient action is taken to defend rights, then they are deemed to have lapsed
• the Music and Video businesses abuse their negotiating power; by for instance licensing formats so even format conversion to play some content on different devices is a breach of licence. Frankly that's fair use in my book, but they also ignore fair use.
• rights holders should not be able to outsource or sell the right to collect the so-called missing revenue, at all, not to the internet industry, not to the public sector, not even to private sector 'associations'; we call them debt collectors
• the internet industry is a source of innovation and should not be taxed to support the music business
• asking the ISPs to take on additional costs to support the discovery and notification of accused illegal downloaders will increase the cost of internet connection for all customers,
• P2P has legitimate uses, especially in the UK with its legacy copper local loop

Specialisation and the Pricing Mechanism are the engines of wealth. These proposals are designed to protect a business i.e. Music from competition and change. It will have the side effect of inhibiting the pace and direction of change in the IT and Software industries, both of which are even more valuable to the future wealth of the UK economy.

Then I look at the human rights dimension, the government proposals seem to me to breech or diminish citizen's rights

• to freedom of expression
• a fair trial
• access to culture
• right to privacy

These are all rights guaranteed under UK law by the incorporation of the European Charter on Human Rights into UK Law, and repeated in the UN Declaration of Human Rights.

Fundamentally, the record and film industries have tried to sell the world that bootleg, or illegally copied content is theft. No-one agrees with them, except it seems some politicians. Its only illegal because the law makes it so. Its not piracy, its not theft. It doesn't deny anyone anything.

Pareto-economic efficiency suggests that the copyright laws should be relaxed. The copyright laws are a constraint on the wealth creation in the UK and should be reformed and liberalised. Economic policy goals, as opposed to a civil liberties and fairness agenda suggest that copyright laws should be relaxed.

The internet industry is as important a wealth creator as the music & film business, (at least in the UK), why should it be disadvantaged by incurring the costs of the rights owners.

Fundamental rights such as the right to a fair trial, innocent until proved guilty, the right to culture and freedom of expression should not be compromised to prop up any business.

I have put some of my notes on Intellectual Property Law, which has disqus comments enabled.

Why did Amazon take so long to deliver SQL in the Cloud?

The storage market has been complexifying, (Is that a word Ed.) over the last few years; I have for a while considered the databases to be just another software abstraction layer between the hardware and the application i.e. completely equivalent to a file system. Also more recently it is clear that the highly scalable solutions builders have moved beyond relational databases.

I was asked my opinion on Cloud Computing and Enterprise Storage late last week, and was pushed to do some reading, John Stanford, had previously pointed me at "The End of an Architectural Era (It's Time for a Complete Rewrite)", by Michael Stonebraker, his team and his collaborators. In this paper, prepared for VLDB 2007, they point at a previous paper, called "One Size Fits All : An Idea whose time has come and gone.". These both explore problems exhibited by the RDBMS market leaders and importantly examine areas where the RDBMS struggles to deliver. Storage is used to support the data involved in many applications classes, from commercial OLTP to Data Warehousing, from Scientific applications such as experimental physics as at CERN to time-series or stream processing solutions, and we're still not sure how best to store XML. SQL has become equivalent to Relational and its just not always the best answer.

In the earlier paper, the author's explore the limitations of SQL in a non-transactional world, such as feed streams, i.e. the streams never close, but once one challenges the relevance of the general purpose relational databases, a whole series of design issues come to the fore, design issues that applications architects in the Enterprise have been happy to leave to database designers & authors; these issues include,

• client server architecture vs. a unique address space
• shared nothing vs. shared everything
• ACID locking vs. pipes
• b-tree indices vs. bitmap vs none
• 3rd Normal Form vs. 4th Normal Form and column based databases
• the Write Ahead Log

Today's application designers and storage consumers are no longer always prepared to accept the compromises buying an RDBMS requires, its about Storage not SQL.

Oracle & Sun & the European Single Market

A lot of people have been busy commenting on the EU's investigation into the competitive dynamics of Oracle's proposed acquisition of Sun, so I thought I'd join in.

I no longer work for any of the parties involved, nor to my knowledge do I own any shares in any of the parties. I am a citizen of a member state of the EU. I have recently left Sun Microsystems where I was the Chief Technologist in their european sales and support organisation and where I represented Sun on the NESSI steering committee. In this role I was one of the contributors to NESSI position paper : European Software Strategy. I was not privy to any private 'material' information while employed at Sun.

The database market, even if narrowly defined, is characterised by a number of players, some of them very large and in the case of IBM & Microsoft, the subjects themselves of previous monopoly investigations. The EU has every right to investigate the proposed acquisition to see if it directly leads to anti-competitive behaviour. Anyone who thought they wouldn’t was being naive. But since everyone seems so free with their advice to Ms Kroes, I shall add mine and state that I hope that the investigation and decisions restrict themselves exclusively to "single market" and competitive/anti-competitive issues. I also hope, that at this time in particular, the Commission does not take the opportunity to act in a protectionist manner on international trade issues.

I tend to agree with those who argue that even the database market is complex and diverse and that Oracle's leadership is under competitive pressure from established players such as Microsoft and IBM, from open source start-ups including at least two MySQL forks, new appliance vendors such as Greenplum & Netezza, new entrants to the data management market such as the no-SQL players, and from cloud providers such as Amazon, Google & Cloudera. It should be recognised that the most devastating and threatening competition will come from where you least expect it, best articulated in the Monty Python, Spanish Inquisition sketch,

Cardinal Ximenez : "Nobody expects the Spanish Inquisition, our main weapon is surprise..."

In his paper, "The End of an Architectural Era (It’s Time for a Complete Rewrite)." Prof Michael Stonebraker and his collaborators argue that the commoditisation of the relational database reduces its efficiency in doing what it originally did best, i.e. OLTP and enterprise solutions and requires rethinking and re-writing. They argue that they

....predicted the end of "one size fits all" as a commercial relational DBMS paradigm.

It's another indicator of the competitive advantage new entrants have in the market, and the legacy inertia, both inside itself, and amongst its customers that Oracle has to deal with to keep up.

So the market is large, there seems to be vibrant competition, new entrants are disrupting it, and the established player's legacy is inhibiting its ability to compete. The new players also show the difficulty in defining a market’s boundaries, is it about relational databases? Databases, SQL databases, or appliances? I think that Stephen O'Grady articulates the market dynamic well in his Blog, Oracle, MySQL and the EU: The Q&A, and also captures some of the best & worst contributions to the debate so far. While the discussions around the ownership of copyright, trademark and licence are fascinating they don't seem too relevant to anti-competition regulation, as far as I can see. Oracle's past track record, current public statements and future best interests both economic and in reputation don't suggest that they'll trash MySQL to reinforce the market leading position of Oracle RDBMS, even if one accepts that the market in question is an SQL/Relational Database market. Neither does Oracle’s record suggest that they'll starve it of resources to allow it to die. (It would probably help their case if Oracle were to state this in writing to the Commission.)

Anyway the open source rights are today ensuring that the MySQL code survives and thrives, and yesterday, Amazon announced that they were offering MySQL machine images, I assume they’ve licensed it from Sun in order to get the trademark right to use, but it’s another proof point that the cat is out of the bag.

The Commission has more research resources than I do but I think they should come to the same conclusion I do; I just hope they keep their eyes on the public interest, total market issues and competition policy. By buying Sun, there is no doubt that Oracle ups its game in an ability to compete with IBM, HP and Microsoft, in the broader IT market customers get more choice; at the end of the day, this may not be about the database market, it becomes about the IT supply market as a whole.

You can comment on this at http://friendfeed.com/davelevy, and I shall look at enabling disqus comments for this post as it seems likely to attract more interest than my usual ramblings.

The short url for this article is http://is.gd/4FNbz

Where are the new developers coming from?

My final note from the Water's Power:09 Conference; Robert Johnson, a development manager at one of the London based banks stated that of the people he's looked at in recruitment,

Many… developers don't have a computer science background…

which makes it hard for them to write code for both distributed computing platforms and multi-threaded CPU systems.

It seems this is a reflection of the trends I have written about at on my old sun blog, tagged 'university' and more importantly at this site, in an article called British Higher Education. Given a choice between studying something easy or something hard, now that they have to pay a lot, students and their families choose the easy route. A further cause is the dead hands on the school IT curriculum design and the gestation period to make changes.

Banks have a lot of computers

Another correspondent at Waters Power:09 last week, a bank employee, suggested that 3rd party cluster vendors are only interesting when they can technically differentiate because, regulation will seriously inhibit buying cloud compute and storage and most of the tier one banks are bigger than what the cloud providers make available.

Data Centre Economies of Scale

At the Waters:Power09, last week, Bob Giffords argued there are three 'gravitational' forces leading to the mega data centres and cloud computing.

• There's too much data to move, it needs to stay where its created.
• Intra system & total latency is still a problem, and hence systems are best co-located with the data.
• He argues that energy management is a gravitational issue.

It remains true today that moving a terrabyte of data, significant distances, by tape and courier is quicker than using a network. This fact shows how much work needs to be done to get data into a system or cluster. Memory to CPU transfer speeds become seemingly slow, when looking at automated trading solutions. It's also interesting to observe that the ratio of storage to systems in HPC grids is growing which itself an indicator of how new architectures enable enormous data volumes to be trawled, they're a new class of data rich analytics applications.

I am unsure about the gravitational nature of energy management, unless he's refering to things like Iceland's selling of its clean energy as a locational advantage, or Denmark's decision to localise power generation in order to avoid the transmission loss. Funny how even Data Centre managers are getting to grips with the power to cool/power to run issues, but everyone's ignoring how much is dissipated into the air as it travels from the generating plant. Certainly, its potentially an anti-gravitational effect in the electricity generating network, unless power genration yields massive economies of scale.

Some insights into managing the cloud

Dave Cliff, Professor of Computer Sci at Bristol spoke to Waters Power:09 in Canary Wharf yesterday. It is clear from many sources that IT is changing and he examined some of these changes. He woke me up by quoting Carlota Perez http://www.carlotaperez.org/ who argues that there a five transformational changes since the industrial revolution, Steam, Railways, Electricity, internal combustion and IT. She also argues that the adoption and maturity cycles are similar, and that "money's out of IT now". Her book is called "Technological Revolutions and Finance Capital:The Dynamics of Bubbles and Golden Ages", which gives on an idea of where she's coming from. Cliff also pointed his audiance at Nick Carr's "The Big Switch", another pundit that argues that IT is done!

Despite the maturation of IT, the new mega-systems, grids, or clusters are more complex than ever before, and this is why computer based automation is so important. Prof Cliff wrote one of the first trading algorithms and used it to trade on several banks trading floors. The reason he told us about this, is that some of his and other colleagues research shows that a trading solution works well within the distributed platform scheduling problem. He is quite funny, unless you're a socialist, in that he compares it to free market vs command economy solutions. A classic rigid programmatic scheduler is seen as the IT equivalent of the Stalinist 5 year plan. I think the original paper where these ideas were explored is http://www.hpl.hp.com/techreports/98/HPL-98-17.html © 1997 (My notes say this started in 1988, was that the IBM stuff.) The fact is that the 'Price Mechanism' is the largest distributed decision making process in the world. Why not apply it to the scheduling problem? http://tycoon.hpl.hp.com/ is an open source MBC implementation which I need to explore. Early grids were simple to schedule, but as we begin to build clusters of heterogeneous nodes and multiple operating system builds and run-time containers the management problem becomes more and more complex. The original ZIP had eight controls, because that's all that the user could cope with, but once you hand over control to the computers then you can deal with many more parameters which is why its been developed, and developed with computer assistance. This is also true of the next generation of computers and schedulers. I have looked, in unpublished work, at the possibility of a pub/sub model where systems advertised themselves as available to perform certain types of work which would be defined by attribute collections, with the simple attributes being things like high memory, persistent storage, multi-thread etc and jobs ask for a container meeting the various attributes. I also envisaged jobs estimating the system capability it required. This solves a slightly different problem, but it may be part of the same thing, with the various attribute collections offering a foreign exchange dimension to market based controls.

Dave also pointed at http://www.marketbasedcontrols.com, the site of an ESPRC funded project, and also at http://www.lscits.org, the home of the UK's national research and training initiative in the science and engineering of Large-Scale Complex IT Systems (LSCITS), of which he is the Director. He also offered some of his people to work beside the delegates on joint research projects.

All in all a very thought provoking session. Thanks Dave.

Theme exported