Monday, May 24, 2010

Coders at work...

I finally finished reading Coders At Work, a book that captures conversations with 15 computer-scientists and programmers. In a nut-shell -- if you enjoy programming, read this book there is a lot of wisdom captured within these conversations. The down-side is that almost all of these conversations start sounding similar by about mid-way.

As I was reading it all there was a consistent theme -- most of these people started their adventures in IT during their teenage and slowly picked up a number of different skills over time. They essentially influenced / invented / shaped and evolved many key technologies slowly over time (Javascript, Java, UNIX, Latex to name a few). The real interesting part is that all 15 started their life in IT when it was a relatively young and comparatively simple field. They (like me) could actually learn and appreciate many of the foundations upon which modern day software systems is built .. slowly and gradually.

Ok .. now to my point....

What is striking is that anyone new in IT today (i.e. the current generation of teen-agers) will be unable to gain the same exposure to this field. In essence, specialisation is the price we pay for the ever increasing complexity. Most senior software developers today will have grown with the technology -- learning new skills and techniques slowly as the field matured. But, the next generation of senior developers will just never have had the opportunity to actually get into a simpler field -- they are entering a fairly mature and certainly very complex field.

Personally, I do think that this generation will miss out on many of the joys and excitement of technology -- it is hard to get a thrill out of something that you have grown up with as a normal facet of life. My 6yo son got more excited about an old typewriter (stunned that you manually feed paper into it) than a new iPod.

All this raises some interesting questions (most of which I do not want to answer in this post -- feel free to leave a comment though):

Can we actually manage the complexity and pass on the knowledge to the next generation (that has not built it -- and hence is unlikely to have any emotional/personal attachment). Can this technology then be maintained? There was a recent Slashdot article essentially stating that Linux is having some difficulty attracting younger talent. Is it really a problem?

The current crop of engineers build technology assuming that others will have the same perspective/background and knowledge. Will the educators and the current-generation (under 20yo) put in the 7-10 years of effort to learn the skills and knowledge? Unfortunately as they learn .. the state of art keeps moving making it harder and harder to catch up.

Will abstraction/componentization and API around these services solve the problem? It buys time and there are trade-offs. There is a big problem with layers and layers of abstraction -- they leak. That is, if something breaks and you do not know what/how this abstraction works you may not be able to debug the source of the problem easily. The common counter argument is that few people understand operating systems -- yet we can use them. This argument has some merit -- but, most of the layers and abstractions that we use in software development have not had the same level of attention paid to them.

If you build applications using 20 different third-party components (some open-source, some commercial) -- two of these break intermittently. Are you able to isolate the problem? These confound people with serious experience, how will someone new to the field cope?

The current expectation is that we will weave software using 100s of services and abstractions (some of which may be on the cloud). I know from experience as an academic, we are not even getting though the basics -- let alone educate an under-graduate in 3 years to face this type of situation.

I for one am looking forward to how the field evolves and deals with the issues that I raised here. I am deeply curious and keen to see how far we can push the boundaries of complexity before it starts to cause serious problems.

-- rv

Sunday, May 09, 2010

Was the stock market crash a conspiracy?

Last week the NY stock market fell by 1000 points in about 30 min. only to rebound right back up. A lot of words were written into the blogosphere. Many posts are essentially saying that it is a massive conspiracy, with some powerful group (typically the US Fed. Reserve, Secret society XYZ, Bilderberg group) intentionally causing the market to crash etc. The more silly explanation is that some one mis-typed "B" for "M" causing billions of dollars worth of trades.

My position is that this was more likely caused by systemic complexity and interconnections rather than some all-seeing and all-knowing group causing the event. I think it is just a cascading ripple that just went way too far. But what triggered it? -- The best explanation so far is that the trading computers did not come to a proper halt when the NYSE gave them the "time out" instructions. A synchronisation issue that should not have taken place -- and this specific error will not happen again because they will put in a solution fairly soon.

I also do not buy the fat-finger theory that someone typed "b" instead of "m". This explanation is the most ludicrous of all. The fact that major newspapers even bothered to report this tells you how desperate they are to create "news" and report "opinion" rather than think about the headline for a few minutes. I know a few people that work as traders. In a nut-shell, no single trader is allowed to spend billions of dollars in a single or few transactions (and certainly not in 30 min.). Billion dollar trades do take place, but certainly not by a low level trader making a mistake. They also undertake such large transactions over slightly longer time frames (days -- weeks).

Now to the conspiracy theory. This is also a bit silly. Unless some powerful group has full knowledge of all the rules (and the bugs in the systems) that different traders have setup into their trading systems there is no realistic way to predict how a trigger will play out. We are talking about people that did not foresee a massive mortgage bubble, the technology stock bubble, the banking collapse, the recession, massive fraud at all levels, inability to take over and manage a small country etc. There is no evidence that they have any ability, nor do they have good insight or sufficient information about what is actually going on.

It is also highly likely that they (central banks and or other powerful control groups) do not fully understand the network that is a modern economy. They have some abstract models that may be able to estimate the situation at a very high level. These models are pretty high-level and can certainly say -- "it will be hot in summer and cold in winter" -- but beyond that it is nothing more than luck. Personally, I am convinced that they do not really know what their actions will do -- they are guessing and hoping for the best (read Prof. Steve Keen's work for more on the silly models that are used by economists and bankers).

I also do not buy the position that many of these powerful and rich people are working closely together allied for some common goal (as in the secret society theory). There is no correlation between wealth/power and people starting to collaborate well. In fact, history suggests that wealthy and powerful people are more likely to have problems managing their ego's, they over-estimate their abilities and are tend to compete aggressively.

Reality will be of the following format -- we have a number of these wealthy groups that are loosely allied, constantly changing their alliances, making mistakes and attacking each other's empires within misguided intentions and incomplete information. In a nutshell, they are just not organised enough to prepare, plane and pull a stunt like this.

-- rv