“Small software” to the Rescue

Over the years I have heard Software Development described any number of ways. The one that fits best is to me the statement that it is “a celebration of the brute force machismo of mutant hero developers”. I don’t know about the hero part but I am certainly one of the mutants.

Ask anyone who knows me, the analogy works on so many levels.

Introduction

When the RECORDER asked me to contribute an article to this section, it didn’t take me long to figure out what I want to write about. I want to write about the problems associated with Drug Resistant Software. Admittedly, that is an obtuse title but as you will see, it does fit as a convenient moniker for what I consider to be the death of software development and its lack of relevance to the geoscience community as a whole.

I believe that software in the geoscience industry has stagnated; that it does not meet the needs of the community and that with our current business model, it never can. I want to talk about why that is from both a technical and business prospective and I want to talk about what we can do about it and briefly, why we will have to.

Given the complexity of the task, I had better get started and so you might give credence to my opinions, I’ll begin with my history as a mutant. That word comes up a lot in my life.

42 Years and Counting

The reason why I think you should listen to my opinions is because I have been writing software for a very long time and I understand the process at a very deep technical and personal level. I began writing in the summer of 1973 while working in the lab of the late Dr. Ross Hallett, a physicist at the University of Guelph. Dr. Hallett asked me if I could translate some code written in a now defunct language called FOCAL into an emerging language called FORTRAN. It took me a few days to catch on but once I did, I was hooked.

I have written ever since and from those early days of punching code onto paper tape, I have progressed through pretty much every stage of hardware and software development. I began by sitting at a teletype writing for a PDP 6 with less memory than a hand calculator. I now write standing in front of three 30” monitors attached to computer with two multi-teraflop gpu’s. And what I write now taxes them to their limits.

It’s been quite the ride. Along the journey I have written on the happiest days of my life, the days when my children arrived and on the saddest, the day when my father left. I have written in basements and office towers. I have written on buses, planes and once, I wrote on an HP 45 hand calculator in the back of a 1970 Datsun. And for two weeks, in January of 1990, I wrote Markov Chain Log Blocking code on a 286 portable in the lounge of the burn unit at the Foothills Hospital in Calgary.

If you think I have mellowed over the years, think again. I spent yesterday afternoon writing in a food court. When I got up to leave, my computer bag was gone so I went to security to report it. It turns out that they took it thinking it abandoned. When I asked them why they didn’t just ask if it was mine they said that they were afraid to, I looked that intense.

I have written code snippets, subroutines, programs, applications and entire systems. And I always write alone and in total silence. I have been doing it that way for over 40 years and in those years hardly a week has gone by when I did not write something. My family calls it an addiction. The Doctors who know my history call it a coping strategy. Perhaps, but for me I write for one simple reason; it is how I create.

My passion is to sit (or nowadays stand), picture something in my mind and then make it real. Devoid of classical artistic talents, writing software is the only way I can do it. And I love it. It gives me a sense of self-worth and something to hold onto when the circumstances of life make the grip so tenuous.

I am, in short, a mutant. The analogy works on way too many levels.

Bruce Atkin and the Definition of “Small Software”

Having established my credentials as an algorithmic addict, I’ll launch into my main subject, which is why software development in this industry has effectively stagnated and what we can do to break it loose. I’ll toss out the opening line that big companies can’t write small software and just let it sit there for a while. Now, I’ll go onto Bruce.

Bruce Atkin was an old friend who is no longer with us. He was and probably still is remembered by everyone who has ever written software in this industry. Bruce was perhaps the most imaginative and progressive geoscientist I have ever known and his suggestions and list of improvements have become the stuff of legend. It is a fact that he never saw an upgrade he could not improve.

I remember one example from the mid-90s. I walked over to his office to install the latest version of my Outrider Structural Modelling software then I walked the eight blocks back to my office. By the time I got there, Bruce had already sent me a five page fax listing all the things he thought I should add/improve/invent. I went over the entire list in detail and had to admit that each and every point he made was valid. One in particular, a novel way of showing where well tops would land on the seismic during an interpretation, struck me as brilliant. I didn’t implement it though because it wasn’t something that fit into my development plans. I didn’t write it and I don’t think anyone else did either and so the concept was lost to time.

Every suggestion Bruce made was good and in a perfect world all of them would have been implemented. But this world is far from perfect and in the end, none of them were and that is to me a tragedy. In his career, Bruce came up with hundreds or perhaps thousands of valid, should be done sometime ideas. Very few, if any of them however, were ever implemented. But the fault was never Bruce’s. The fault was and still is not in our science but in our business model.

All of Bruce’s ideas came under the heading of “small software”, ideas that although technically valid lacked a business case to attract the attention of the companies tasked to develop them.

Drug Resistant Bacteria

Small software is my term for software that we need, sometimes desperately, but can’t develop because our business model for developing software is broken. This is not just our problem and it doesn’t just apply to software. It is ubiquitous because over time almost every industry becomes dominated by huge public companies that are slaves to their quarterly profit statements. Those statements and the need to increase the company’s share price for their investors are the death of small ideas.

To bring this into perspective, I will use an analogy to something we have all heard about and should be more concerned about: drug resistant bacteria.

I read an article some while back on the subject. Its main point was that these so called super bugs were nothing of the sort. There was nothing super about them, they were simply bugs that had developed a resistance to our existing drugs but that would be susceptible to new drugs, if we developed them. The problem in developing new drugs they said was not one of science but of business model.

It takes billions of dollars to research, test and get approval for any new drug and so the powers that decide which drugs to research have to keep a sharp eye on the potential financial return. So where do they put their research dollars? If you were sitting on the board of a Pharmaceutical company would you invest in developing a new antibiotic that may be used by 100,000 people per year for a course of about two weeks or would you put it into developing a new drug for diabetics that would be used by hundreds of millions every day of their life?

The answer is too obvious to state. The pharmaceutical industry is dominated by huge multi-national companies that need billions out of each new drug just to break even. As a result, no one is researching new antibiotics because even if they eventually paid back the research and development costs, the ongoing revenue from them would barely ripple the company’s bottom line. Remember that next time you go to Hospital.

New antibiotics are “small drugs” that are needed to solve a problem (drug resistant bugs) that the pharmaceutical business model leaves behind. But returning now to our own field, what is the geoscience equivalent of those antibiotics? Where and how do we write the small software that we need to solve problems (Bruce’s ideas and probably yours as well) that our existing software business model leaves behind?

Before we get to answering that I want to talk in more detail about why big companies can’t write small software.

Churchill Rears his Head

Just as new antibiotics are needed but not profitable, most of the software that we will need in the coming years will not be profitable enough to develop. But does it have to be that way? Software in the geosciences is dominated by giant multi-national companies. We all know that and we all understand that they have to be profitable, fair enough. And we do need them because whereas most ideas are small ideas there are also the big ideas that need the resources of the giants. But after they have made their profit with big ideas, why can’t they invest a little of it in small ideas as well?

Why can’t they develop small software? After all, money is money and making money with software looks so easy to the uninitiated. On the surface, software always looks like a cash cow. Write a program once and sell it a million times. Unlike a manufacturing industry, there are virtually no material costs and you are never held up waiting for suppliers. And distribution is easy, or so it seems. Put it up on the web, give everyone a link to it and watch as the dollars roll in.

But it never quite works out that way for a number of reasons one of which is that writing software is a creative process and it requires people who can create. This is where I go into my rant, one 40 years in the making. I’ll begin it with Churchill. Every rant should begin with Churchill.

I love Churchill. He came up with such great lines, including the one that two people can keep a secret if one of them is dead. So what has that to do with anything? I’ll paraphrase and say that two people can write a program if one of them is dead. That should give you some idea of the general direction ahead.

There is an old adage in software that 10% of the developers (the mutants) write 90% of the code. In my experience it is more like 5/95. The sad fact is that most of the people writing software these days would contribute more to society by driving a garbage truck. They may have knowledge but they lack the fundamental logical thought patterns necessary to be productive.

Knowledge, however deep, is no substitute for talent. An ability to feel, follow and develop logical patterns is as essential to a software developer as color sense is to an artist. I could spend decades studying drawing and painting techniques but in the end I would still not be able to draw stickmen. I can’t draw, I can’t paint, I can’t write fiction and I can’t play music. I have tried but I don’t have the fundamental talents and no amount of training would elevate me beyond the mundane.

In the same vein, most developers study patterns and languages but in the end, they can’t stitch them together into a cohesive whole. They lack the fundamental logical talents needed to be truly productive. So most software is still written almost entirely by the few mutants who, completely devoid of social skills, invent things like Agile Development (Wikimedia Foundation Inc., 2015a) (not to be confused with Agile Geoscience) to give the incompetent the illusion they are contributing.

We try to keep them quiet and out of our way but in a big company environment that is almost impossible. Big companies love defined processes. They love to predict when something will be finished, what it will do and how it will do it. They want stages and milestones and tickets and meetings and meetings and meetings...

And they get them. They enforce standards and rigid organization and team structures on what is fundamentally an individually creative occupation. In the process they either drive the mutants away or so hamper them with sprint meetings and scrums and code reviews that any individual brilliance is mixed into the collective whole. Big companies... Welcome to software’s equivalent of the salt mines, comrade!

As a result, all big companies can produce (always late, always buggy, never what people really want) is “big software” that costs far too much and does far too little. This is the tragedy of Bruce’s inspirations. Each and every idea he came up with was valid and should have been developed but very few ever were because they were all “small ideas”, ideas that were needed but that wouldn’t generate enough revenue to tweak a big company’s interest.

His ideas never failed on science, they only failed because they lacked a business case. It was that way. It still is that way. But it doesn’t have to stay that way!

Even Mutants have Feelings

So what can you and I do about this?

Every few weeks I meet with my Doctor and a “mental health professional”. I wrote about the reasons why with Oliver Kuhn (Khun and Lynch, 2014) a few months ago.

In one session, they asked me to list all of the emotions that I had. I got to about eight when they stopped me. “Wrong”, they said. “You have two. You have up and you have down and that is all anyone has ever had”.

There is up and there is down. To borrow from Democritus, everything else is merely opinion. Who would have guessed?

Everything can be either up or down and software development is no exception. When I began writing in 1973 it was in an up phase. It was new and exciting and full of potential. And what is more, it was easy to learn and everyone wanted in on it. Everybody wanted to write and express themselves.

What it has evolved into is a pity. From being in a youthful and manic up phase software development has become dull, staid, and full of processes, overly complicated and nobody who is not trained in it wants any part of it. And those who are trained in it only stay in it because it pays the bills. It is now in a down phase and in short, for anyone but the mutants, it is just not fun anymore.

But if Bruce had his way, it would be. Getting back now to his small software. Small software is fun and it is always up. Small software are the ideas you come up with when you are looking at your data and think “what if”. Small software are ideas you may have read about from some other field that you think might help here. Small software is anything you want to try but can’t either because your existing software won’t do it or because it is too cumbersome.

Small software is creative and imaginative and if you get it working yourself you will discover some of the joy we all felt back in the day when software was young and we could accomplish something meaningful in just a few hours.

The Mutant Apocalypse

I set out here with a goal. It was to prove that you can’t rely on big companies and big software to solve your day to day problems and to encourage you to learn a simple, flexible, open language like Python so that you can do it yourself. And in my opinion, we are all going to have to do that because coming down the road is a culture shock that nobody is expecting.

Over the past few months, I have been slowly and carefully introducing a new technology into the industry. The technology gives the ability to visualize seismic in its natural state, as an analog acoustic wavefield. I have been working on it for a long time, both the technology and the theories behind it. What it all adds up to is that it gives you a complete visual impression of your seismic and that impression is stunning in its complexity.

Seismic is nothing like the dull, staid and dispassionate data we have all become familiar with. It is vibrant and exciting but it is also confusing and confounding. You cannot look at seismic in its natural state without triggering a cascade of questions and lines of investigation that desperately need answering. And if there has been a point behind all this it is that if we have to wait for our conventional research and development models to answer them, we will all be long gone before they do.

If we are going to find the answers to the questions seismic is about to ask us, we are going to have to do it ourselves. We need a new paradigm for how we do research and how we do development. Both have to come out of backrooms and onto the desks of individual geoscientists. And doing that is not an option because seismic has enormous untapped potential. But to reach that potential we will have to solve problems that we barely recognize exist and we cannot wait for the ponderous footsteps of the giants to trample over them.

I’ll leave you with this. My research into visualizing the seismic wavefield is called “The Visual Wavefield Project”. When I sat down to design the software behind it, Bruce and his small software was very much on my mind. I designed the software specifically so that it would interact, in real time, with languages such as Python or Matlab®. I wanted to write a program that beyond visualizing the wavefield, offered everyone the possibility to experiment with it. In essence, I wanted to give you the opportunity to explore your inner mutant.

I hope you do. I hope you learn how to use Python or other languages like it and use them to write your own small software. Becoming a mutant is a lot of fun because mutants make their own rules, find ways to answer their own questions and don’t wait on the timing of others.

And finally, if that is not enough, become a mutant because of why I became one. Obsessions, addictions and coping strategies aside, I write every day and I have written almost every day of my career and I do it for one simple reason: I write because mutants are never down.