Science Break: Occam’s Razor

Mathematics is the language of science, and a language in which I am admittedly only semi-literate. In spite of my lack of mathematical sophistication I am keenly aware that many of the behind-the-scenes mathematical techniques, principles and assumptions we use today have their roots in human thoughts which significantly predate modern science—a fact which I find utterly fascinating!

Geophysics is a field particularly blessed with underdetermined problems, meaning that our geophysical data can often be explained by an infinite number of models, or at the very least it is uncertain whether our methods have converged on the best solution. This kind of situation has resulted in the use of techniques to discriminate between alternate solutions, and many of these techniques are based on rules of thumb from hundreds or thousands of years ago.

Occam’s Razor is one such rule of thumb, although it would be better described as a general maxim. William of Ockham was an English logician and friar of the Franciscan Order who lived from 1285 to 1349 AD. The maxim attributed to him – Entia non sunt multiplicanda praeter necessitatem, or in English, “Entities should not be multiplied unnecessarily” – was probably not actually written by him, as it has never been found in any of his works. However, he did make several similar statements, and he was a widely read and respected thinker in his time. The gist of his maxim was definitely not new when he was alive, and variations of it are found as far back as Aristotle’s writings in the 4th century BC. Indeed, humans seem to intuitively prefer simple or “elegant” explanations to problems over complex ones (although all of us have met people who invariably prefer the reverse!) The spirit of Occam’s Razor is found in our modern saying, “Keep It Simple, Stupid”, or KISS.

The essence of Ockham’s Razor is that if you have more than one theory, and each of the theories fits the observations equally well, then the theory that makes the fewest assumptions and postulates the fewest entities is likely to be the best one. Occam wrote, Frustra fit per plura quod potest fieri per pauciera – “It is futile to do with more things that which can be done with fewer” – hence the image of shaving away unnecessary assumptions and entities with a razor.

A common misinterpretation is that a simpler solution is always better, but the maxim very specifically implies that all other things being equal, the simplest solution is the best. So if a more complicated theory fits the observations better than a simpler one, then the more complicated one is obviously the better. Albert Einstein said it well with, “The supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.” Others have paraphrased this as, “Theories should be as simple as possible, but no simpler,” or, “Make it simple, not simpler.”

Einstein’s career can be viewed as a single minded pursuit of simplicity in his theories defining the fundamental laws of nature. He passionately believed that the simpler a theory was, the more likely it was correct, and furthermore that the innate beauty of nature was a reflection of simplicity. He once said, “Nature is the realization of the simplest conceivable mathematical ideas,” and also liked to repeat Isaac Newton’s words, “Nature is pleased with simplicity.” It is interesting that Einstein’s greatest work occurred earlier in his career, when the beacon of simplicity, essentially Occam’s Razor, was burning brightest for him. Later in his career when he got bogged down searching for a unified field theory, he validated or judged his theories more on the grounds of whether they could be expressed with mathematical formulae, rather than whether they adhered to Occam’s maxim. But inferring that his inability to define a unified field theory was a result of a deviation from his earlier pursuit of simplicity in nature is probably unfair. It is more likely that the observations related to quanta are so innately complex, and perhaps beyond the comprehension of the human mind, that there is no “simple” theory to explain them.

Occam’s Razor and similar principles go back to the dawn of human thought, and fall into the field of Philosophy more than anything. Many of the theories and principles out there which are nominally attributed to modern science actually have their roots in philosophy—who would have guessed that some of the thoughts and musings of mediaeval monks would find their way into the workings of modern science? For example, the digital logic found in computers goes way back to true/false logic employed by philosophers. The principles of parsimony and simplicity in the context of evaluating competing theories were originally developed to help answer or understand philosophical questions such as, “Is there are a God?” or, “Do we exist?” Nowadays we find ourselves using them to help us converge on the best seismic images!

However, a word of warning – delving into philosophical thought is not for the fainthearted! Just because some of the concepts and principles we scientists use have their origins in philosophy, does not necessarily mean we can open up a book on philosophy, or even more so a mediaeval philosophical work, and understand it or glean anything useful f rom it. It’s a confusing world of obscure and esoteric theories, and to us in applied science, philosophers can seem to have far too much intelligence, far too much time, and far too much interest in matters that are completely irrelevant to ordinary people. But perhaps it is the role of philosophers to apply their specialized form of thinking to matters that transcend the day to day concerns of regular folk, digest those matters, and then deliver their thoughts in the form of maxims, principles, axioms and so on that can in fact be applied to actual problems, much like we employ variations of Occam’s Razor in scientific algorithms and theories. And certainly there will always exist an area of overlap between science and philosophy.

At any rate, there are many Occam Razor-like applications in science. Within seismic processing several methods are used to help algorithms find the best solutions to underdetermined problems. Some examples of these are:

Parsimony – this is essentially the implementation of Occam’s Razor; that is, explaining results with the theory that is both simplest and has the best predictive power.

Bayesian inference – this is an approach that involves expressing all forms of uncertainty associated with a particular problem, including our level of conviction about a priori model information, in terms of probability; although strictly speaking this is not really related to Occam’s Razor, in practice we prefer to construct our subjective a priori model following Occam’s maxim of simplicity. A good close-tohome example of this is Ulrych and Sacchi’s sparse Radon transform which uses Bayesian inference in conjunction with a sparse model. The Bayes theorem was named after Thomas Bayes (1702-1761), who was a British mathematician, statistician and religious leader.

Minimum norm solution – of all the solutions that explain the data, the minimum norm one is the “smallest” one, and is unique. This is a form of solution that was definitely formulated following the principles of simplicity.

Many of us equate logical thinking with the scientific method, but in fact humans have employed and honed logic via the study of philosophy for thousands of years, and many of these ancient elements now underpin our modern scientific theories and methods.