Open Collaboration: Hackathons and Tomorrow’s Subsurface Software

There’s a quiet revolution happening in subsurface science and engineering software. The last one happened about 12 years ago when powerful Linux and Windows PCs dropped under $20k and displaced Sun workstations, lowering the bar to fast, attractive 3D visualization. But, looking back, it happened in slow motion and at the end of it nothing had really changed, except no-one builds viz rooms any more.

The next revolution will be different. It will be profound. And it will not come from a big software company. It will come from nowhere, and the products will live on the web.

And there’s a good chance that the prototypes will have been built in a weekend.

The concept

In May 2013 we hosted a conversation at the GeoConvention and asked ‘What are the unsolved problems in subsurface geoscience?’ (Hall & Bianco 2013). The top three problems we unearthed were:

1. Less secrecy, more sharing
2. Improving seismic resolution
3. Acceptance of error – live with it

At the end of this “unsession”, we challenged the community to step up and start attacking these problems, and the others on the list. In this vein, we held a second unsession in 2014 to explore ‘Less secrecy, more sharing’ further, but we also wanted to go beyond conversation, to action. So we started hosting hackathons.

Hackathons are for hackers. Not the black-hat hacker variety that tries to break into corporate databases to steal credit cards. Not even the white-hat kind that tries to break into corporate databases to help improve their security. Hackathons are for the original kind of hacker – the kind born from MIT’s Tech Model Railway Club. Paraphrasing Wikipedia (2015), which in turn draws from über-hacker Richard Stallman’s writings:

A hacker is one who enjoys the intellectual challenge of creatively overcoming and circumventing limitations of [...] systems and who tries to extend their capabilities. [... The] defining characteristic of a hacker is not the activities performed themselves (e.g. programming), but the manner in which they are done: Hacking entails some form of excellence, for example exploring the limits of what is possible, thereby doing something exciting and meaningful.

That sounds like something we need in subsurface geoscience!

The events

To date, we have run four events, with four themes. We’ve run them all on the weekend right before major geoscience conferences:

• Error and uncertainty: September 2013 (before SEG), at START Houston
• Well logs: May 2014 (before AAPG), at START Houston
• Resolution: October 2014 (before SEG), at Thrive Coworking Denver
• Web connected: May 2015 (before GeoConvention), at Calgary's Convention Centre

In all, about 40 individuals have taken part in these events. Ben Bougher and I took part in all of them; Evan Bianco and Jacob Foshee have hacked at three each. Perhaps surprisingly, only a few professional developers have come to the events, and many participants have been beginners at programming, never having developed any kind of application before. As in all community pursuits, the variance and diversity reign.

With the exception of the ‘well logs’ event, which was very small, the events have run as friendly contests, with people competing as teams of up to four. The usual format is:

The Saturday morning is full of buzz and promise, as people meet and swap ideas. The openness is contagious, as everyone realizes they have far more to gain from others’ input than they have to lose. Gradually, a sort of skills bazaar emerges, as people either recruit talent to their teams, or peddle their abilities to others. By the way, non-programming skills – geophysical insight, workflow knowledge, mathematics, graphic design, user testing, and so on – are as much in demand as coding skills.

Once teams settle down and start making things, the day goes quickly. At 5 pm most people are ready for a break and, while some hackathons go all night, we prefer to keep it civilized and, after some food and drink, and dissecting of the day’s activities, we call it a day.

The Sunday is highly focused, as people whip their skeletal code into something they can demo. The demos are only 5 minutes long, but that’s plenty of time to share something about the vision, and show what the team has achieved – in what amounts to only about 50–60 person-hours of coding.

The demos are witnessed by judges, though this makes it sound more formal and more intimidating than it actually is. For one thing, the judges themselves have always been quite informal and unintimidating people – instead they are enthusiasts of technology, and champions of creativity. They have included dGB founder Paul de Groot, Enthought founder Eric Jones, SEG Executives Chris Krohn and Dennis Cooke, tech manager Shannon Bjarnason, software gurus Tracy Stark, Steve Alcock, and David Holmes, and web-savvy scientist Maitri Erwin.

Without fail, the people in the room at the end of the hackathon – including the participants themselves – have been blown away by what the teams have achieved. We do give out a few prizes, mostly just for fun, but everyone walks away with new friends, new skills, and a sense of accomplishment.

The outcomes

There’s quite a bit of variety in what people have produced at the events so far. This is all of them:

Houston, 2013 (ageo.co/1KWnFZ8)

Velocity modelling in iOS. Essau Worthy-Blackwell (Southwestern Energy), Jacob Foshee (Durwella), Evan Bianco (Agile), and Ben Bougher (UBC and Agile) built two tools exploring velocity modelling and depth conversion – a Python desktop tool and an iOS app.

LAS soup demux. Joseph Jennings (Colorado School of Mines), Mike Stone (Lukoil), and Karl Schleicher (University of Texas at Austin) built a tool in Python for coping with messy data. Repo: github.com/toastar/Las_View_tool.

A 1D forward model. Duncan Child (Spectraseis) joined Chris Chalcraft, Paul Garossino, and James Alison (all from OpenGeoSolutions) to help out with his crazy Java and JSON wrangling skills. Repo: github.com/duncanchild/blockfilter.

Houston, 2014 (ageo.co/1KXuYlW)

Dynamic log display. Jacob Foshee and Ben Bougher worked on a dynamic display not just for one log with its zoomed version, but for multiple logs at multiple scales – and core photos too. Repo: github.com/jfoshee/rockhack.

Multi-scale log attributes. Evan Bianco and Mark Dahl (ConocoPhillips Canada – who was new to Python on the Friday before the event!) – built some fascinating scalespace displays. It’s a little like spectral decomposition for 1D data. Repo: github.com/EvanBianco/log_moments.

Facies from logs. In between running for tacos, I worked on computing texture attributes, sensitive to log shape, to help predict facies from well logs. Notebook: ageo.co/logGLCM.

Facies from core. Wassim Benhallam (University of Utah) worked on machine learning algorithms for computing facies from core. He started with self-organizing maps as an interesting line of attack.

Denver, 2014 (ageo.co/1Mk4aNX)

Magnetotelluric web app. Andrew Pethick (Curtin University), Josh Poirier (NEOS), Colton Kohnke, Katerina Gonzales, and Elijah Thomas (Colorado School of Mines) built MT Black Box, a magnetotellurics modeling web application. Repo: github.com/AndrewPethick/MTBlackBox.

Magnetotelluric Android app. Elias Arias, Brent Putman, Thomas Rapstine, and Gabriel Martinez, four young geophysicists at Colorado School of Mines, pulled together a great little app for forward modeling magnetotelluric responses. Repo: github.com/Hackathon2014/Seemingly-Concerned-Neighbors

Well tie app. Mark Roberts (geophysicist), and Michaël Montouchet and Graham Dawes (ffA GeoTeric) hacked on a well-tie web application, rightly identifying a gap in the open source geophysics ecosystem.

DC resistivity web app. Rowan Cockett and Seogi Kang (both at UBC) didn’t get going till Sunday morning, but in just five hours they built a sweet web app for forward modeling the DC resistivity response of a buried disk. Minds were blown.

Pick This! Jacob Foshee (Durwella), Ben Bougher, Evan Bianco, and an immiscible mixture of Chris Chalcraft and me, built a simple way to share images and invite others to interpret them. Not only did this team demo live software at pickthis.io, but the audience provided the first crowdsourced picks in real time.

Calgary, 2015 (ageo.co/1T69bwO)

Data hack. Steve Lynch (3rd Science Solutions), who had hardly played with Python before the hacker bootcamp the day before the hackathon, built a Python interface for his Seismic Terrain Explorer application.

A geoscience crowdfunding tool. Raquel Theodoro (Rio de Janeiro) and Ben Bougher built the first version of GLauncher, a tool for finding and supporting geoscience projects. Raquel found an open source app for making crowdfunding sites, and spent the weekend adapting it.

LAS file web app. Gord Foo (Parex), me, and Gerry Cao and Yongxin Liu of Calgary, built Hacksaw, a quick-look for LAS files in a mobilefriendly web app. Repo: github.com/kwinkunks/hacksaw

Turning sketches in to models. Ben Bougher, Evan Saltman (BP), Elwyn Galloway (Calgary geophysicist), and Matteo Niccoli (ConocoPhillips) built a new script for modelr.io (a forward modeling web app) which turns back-of-the-napkin sketches into synthetic seismic sections.

Awesome sponsors are awesome

Making anything happen takes money as well as effort. Making new things happen also takes a measure of trust. The companies that have financially supported the hackathons, which have all been free of charge, have demonstrated their vision and the sort of abiding curiosity that this industry is going to need if it’s going to continue bridging the energy gap between yesterday and tomorrow. Here they are:

• dGB Earth Science – the Dutch company behind OpendTect, the only free and open source seismic interpretation tool, has supported every event.
• OpenGeoSolutions – a Calgary company that punches well above its weight, owing to having lots of very smart people. The company has boldly supported every event.
• Enthought – the original scientific Python company, full of fantastic developers, including quite a few geophysicists.
• GeoTeric – the UK company behind the GeoTeric software, a seismic analysis toolbox famous for beautiful spectral displays.
• IKON – the Canadian office of the UK rock physics company generously supported our Calgary hackathon.
• Geomodeling – the Calgary reservoir simulation company knows all about innovation and tight code.
• CSEG – the wonderful RECORDER placed a couple of gratis ads for the Calgary hackathon. Thank you!

What’s next?

Hackathons are not new. For at least a decade, they have been an important way for groups of technical professionals – in corporations, government, non-profits, educational institutions – to nudge innovation along. IBM, Amazon, Google, the Guardian newspaper, the New York Times, the City of Houston, Rice University (e.g. Frazier 2014), NYU and Columbia, and hundreds of other known innovators use public and private hackathons as one tool to find and champion people who will make change happen in their fields.

The list of projects might give the impression that code is the primary product of a hackathon, but that would be wrong. The Unsession was an idea hackathon, resulting in a list of pressing questions and an article (Hall & Bianco 2013). Some scientists are using hackathons to write scientific papers as well as software (e.g. Groen and Calderhead 2014). But still, it’s the intangible products we value most.

The connections people make go far beyond the sort of chance meeting in the conference hallway, because the connection is relevant and purposeful. The transfer of skills during the event exceeds what most people experience in a training course. And the events are a chance for the community to prototype new ideas and push the boundaries. Others have reached similar conclusions, e.g. Trainer and Herbsleb (2014):

“hackathons for science”, hackathons used to advance infrastructure for science, facilitate the circulation of important knowledge, such as the scientific community’s priorities, technical know-how, and social connections.

Möller et al. (2013) found the same thing in the biosciences:

[A hackathon’s] true worth is more intangible in the form of the community itself, new and strengthened collaborations, and the spread of ideas and best practice – both scientific and for software development.

Convinced of their effectiveness as accelerators, we are going to run more private and public collaborative events in the coming months and years. The current economic landscape is a bit of a challenge for the public events but we hope to coordinate some sort of group coding at the SEG Annual Meeting in New Orleans – watch for alerts on agilegeoscience.com and @kwinkunks on Twitter. After that, we plan to do something in Europe next year, and will be in Calgary at the AAPG Annual Convention and Exhibition next June.

If you’re even a little curious about these events, please come along to the next one, or just drop in for a visit. And if you have ideas for hackathon locations, themes, or partners, please get in touch at matt@agilegeoscience.com.

Acknowledgements

A huge thank you to everyone that has participated in these events as a hacker or as a judge, and to the dozens of people that have dropped in to say ‘hello’. Thank you again to the event sponsors. I’m grateful to the spaces that have hosted us, especially START Houston for putting up with our early flailing as we figured things out. I’m also grateful to Matteo Niccoli for his help getting the Calgary event together.