Comparing Creative Writing with Software Development

Recently I talked with an acquaintance of mine, who is writing novels and essays since over 50 years. He asked me what I was doing, and I said I am doing software development. The discussion lead to an interesting philosophical discussion of how software engineering compares to writing novels.

As a software developer, we are actually usually doing creative work, because we solve novel problems. If the problem was already solved, we usually either automate it, using the power of abstraction, or we can find the solution elsewhere by accessing libraries and open source projects. Of course there can be tasks that don’t require the creative problem-solving abilities of the software engineer, but in my experience, the creative part of development is about 80% of the time. A good engineer also will always work towards automating and abstracting repetitive tasks and continuous improvement - this is in my opinion the essence of effective software development.

The trouble with software development is that we write huge amounts of text, hundreds and thousands of pages. To develop a running software product, we usually employ a team of engineers. This team has to work together to produce thousands of pages of text, that somehow have to go together coherently. How would you think about that if you read a novel that has been written by 5 or 6 authors? For sure it wouldn’t be a very coherent novel. It would be extremely hard to make a great novel in this setup. However this reflects actually the real challenge of software engineering - how can we form and organise teams that can produce coherent source code in order to produce a running software product, that is on top of that functional, performant and fulfilling the users needs efficiently? And imagine that, there are software teams with 10s and even hundreds of team members. A truly herculean task to produce something worthwhile here. What could be the solution to this conundrum?

How would we write a book or a novel with many different authors? We probably would have chapters, and each chapter written by a single author - so at least this chapter would be coherent and in a single style by this author. There would maybe be an editor whose task it is to give a theme to the authors for each chapter, have a soft of outline for the whole book, and edit the chapters to make them fit together. Maybe another allegory would be a newspaper. A newspaper is a collection of small texts, all written by maybe 1 or 2 authors. They are inside relatively coherent, but a newspaper in itself usually offers a wide spread of writing styles, topics and also varies in quality.

But the story goes on.

In software development, we defined our own language. There are generations of languages, and they are increasingly abstracting things. When in the past we maybe had to write 10s or hundreds of pages to build a web app that publishes a single service, with the latest generation of languages and frameworks, for instance Elixir / Phoenix or Ruby on Rails, it is now sufficient to write a few lines. This will go on an on - every 5 years or so we will experience a new leap in abstraction.

Also, for example, look at technologies like Docker - abstracting the complete runtime of a single server software in a simple file with maybe 10 lines or so. Or a Platform as a Service, which completely abstracts the large components of an application, and includes deployment and provisioning with a single line of code.

In order to deal with this huge complexity, we are trying to trim down a specific application or module of a software, making it as simple as possible. Recently, the micro-service paradigm came up. Here the idea is that a very small team produces one small part, a stand-alone service, that covers a fraction of Much thought has to be invested though in the orchestration of those services. In a way, we now have the newspaper model as described, but employing this model also means that there needs to be investment in the editing - i.e. how can the different, diverse services be put together to produce something coherent?

It gets even more fantastic, if you think about the open source community. People create, often in their spare time, products, libraries and other pieces of software, which are later integrated to produce new products. This is as if someone is ghost-writing parts of a novel or a newspaper, and it is later edited into the text that is published.

All this basically highlights that writing software is much more of a complex task if you look on the scale of it, due to the sheer amount of text that is being written. Also it is a problem of people working together. People that have to do creative work, and improve the way they deliver it continuously. This concludes that we must constantly invest in improving our development techniques, our ways of working together, evaluate and incorporate new technologies, with the goal to unlock the next abstraction level.