An ever evolving struggle against waste


October 31, 2016
Jon Pither

Waste and Me

Everyone talks about wanting to reduce the waste in IT. Or if they don't, then they should be. Waste bears down personally on my soul. I fret constantly about the complexity of the household recycling system and I get enraged by direct mail advertising pushed through my letter box.

I am staggered, overwhelmed, and dismayed by the epic waste on show in Enterprise IT software development. At the lower end it's whimsical code that occupys CPU cycles, as though the universal heat death needs spurring on. At the top end it's the sheer disorganisation of management - projects being started and later abandoned with years of peoples' lives given barely a casual afterthought.

I remember the project that raised my consciousness of waste. I was working for a large consultancy and we were tasked with improving Agile practices for a huge conglomerate with a multistory IT department. On the surface our job was to implement stand-ups and to push an automated testing agenda; to sprinkle across a bit of modernity and best practice.

It felt like fiddling with deck-chairs on a iceberg inclined cruise liner. This place felt broken but I didn't have the language for the 'why'. I lacked a framework for structuring my observations and judgements.

When I began to more deeply question my own existence a colleague suggested I read 'Lean Software Development: An Agile Toolkit' and my eyes were forever opened1. It was 'waste' - the glorious 7 Mudas of waste that could be articulated and understood by anyone. Waste is impersonal and omnipresent.

I could now see waste everywhere. I saw that the client had two version control systems for no reason; that they expensively built the software from CVS and then built the whole thing again from StarTeam, with both build processes requiring manual oversight from unspecified cubicles.

I saw that the super large Java project actually consisted of 255 smaller Java projects and that a developer would have the projects checked out on variety of different branches compared to the next developer. Yes, you heard that right, and if you processed it then you should take a moment. Let me say it again just in case: they had 255 Java projects all on different branches. This is why they required a department of people to manually build the software into a good shape. No two developers worked off the same baseline.

I couldn't sleep, and I certainly found it difficult to rouse the energy to add another unit-test to a branch of code that may never be checked out and inspected.

So I made a plan; I petitioned my client account manager for a sit-down with the client CTO. I spent time making a gorgeous looking Value-Stream-Map that highlighted with red hues where I thought the blockages were; the location of the expensive slow-downs.

When the sit-down happened for one of the few times in my career I felt like I'd truly cut through the mists of bullshit. Just after I'd finger pointed the waste, she looked at me gravely and said 'why didn't you mention all this earlier?'

She was enraged and I don't blame her. The waste was so epic, so expensive that we were duty bound to have raised our hands the minute we arrived. We didn't and we became part of the problem; perpetuating the waste by endorsing it with our silence.

The next frontier of waste

When I co-founded our software consultancy my partner and I hashed out what our core values were. Reducing waste was a joint value; we wanted our company to build production grade software for our clients with less of the waste overhead by using superior technology2 along with the best people we could find3.

Waste though, is a moving enemy. When you use modern and exciting technologies then you have to be vigilant that the developers do not overcook the pudding. Here are some recent examples that I've come across:

  • expansive Microservice architectures delivered required or not4.
  • overly formalised hard-to-change code, as a result of dogmatically applied type-definitions and enthusiastically applied low-level unit-tests.
  • dev-ops fiefdoms where the hardware provisioning, environment configuration and software deployment get muddled up together, leading to masses of scripts that require specialist full-time attention.
  • custom PaaS platforms.
  • premature abstraction layers over libraries and databases.
  • premature adoption of sophisticated distributed computational systems.

These examples have taught me the value of having leaders embedded in teams who are waste-conscious and business focused and who are resoundingly respectful of the trust placed in them to spend client resources wisely. Software development is not a democracy and someone needs to helm the bridge.

If you're going to make a software company for profit then waste can be a best friend. There are projects out there with hundreds of developers attached to them with the culpable monolith or microservice-big-bang-explosion sucking in resources like a black-hole.

I want our company to keep it honest and to avoid waste just like the client will want us to. This requires thinking like the developers who aren't consultants and who will be living with the solution for the long term.

It's requires vigilance and business focus, and being able to weigh up if a new technology is an enabler or a trap.

  1. The same colleague also recommended Non-Violent-Communication to help me get my message across.
  2. We use a Clojure stack to help us beat the averages.
  3. I believe the '10x developer concept' is largely true.
  4. Monolith first is often a much wiser route.