Continuous Desintegration

All quiet on the technical front

John is a prolific developer who is currently working at the F-Corp company. He is a very curious guy, who is constantly looking to learn new ideas from other domains, besides new technologies.

In that role, not only is he in charge of an important part of the application, but also leads a team of other three junior developers. He likes to guide (not manage) this small fellowship, as they like to call themselves. And his developers admire his broad knowledge, aside from his technical skills. I am sure you would like him.

As part of the “Agile” methodology that F-Corp uses to develop software, besides programming and helping his team, John reviews the PRD that the Product Team plans to build before starting their iterative building 3-week cycles, also known as Sprints.

PRD stands for Product Requirements Document. Most people agree that the document should describe the new feature that the team is supposed to build with its functional requirements, but at F-Corp there isn’t a specific template. Sometimes reading the PRDs feels like a pitch for investors instead of a blueprint for the building team.

John hates reviewing these documents. And why is that? Because the Product Team usually submits for review this document just two or three days before starting the next cycle and usually they lack a lot of definitions. As I mentioned, usually it is not clear what John’s fellowship is expected to build.

But we are not being fair here, some of those PRDs come with a high-fidelity Figma[1] file made by the solely exhausted designer. Yes, there is only one designer that needs to supply high fidelity mockups to 4 teams. But still John is right that those high-fidelity designs don’t help him much. They tend to have too many details and are too rigid for change suggestions.

From time to time it happened that John said: ‘We can’t build that’. And the Product Team, furiously, had to invest another cycle to change the whole design. In order to avoid those angry looking faces, John applied his Darwinian skills and learned to adapt.

Now, Instead of saying, he can’t build that, he just spends a cycle trying to make it work. Not without bumps, as he has to ask for (many, too many) clarifications for what is expected to be built and suggest changes to the UI design so it can be implemented.

The whole back and forth takes him the whole cycle. And this causes his team to delay the smooth building phase of the PRD until the next cycle. What was supposed to be a project of one cycle ends up taking two full cycles (sometimes three, with more angry looking faces ). One (or two) cycles for what technically is known as Spiking and another for the real building.

And sadly for John, who would rather spend his time focused writing code and helping grow his mentors, all these exchanges happen at the profane business gathering known as a meeting. Somehow the F-Corp Product Team finds it hard to clearly express in written format. They need a meeting to make a decision, no matter how small it is.

But let us be less harsh with them, because it isn’t entirely their fault. They have also applied their Darwinian skills and adapted too. Because at F-Corp, the business executives, the ones who end up making the business (and product) decisions, are always in meetings, so they can’t read nor write emails (they have too many).

Back to where we were. John finds it disgusting to spend two cycles on a project that was supposed to take just one. In his mind, the PRD would be better if it had the technical clarifications that emerged with the spiking. But somehow he can’t think of a better alternative, even though deep in his heart he feels this is something that he knows.

From Toyota to XP and back to College

Developers, like John, have fully embraced a practice called Continuous Integration. It was highly promoted by the practitioners of Extreme Programming methodology, and specially by (and I stand up), the great Kent Beck who describes this idea in full detail in his book Extreme Programming explained.

When building software became more complex and required more people to successfully ship it in time, the maxim divide and conquer became an allied. Software had to be divided into different parts that could be worked independently so different people could work in parallel. But that orchestration and design wasn’t (and still isn’t) easy.

Someone had to take the difficult task and lead. The Software Architect became real. A very senior developer who had the responsibility of splitting the software into different modules that at some point would interact between them.

Developers and the architect negotiated the interfaces on which their modules would interact with, and then, they just went to their obscure desk to write what their module was expected to do.

As the saying goes, “there is no such thing as a free lunch.” As the developers deepen into the problem, some interfaces change. Days and months, however, could transcur without anyone trying all the modules together. This integration as a whole was usually left for the end of the building phase. And when it finally happened, people realized that what they had aligned to do wasn’t there anymore.

This was a heavy burden, because in order to integrate and make sure everything worked correctly, many parts needed to be rewritten. In order more simple words, rework and delays.

This is the problem that Continuous Integration solved, it enforced developers to integrate their software at a higher frequency. The months became weeks, then days, to just a few hours, and in highly productive teams, those who practiced XP programming, the integration happened automatically on every commit. Building all the modules together and then running all the tests automatically.

With this continuous practice the developers were alerted that their modules were causing trouble to other developers modules, leading to talk and design a solution together. Attacking the problem before the egg hatches, in other words spending small time now to prevent future big problems in the form of rework, highly prevents overspending. It is a cost-effective risk mitigation strategy.

And this isn’t a software industry idea only, this form of risk mitigation is similar to the concept of pulling the Andon Cord, a practice developed by the Toyota Production System and its preacher Taiichi Ohno.

It is interesting that the continuous integration practice doesn’t come intuitively to us. I was a teaching assistant at the course of System Design. And in that course there was a big software project that students had to build. The idea was that they worked in groups so they can design the system and divide the modules.

Continuous integration, however, wasn’t part of the course content. In the college program it was supposed to be taught in another course that would come next, Software Engineering. What I saw at my course was that most students tended to integrate their project at the end, which caused them many nights without sleep before the deadline. But the few students that were conscious about integration problems were the ones that had some real working experience.

That is why with all the other teaching assistants, we decided to introduce the continuous integration practice in the System Design course.

Good teachers help students succeed. Bad teachers focus only on the program and the exams.

Domain-Driven Dilemma

John as a software developer, but also as a scholar of software building practices, of course knows about Continuous Integration (yes, he belongs to that rare species of employees who in their dead time at the office read software engineering books instead of newspapers and blogs).

In fact he helped to set up the company CI Server (I think it was a Jenkins) to automatically build and run the tests of all the company projects. But somehow, he couldn’t see that his problem, the PRD rework and overspending time to understand what needed to be built, could be solved by engaging with the product team more frequently and being involved before the first draft of the PRD. In other words, by continuously integrating the product team knowledge with the engineering knowledge.

Please, let us not blame the Product Team for not knowing this, they weren’t formally trained with software engineering ideas and practices. But also, let us not blame John either. As John suffers a heavy case of what is called domain dependence.

What is domain dependence? It is the problem of not being able to see that an idea that was learned in a particular domain, it can also be applied successfully in another one. The patterns, knowledge and practices we learned through our lifetime are heavily tunneled.

Particularly, I hate myself when It takes me a long time to realize that an idea that I learned from software engineering could be also applied to business management. It is a bias that we all must suffer. There is no escape.

What John didn’t learn, and here we can blame him, during his career of 15 years working in the software industry, is that his role and responsibilities change as technology changes. When he started software products and apps weren’t as common as they are nowadays.

There were many professional people, companies and other realms that didn’t have any specialized software to manage their business processes. Deeply understanding their processes and struggles wasn’t a problem, any general (or poorly designed) solution would fit.

The hard problem, instead, was how to write and deploy software into production. That is why the role of software engineering was heavily focused on writing high quality and maintainable code.

The pin factory meets the (software) office Software devoured, digested, regurgitated and devoured the world again, to paraphrase the words of Marc Andreessen, the founder of the venture capital firm a16z (but I prefer to remember him as the writer of the first browser and founder of Netscape[2]).

Industries changed, and every industry that incorporated software changed fast. Writing and deploying software became easier. There are new languages and frameworks (and new versions) that with 10 lines of code can do the same of what required 500 before.

At the time of writing, there is a new kid on town, Vibe Coding, a methodology on which the developers (and not only technical people use it) don’t need to write code at all, the just write what they need to an LLM agent, and the agent just outputs the whole code that makes it work.

Of course this lowers the entry barriers, and makes it easier for new firms to compete in existing markets. And they did, as competitors appeared everywhere. Now, there is an app for everything, there are apps that monitor how you eat, walk, drink and sleep (and I bet there are apps that monitor something else you also do where you sleep).

With this fierce competition, developers’ expertise wasn’t enough to turn users into real customers, to build products that helped them succeed. In the past the competing substitute was a manual and paper process, or nothing at all. Now, in order to convert users, differentiation and specificity is a must.

The first major improvement was to introduce good looking UIs and better flows. The developers, like John, weren’t good at designing UIs. That is how the UI designer came to the scene. Their role was to improve the UIs to make them good looking. Some of them (the good ones) started to focus on the whole experience, thinking about the way the user interacts and flows through the app. Some others (the great ones), following the ideas of (now I stand again) Don Norman and Jakob Nielsen, tried to build simple prototypes to test what struggles the users had.

When work becomes more complex, more knowledge, in this case specific knowledge, is needed to achieve better (and sometimes the same) results. The people who attended business school took what they learned about Taylor, and introduced the well-known tactic of division of labor.

Still these two roles weren’t enough to keep competition on the line, there was needed another person to connect both sides and also think about the business and the product as a whole, but also, who could also focus on understanding the customers struggles and needs.

At some small private companies the CEO fit the role very well. However, when these companies start to grow, the CEOs have to step aside from their down-to-earth work, to focus on what they call “culture”, “business strategy” and the most important thing “shareholders’ value”, and of course attend (boring) meetings.

Two for one, and one for two

Someone else had to step on the product shoes. The product manager became another person in the group.

Someone who gets to know the user. Their problem, their struggles, how they define progress, and what are the workarounds they incur in order to solve all these. But also someone who learns how the business makes (or could make more) money and how to differentiate from the competition really well.

The original idea, in theory, was that the product manager with all this knowledge would take the decisions and ownership of the direction of the product.

In practice, they don’t have full ownership of the product, and they have to ask for permission from the CEO for each important (and sometimes trivial too) decision.

The product manager’s primary goals are to take her knowledge, the input from the designer and the developer in order to choose which features to build. The product manager and the designer are very sociable people. When they talk about implementing their ideas they mean to help other people (usually the users).[3]

But developers, like John, when they talk about implementing their ideas, they mean to help the computer work better. I am being harsh here, some of them, the good ones, at least think about helping their future selves understand the code they wrote better; some few, the great ones, try to implement their ideas in order to help other developers understand the code better.

They were supposed to work in the way Alexander Dumas imagined the Three Musketeers, un pour tous, tous pour un. But the traits that make them excel at their job come with a trade-off. Those attributes drive the product manager and the designer to be working together on one side, and the developer on the other one.

That is what happened at John’s company.

The doors of perception

As I mentioned a few paragraphs before, with the technological advances, writing software became easier. What took 500 lines of code, now it can be done with 10 and still preserve (or even increase) the expected quality for the user.

But somehow, the role of engineering in most companies like John’s, with this new surplus time, instead of connecting with business experts and integrating their knowledge to define a solution, they closed the doors and worked on improvements with detrimental margins. With the side-effect of making everything more complex.

As a variation of the Jevons Paradox, which states that any efficiency improvement instead of saving energy, leads to increased consumption: Any technological improvement that reduces the code needed (frameworks, computer performance, languages, etc), ironically, leads developers to write more code.

This time surplus doesn’t get spent into thinking better solutions. It gets spent writing new software to manage the existing software complexity. We live in a crazy world. But let us be permissive: A complex solution is a necessary step for a simpler one. Some teams go the distance, some others don’t.

The great Rory Sutherland tells this funny story about the Post Office[4]

The managers wanted to improve their success delivery rate for their first-class service. At that time 98% of delivery arrived the next day, but they wanted to make it 99%. Of course these last mile improvements require much more effort than going from 80% to 90%. Costs are non-linear, and every new improvement marginally is less effective. This case was no exception and they invested a lot of money and people’s stress, to increase it only 1%.

But here is the moral (and funny side) of the story. The average Post Office user on the street thought that their success rate was around 50%. They focused on improving something that didn’t bring any perceptual value.

In the case of John’s the story is almost similar. If instead of focusing on writing more code, he used his technical knowledge to help select the problems that are easier to solve, define more possible solutions, and make trade-offs, the user could have a greater perceptual value. This role is what now takes the name of a Product Engineer, and will become more important as code becomes easier and easier to write.

And this leads me to the next section.

Aren’t we supposed to be knowledge workers?

What is really the code? In most software products the code is a way to automate and scale a process which is too tedious for a human to do or would require too many people to do it.

But going one level further, the code is a way to describe a process, its entities and transformations. In other words, the code represents knowledge. The knowledge the organization needs to deliver value to its clients, and programming is a way to build that knowledge in a clear way.

This is the view of Peter Naur, who in 1985 wrote a paper called “Programming as Theory Building”[5] which can be summarized almost as if the code isn’t the end. The code is a means to collect knowledge about the problem and its possible solutions, the present ones and the past ones that were tried but didn’t work.

If we take a moment to process this, it means that the role John has in the organization should be different. He and his engineering team have all the knowledge about the system, the organization, its processes, and its transformations to deliver value with its nuances. Add to that that they also have the technical knowledge that makes it work under the hood automatically.

My point is that John, as a technical lead, shouldn’t be continuously integrating code. He should be continuously integrating knowledge. And the people he should be integrating with are the product manager and the designer.

To go further with the analogy. Let us remember that without continuously integrating code (thank you XP practitioners again), the deviations between the modules in development can expand further which will always lead to rewriting and huge delays.

So in an (almost perfect) similar case, when the product manager, the designer and John don’t integrate their knowledge, their idea of the product deviates further and further from what it can be implemented. And this is what leads to constant problem and solution redefinitions with massive delays and stressful situations.

Low frequency, high time period

The next question would be how frequently should this continuous knowledge integration be. And sadly, I don’t have an answer for that. It highly depends on the capacity the tech lead like John has, and how technically complex is the feature or problem to solve.

The interesting thing is that the more time John can have because of technological advances, the more he can dedicate time to help design the solution from the technical perspective.

Thankfully we have (and here I stand with my hands to the air) Ryan Singer to teach us mere product engineers mortals how to work better. He introduced the methodology of Shape up in his book and talks. And in his course he mentions that this integration happens in what he calls a Shaping Session[6].

This shaping session should be a live session, not asynchronous, where the three hats (design, product and tech) align on a clear solution, defined so that the build team can work autonomously knowing what is expected from them and how they can start.

Of course these Shaping sessions can happen multiple times for a specific project if no clear solution is found. And the number increases if multiple projects can be tackled in parallel by the different building teams on each cycle.

However, this time invested by the tech lead has a huge payoff, because the building team can work more autonomously and with less rework.

As all of us like John, who come from the technical realm know, and the more important concept to take away from this essay: Integrating at the end doesn’t work. It only causes stress, rework and delays.

And if you can only do one thing in your organization, modify the pipeline so that the tech lead can be present during feature decisions. Schedule a Shaping session. It can give you big improvements with a small cost.

A lever is what moves the world As I am writing these ending lines, I have the feeling that John will read this essay and he will totally understand this perspective and try to implement it at his office. He is a very smart guy. Please send it to him if you know him. I hope that he has sufficient leverage so that other people in his office can pay attention to him. Sadly he forgot to study greek rhetoric and business verbiage.

And to you reader, thank you very much for taking this (long) time with this piece. Farewell.

–-

Notes and Further Reading

1. At the time of writing this Figma is the software that many UX designers use to make high fidelity designs, that developers don’t know if they can build.
2. You know how the meme says: “You either die as a founder or you live long enough to see yourself become a VC”.
3. Marty Cagan in his books Inspired and Empowered defines the responsibilities of the product manager and how this triumvirate is supposed to work together from a higher perspective. I added Bob Moestas’s JTBD ideas as the product manager’s main focus because he gives a very concrete framework to work with.
4. If we were able to choose we wouldn’t want a choice by Rory Sutherland
5. Programming as Theory Building by Peter Naur
6. Ryan Singer has very good videos on youtube and has a course about Shape Up. But here you can check out how a shaping session works.