The Agile Paradox

It is hard to find a company today that doesn’t claim to be “doing agile” in some form of another. It is also easy to find criticisms of the approach and many examples of organisations who have struggled or failed to succeed with it, whether in software development, HR, organisational transformation or service delivery.

The values and the four principles of the Agile Manifesto are simple to understand, and Scrum, the most common Agile approach, is described in a brief guide of just 14 pages (and that includes the cover page and the table of contents).

The simplicity of Agile seduces people into believing that they understand it and can apply it. After all, you can read the Scrum Guide in half an hour and be certified as a “master” in just two days.

People are however complex and unpredictable, and they don’t like change. They have got to where they are by following a particular path, adhering to particular values and accepting particular truths. When organisations begin to adopt Agile, many of those historic truths are challenged. Read with a non-Agile frame of mind, the values and principles of the Agile Manifesto are often misunderstood, misused and reinterpreted to fit the prevalent mindset. Interventions to try and help often instead undermine the core values and principles.

Agile courses are often marketed like many other skills courses, and people assume that completing the course will mean they can get immediate results. In addition, Agile is often presented as a set of rules, practices and techniques that can be learned and followed. We are used to other skills that are very deterministic – when we follow the rules correctly, we will get predictable results.

In reality, however, very few people succeed straight away and even highly experienced practitioners don’t find it easy. In fact, to achieve some of the highest levels of certifications, the examiners will expect to hear stories of where things have gone wrong to be believed.

Overcoming uncertainty

Agile is hard because of the very problem that it was conceived to address – uncertainty. Traditional planning approaches assume that the problem space is relatively stable and predictable and that putting enough effort into the analysis and planning will result in a successful outcome. This thinking is sometimes even reflected in the names of such legacy approaches, for example “PRINCE”, which is short for “PRojects IN Controlled Environments.”

The idea is that by imposing the harness of such a traditional framework onto a problem space, the environment becomes controlled and uncertainty is removed. Anything that doesn’t fit the controlled environment is viewed as an abnormality and must the mitigated through certain controls and processes. Agile specifically assumes the opposite – that there will be change and that the future beyond a few days or weeks is impossible to predict.

Agile teams handles uncertainty by specifically not dealing in absolutes. Decisions and judgements are much more nuanced. Decisions are very people-centric and people, being inherently complex and unpredictable, adds additional layers of uncertainty to already challenging problems, meaning that decision-making must be particularly nimble.

A conflict of cultures

Even when teams embody an Agile mindset and values, the same is not always true of the management, leadership and governance around them. Agile is more than adopting new terminologies and unusual events. It’s about a fundamental shift in the things we value throughout the organisation. Because of our natural resistance to change, this can be a hard thing to overcome.

Teams and organisations begin making local optimisations to help it fit in with other processes or find jobs for staff who don’t fit neatly into Agile roles. They attempt to make the work more predictable, try to forecast over longer periods or scale to larger teams and organisational structures.

Encompassing frameworks have been built around Agile teams to address this. Frameworks such as SAFe are particularly popular. These typically try to help by adding guidance and structure (and complexity) to the Agile approaches. In the best of worlds they can provide guard rails and confidence to Agile teams that help them make decisions while upholding the Agile values and principles.

More commonly, however, they serve as a rulebook and set of instructions that must be followed.

For many people, especially those whose history and previous successes have conditioned them to seek stability and certainty, that second effect is the one that dominates.

Given these factors, even experienced and confident teams can struggle. For new, inexperienced or junior teams it’s even harder. We ask them to follow an Agile approach that they have been told is easy. Then they interact with people who change their minds, aren’t certain of the future and don’t behave predictably, and we expect them to embrace such changes and deliver in a predictable manner. No wonder they struggle. And that’s before we even think of scaling Agile, which compounds this problem in multiple dimensions.

The Agile Manifesto

The Agile Manifesto says we have come to value certain things.

Individuals and interactions over processes and tools: Most complex problems are complex because they involve people. Focusing on people and how we interact will help us navigate that complexity better than trying to predict in advance and codify it into a process or a tool.

Working software over comprehensive documentation: What matters is that the sooner we get something to the customer that helps to solve their actual problem, the sooner we can get feedback that we can act upon. It’s no until the user actually uses the product that they can be certain they were right about the problem and the team can be confident they were right about the solution. Delivery of working software generates realisation and discovery among users which guides the team’s work. Earlier delivery also means earlier value to the business.

Customer collaboration over contract negotiation: Contract negotiation implies an end point is reached before work begins. The contract is often an internal agreement such as a requirements specification or a product roadmap, and the expectations on the team that stems from this. It implies that the role of the customer ends with that agreement and that they aren’t allowed to change their mind. However, when the problem is complex with inherent uncertainty, anything that locks out change will result in a solution that is at least partially wrong and often even severely wrong. The understanding of the problem and its solution must be constantly developed, and we can’t do that without the customer’s involvement.

Responding to change over following a plan: In Agile, we expect change and act accordingly. In many other approaches, we try to prevent or at least control change. When we value following a plan, we assume the plan is correct and that what we thought was true at the start remains true throughout the period. As it happens, that is rarely ever the case.

Beginning to succeed with Agile

The four principles of the Agile Manifesto really describes the differences in mindset required for Agile to succeed. On the left-hand side we find the Agile mindset, and on the right-hand side is the traditional legacy mindset. Unfortunately, the right-hand mindset is the one that overwhelmingly dominates traditional solution delivery processes and organisational management.

To help organisations adopt an Agile mindset and help Agile teams to deliver, it is important to recognise this and help shift the mindset towards one that values the left-hand side of the manifesto. Several enablers are required to make this happen. Research made by Google and other organisations have found that the most important enabler is psychological safety and mitigating fear of failure. Without this in place, it doesn’t matter what other enablers are introduced.