Summary

Many developers don’t speak to users or collaborate with the business, instead they take direction from email threads and poorly written tickets. Remote, outsourced and offshore developers know this only too well. Even with better access to people, these developers would struggle to navigate organisational complexity, political decision-making, and siloed systems.

Being on the receiving end of a poorly performing development team is upsetting, particularly if you’ve tried everything to fix the situation but low throughput and poor workmanship remain. The business pays the hidden price for this; individuals least well-placed to know what’s going wrong. Unfortunately, these experiences are only too familiar.

This handbook is written for development teams and their managers who need to write excellent software and delight users. It explains when software development works best, what happens when your team can’t work like that, and provides practical advice to address common pitfalls. You can share your experiences of Better Software Requirements directly with the author, who will be happy to respond.

Frank Ray | info@bettersoftware.uk | Website gateiogateio, 2024.

Table of Contents

Introduction

When software development works best

Software development works best when developers interact directly with users, hearing about their experiences with the software and discussing what should come next. Conversations replace the need for requirements, and ideas and concepts are quickly put together and pushed out the door. Developers confidently release to real users because high-quality processes guard against adverse effects. Feedback happens in near real-time; a simple comment, under-the-breath muttering or facial expression is often enough to gauge reception. It sounds like fantasy, but commercial software really was built like this circa year 2000, and still is in many startups and small companies.

This approach works because developers know multiple technologies and can work up and down the stack, delivering entire working features themselves. Developers create database tables and stored procedures, develop middle-tier code and author front-end interfaces (often not very good ones, I admit). They may have strengths in particular technologies, but what’s most important is delivering whole solutions. Additionally, these developers converse with real-life humans, sketch out rudimentary designs, and even test finished code, a far cry from most developers today.

Why software development goes wrong

The distance between the end user and software developer is the biggest indicator of whether your software product will delight users, or be a poor proxy of misunderstood needs. The further you stray from close collaboration between users and developers, the greater the likelihood of things going wrong, and the greater the need to manage the software development process carefully. Developers not sitting with actual users, or user advocates, break the close collaboration and direct feedback loop that avoids the need for formal software requirements and software specifications in the first place.

Technical silos

Modern software development is very different from the early 2000’s. Technical complexity means developers specialise in one or two technologies or frameworks, teams are distributed globally, ‘product ownership’ and ‘user experience’ are entirely new functions, and DevOps sits outside the development team. I’m not against any of this, nor am I advocating a return to an earlier time; rather, I’m pointing out how things have changed in the software development industry.

Large corporations develop software in ways that definitively break the link between users and developers, leaving a communication gap of monumental proportions. Users and developers no longer collaborate; sometimes, they have never even spoken to one another. Remote development teams, outsourced and located offshore in supplier ‘centres of excellence’ is standard practice. The user feedback loop is broken too, needing to traverse through layers of product owners, user researchers, designers and business analysts, before ending up in ticketing systems like Atlassian Jira and Azure DevOps.

Poor management

Someone needs to tell the developers what to do, otherwise certain chaos ensues. Good software developers can tolerate a measure of ambiguity in their work; many enjoy putting their skills to good use and resolving unknowns. What developers cannot do, however, is make decisions on behalf of a business that doesn’t know what it wants or navigate political environments and organisational complexity from behind their development IDE. Developers struggle to do this.

Someone, somewhere, needs to convert ideas and concepts into instructions for the software developers, but it’s hard to do well because you really need to understand the business problem you are trying to solve. Then, you must communicate effectively. Not knowing what to tell the developers isn’t an option, unless you want poorly built software, idle hands, or conversations and ideas that never become a reality. Each developer needs a prioritised work stack and sufficient work to do; large development teams with broad skill sets make this particularly tricky.

High complexity

Some software projects are more complicated than others. The risk of unexpected problems and total project failure is higher when one or more of the following are present:

  • New proposition and product development
  • Bespoke and custom software development
  • Integration of multiple backend systems
  • System rationalisation
  • Retiring end-of-life technology
  • Several suppliers present
  • Outsourced and offshore development
  • Poor quality or inexperienced developers
  • Absence of critical project roles
  • Operating in regulated environments
  • Developing safety-critical applications
  • History of failed projects and poor delivery

My software development is going wrong

You’ll attend a product demo and be either thoroughly underwhelmed at the progress made or simply furious at the sloppiness you see, or you’ll lie awake at night worrying that you don’t know what is being delivered, and when, even approximately or on a prioritised basis. These are major indicators that something is wrong with your software development process.

Regularly cancelled product demos because of overrunning development. Infrequent production releases to actual users. Production releases that go wrong; unexpected outages, data corruption and spiralling bug counts. Unhappy users when they finally get new software versions. Frustrated developers working in a team with low morale and high staff attrition. These are other indicators to look out for.

What developers need

Developers need to know what to do and be able to make the right decisions. Some developers tolerate ambiguity better than others, however they still need a clear understanding of what to do. Experienced developers in small teams with good access to users require less prescriptive requirements; junior developers in remote teams building enterprise software require more. Some folk believe modern agile practices remove the need for software requirements, but this couldn’t be further from the truth.

The further developers are away from end users and business stakeholders, the greater their reliance on written communication. Developers who work without direct, personal user feedback depend heavily on written requirements as their primary source of guidance. By all means, bring users and developers closer together, but when you can’t, you need software requirements. Software requirements are far from perfect, often only a poor approximation of close collaboration and direct feedback. Still, they are necessary to bridge the monumental communication gap that exists in modern software development. This is particularly true in large corporate and enterprise settings.

Occasionally, the exact thing to build is known but the software requirements lack detail, such as acceptance criteria, business rules, testing scenarios, and other important things. You’ll see developers becoming blocked in their work, needing conversations to iron out the details, and then waiting for system access and environments to be made available. Better software requirements, adequate time to review them as a team, and consideration for third-party dependencies are usually the solution.

Better software requirements

Better software requirements are a team sport that requires the participation of each team member. The idea that someone, single-handedly, can write down exactly what the developers should do to create a market-leading software product is unrealistic. It also shows contempt for your teammates who have much to offer in the process.

Many concepts, ideas, and opinions feed into the early stages of requirements gathering, each involving deliberate sense-making and consideration before informing the technical team’s work. It‚Äôs a messy and complicated process, but software requirements are a human affair. Read on for some practical advice about how to better navigate this endeavour.

Gathering software requirements

“Have you got the requirements yet?” is a well-intended but misguided question anxious stakeholders ask when faced with an urgent need to deliver some kind of system, unrealistic deadlines and the desire for progress. If only the requirements were complete, then the real work could commence, goes their thinking.

Unfortunately, software requirements aren’t necessarily known in advance, they aren’t located in the heads of the stakeholders, and they aren’t solicited in back-to-back Zoom meetings, nor are they lying in hallways waiting to be collected like eggs from a chicken coop. Equally, going around saying “Tell me what you want” is a pretty terrible approach and certainly won’t provide the answers needed or hoped for. You wouldn’t need to hire a business analyst if it really was this easy.

Requirements gathering isn’t a simple linear process, and unfortunately, it often can’t be planned and executed in a deterministic manner. Sure, you can schedule two weeks of stakeholder interviews on a project plan, and you will find out a heap of stuff, but you may not get the answers you need to build the kind of software the business demands, and users want. Interestingly, IT system builds often represent a deep, unmet organisational need and there is immense value in accurately clarifying and surfacing it.

Quickly capturing enough of an initial backlog for the early sprints demonstrates progress, aligns developers and allows work to commence. Everyone starts to feel better. The risk of rework and the ability to pivot is considered when compiling the initial backlog, given the high chance of this early work being off the mark. Importantly, commencing development secures the time and remit to commence the real work of understanding deeper organisational needs through immersion in day-to-day interactions with users and stakeholders over time.

Writing software requirements

Expecting developers to magic something from a vague idea or a one-sentence user story isn’t a particularly helpful approach. Technical staff are great at solving relatively well-bounded technical problems but dire at stepping into the customer’s shoes.

Unfortunately, many software requirements are just too vague and open to interpretation. Select a few requirements from your team to inspect. Are they clear and effective? Do they convey the outcome required? Please tell me what you find.

What’s usually missing is all the context. Here are some helpful things to consider including alongside each software requirement, mindful not to slip into ‘solution mode’ when doing so:

  • Background information
  • Acceptance criteria
  • Alternative scenarios
  • Business rules
  • Error handling
  • Illustrations, UI mockups and designs
  • Research data and user feedback
  • Related product features
  • Clear directions stating what technical approaches are acceptable
  • Assumptions, constraints
  • Anything else relevant

Performance attributes and non-functional requirements (eg. accessibility, interoperability, redundancy) should also be defined. It is usually best practice to document these centrally for an overall product or product sub-system, referencing them from individual requirements. Technical details such as API endpoints and message schemas should only be included if they are significant or enhance readability.

Read more: Painless Functional Specifications

Reviewing software requirements

Software requirements aren’t something that get written and handed over to the technical team, never to be discussed again. Rather, they are actively ‘developed’ over time in a process best seen as an ongoing conversation between users, developers and testers (nb. often a business analyst or product owner is the proxy for the user in this arrangement).

The technical team should be involved fairly early in reviewing the emerging software requirements and consulting on the feasibility of building such things. Their involvement helps form an opinion on the classic ‘iron triangle’, ie. time, cost and features. Management are usually very happy for an informed view, and any estimates can be revisited as the software requirements become better known.

Software requirements are best reviewed as a team in frequent, regularly scheduled review sessions. Once or twice a week usually works well for a team of up to 5 developers. The idea is to ensure each software requirement has been reviewed before development commences so that significant clarifications have been bottomed out. Incomplete, ambiguous and poorly written requirements should be identified quickly and marked as unsuitable for development.

Practically, here’s how to conduct a good review meeting. Open each software requirement and have someone read it aloud to the team. Each person should use their professional expertise to assess what they hear critically. Is the requirement clear and unambiguous? Are business rules, edge cases and third-party dependencies known? Do we know enough to build and test it? If not, are we still happy for a developer to commence working on it? These are all good questions to ask. Developing a standard set of review questions together as a team is often a helpful exercise.

Unfortunately, some developers can quickly become bored when asked to review requirements that have little relevance to their specialised skillset. The solution is to triage all new software requirements as a team, identifying what technical skills will likely be required to build each requirement. Reviews are then performed in smaller breakout sessions with only the relevant parties. Sometimes, these reviews are performed ‘offline’ or asynchronously through ticketing systems and collaboration platforms. Either way, ensure that testers are always involved, irrespective of how reviews are conducted or the technical skills required by the software requirement.

Often, the same software requirement goes through several rounds of reviews, successively clarifying and elaborating it each time as more becomes known. The result isn’t a detailed specification, unless you are manufacturing an aeroplane engine; rather you are gathering together enough information for each requirement so that developers can resolve the remaining details themselves and work without becoming blocked. Better software requirements are best described as ‘developed’ rather than ‘written’.

Read more: Card, Conversation, Confirmation

Implementing software requirements

Developers prefer to solve technical problems rather than hash out incomplete or poorly formed requirements. They want to apply the breadth of their technical knowledge to design appropriate solutions and learn new skills along the way.

Developers also enjoy working in a “flow state” for extended periods of time, focusing on solving the problem at hand. Every software developer has experienced being so immersed in their work that hours literally flew by without being noticed. This is where the real enjoyment of software development lies.

Picking up some work to find the underlying need hasn’t been fully articulated is annoying, although often easily addressed with a quick conversation. Continuing work to hit another blocker that can’t be as easily fixed is more irritating, requiring emails and perhaps more conversations. Sometimes, the work becomes completely blocked as the need for further analysis emerges. Seeking further clarification is required.

While some developers can perform the analyst role, they really shouldn’t have to. Instead, developers are most effective when they can resolve the right amount of detail upfront or early in their work, before proceeding with the software development relatively unhindered. Developers thrive when they can determine their own work schedule.

Working in a focused manner is not a ‘perk’ of being a software developer; instead, it’s an inherent part of working effectively on complex problems in complex domains. One unscheduled phone call or interruption, even just the threat of that, can be enough to break concentration. Being able to focus is a reasonable expectation for any knowledge worker.

The measure of good software requirements is whether developers can schedule their own day and work in a focused manner without unexpected blockers or the need for unplanned conversations. Ceremonies, appropriate conversations and alignment of work should only happen because they are valuable to the software developer, rather than imposed by a framework or overbearing agile coach.

Special considerations

User Interfaces

Not all software has a user interface, but those that do require special consideration as to who designs it, when it’s designed, and to what level of detail. Choosing the right approach to user interface design is particularly important given that many software products span desktop, mobile, and tablet devices, each with different layouts and interaction patterns. There are two broad approaches to user interface design and development: developer-produced and UX designer-produced, and each will be discussed.

Developer produced interfaces

Software developers are solely responsible for designing and implementing the user interface, either because they are experts in UI development, the software product isn’t complicated, or the team isn’t big enough for specialised developers. The product backlog is populated with user stories and developers build them as best they can, including any necessary user interface. Wireframes and sketches may accompany each story, more as a guide than anything else. Alternative flows and unhappy paths are usually not considered ahead of time.

Developers collaborate directly with the user story author (product owner, business analyst, onsite customer etc) as the software takes shape, making sure the acceptance criteria are met and the user interface performs acceptably. Standard UI component libraries and frameworks are used to accelerate frontend development and ensure consistency of look and feel. Over time, a library of standard UI layouts and interaction patterns specific to the product is developed.

Read more: DEWA88

UX designer produced interfaces

UX designers (sometimes called product designers) become the ‘voice of the customer’ and are responsible for designing entire, end-to-end user experiences. Gone are the grey screens, comboboxes and modal popups some developers typically favour. User research and focus group sessions, conducted periodically, inform the product roadmap and which features to design next. Wireframes and high-fidelity UX collateral become the defacto requirements developers work from and your typical ‘as a, I want, so that’ user story no longer appears in the development backlog.

Developers build entire end-to-end journeys and the technical team is included in the design process, at least at key points, to assess the feasibility of each design, including the required backend capabilities. Finished journeys are ‘cut up’ into frontend and backend stories and added to the backlog for developers to refine and build. Close coordination of development work ensures backend systems are ready for integration with frontend screens at the right time. Direct user feedback follows each new product launch, often infrequently every three to six months.

Read more: How agile product teams can adopt a UX centric approach

Remote development

Developers struggle when the people who can answer questions are not readily accessible or available. Remote developers struggle more due to their lack of co-location and increased reliance on written communication. Difficulty getting answers in real-time, waiting on emails, and language barriers are all things a local development team would never face.

Collaborating effectively across time zones is possible, but shifting to asynchronous communication and mostly written forms of communication is required. Remote teams working similar hours to the local business can more readily act as an extension of the local workforce, whereas other offshore teams perform best when responsible for end-to-end work, avoiding part-finished work bouncing between distributed teams.

Regardless of the approach, the remote team must work with you rather than against you. Accommodate their needs as best as possible and avoid the temptation to impose local processes without assessing their suitability first. Meet up in real life, at least once, to put names to faces and start developing some human relationships.

Neurodiverse developers

‘Can’t you just collaborate for once’ said the frustrated manager, perhaps not an uncommon experience for the developer it was directed at. But maybe they really could not collaborate, at least not in the way expected of them. Noisy office space, unstructured meetings, verbal instructions, business jargon and lack of written documentation might have been factors at play.

Around 1 in 6 people working in technology are believed to be autistic, a much higher rate than the general population (Tony Atwood, 2022). Neurodivergent traits can be incredibly valuable for technology workers, hence the increased prevalence; however, these individuals see, hear and process the world differently, sometimes drastically so. ADHD and other conditions often co-exist, adding to an already complex situation. Difficulties can arise, both interpersonally and at work, such that autism is a recognised disability under UK law.

Professional understanding, diagnosis and access to support are relatively recent occurrences, not widespread but certainly more prevalent than they were 20 years ago. Many neurodivergent individuals have lived their entire adult life flying under the radar, undiagnosed, unsupported and coping as best they can. Can you really be sure your ‘difficult developer’ is wilfully acting this way? You can’t.

Certainty, structure and routine are everyday needs amongst neurodiverse individuals, coping mechanisms for high levels of hidden anxiety. By their very nature, software requirements reduce ambiguity and clarify what’s required. Good software requirements do this to a level sufficient for the developer to work without worry. A welcome relief for many individuals.

Read more: The lost years of a young software engineer

Agile transformations

Some organisations don’t want agile. They say they do, but shop floor behaviour indicates otherwise. Unless the CEO personally spearheaded agile ways of working; small, autonomous, cross-functional product teams who release software frequently and practice continuous improvement are aspirational at best.

Agile remains one small initiative in the product or technology department, ready to be overridden, deprioritised or ignored when something more important comes along. This is the reality of building software in large enterprises, particularly if they aren’t in the business of technology.

Developers find this incredibly confusing, particularly when an agile coach espouses individual empowerment and bottom-up transformation despite an organisation that acts otherwise. The internal conflict of trying to work one way versus being inadvertently prevented from doing so can cause irritability, tiredness, fatigue, burnout and poor mental health. High levels of staff sickness and turnover are good indicators of this.

Software requirements are the antidote to confusion, ambiguity and undue personalisation. Slice the requirements thin, call them user stories, practice story refinement, get a solid definition of ready, adopt a rigid Scrum cadence and ask the agile coach to protect your boundaries.

Conclusion

Software development works best when developers interact directly with users. Conversations replace the need for requirements, and ideas and concepts are quickly pushed out the door. However, the further you stray from close collaboration between users and developers, the greater the likelihood of things going wrong.

Large corporations develop software in ways that definitively break the link between users and developers, leaving a communication gap of monumental proportions. Developers who work without direct, personal user feedback are heavily dependent on written requirements as their primary source of guidance. By all means, bring users and developers closer together if you can, but when you can’t, you need software requirements.

Software requirements are a messy and complicated process, but they are also very much a human affair. This handbook presents practical advice based on many years of personal experience that your team should consider adopting. You can share your experiences of Better Software Requirements directly with the author, who will be happy to respond.

Get in touch: info@bettersoftware.uk

About Frank

Frank Ray

I’m Frank, a former Microsoft C# .Net software developer and owner of Better Software UK, a consultancy specialising in software requirements for agile development teams.

Remote, outsourced and offshore development is difficult to do well, and my time in enterprise software development teams, firstly as a hands-on developer and then as a business analyst, clearly showed me this.

I’m the author of Better Software Requirements and I have dedicated my professional life to improving software development. I discovered that I am autistic in my mid-40s and learned that it‚Äôs not uncommon for engineers. It’s also why I understand and communicate with other developers so well.