Of the 4 core values as stated by the Agile Manifesto, one value stands out for me: working software over comprehensive documentation. It’s generally believed that the traditional big requirements document is incompatible with Agile principals of welcome change, iterative development and collaborative working.
However, in practice, lots of big requirements documents are still written. I’ve been involved in Agile projects with 6 organisations, 5 of which had policies which utilised some form of documentation in addition to user stories. In some cases, the process used by the organisation included a full waterfall-style requirements document to be signed off before the project got under way and user stories were populated. I’ve found this approach in a small and relatively young company as well as 2 large conservative banks. Perhaps it’s because many business stakeholders like the waterfall process with its formal, if often illusory, sign-off steps.
A hint that many organisations still retain some formalised waterfall-type requirements documents can be found in the Confluence Product Requirements Document template. Most Confluence users are probably also Jira users and therefore using Agile. Atlassian obviously feel that it’s worthwhile to include a Product Requirements Document (PRD) template in a set of tools aimed at facilitating Agile development. So, I suspect my experience is not untypical.
A further reason is that the Agile manifesto is based on some implicit assumptions which may not be true of the particular organisation or project you are working in. “Pure” Agile suggests that a user story forms part of a conversation between the business user and the developer. This implicitly assumes that everyone is co-located whereas, in reality, it is probably more likely that the team is virtual and split across timezones.
Of course, the Agile manifesto does not prohibit formal documentation. Derwyn Harris suggests that documentation can mean A white board, sticky notes, wiki, or collaboration software, these are all documentation
So what documentation should you use on an Agile project? There are 4 main types of documentation, each with its own audience and appropriate mechansim:
- Developers and testers need to understand what the story is meant to deliver, including the UI.
This type of documentation is best delivered through attaching the relevant elements to the story, e.g. wireframes. Artefacts like these are really just extensions of the story. They don’t need to be fancy: I’ve seen photos of whiteboard sketches used successfully. It’s important to ensure that the attachments aren’t changed after the story has been taken into a sprint so that everyone knows what they are up to, so if the elements is linked to the story by means of a hyperlink then you need to make sure versioning is in place.
- Developers need to have a shared understanding of the policies, standards and frameworks which have been agreed for use
There’s an Agile principle that teams are self organising. Thus as a general rule, policies, standards and frameworks should be discussed and understood by the team. However there’s still a need to document these as often such decisions are made by a specialists working across teams. Documenting policies, standards and frameworks avoids ambiguity and means that new team members can quickly get up to speed. The most suitable format is a wiki or other collaboration tool, perhaps part of a shared repository of developer tips.
- Everyone needs to have a shared understanding of the high-level goals
The important word is “shared”. On a big project with multiple teams, the big picture can easily get lost as each team focuses on delivering its own backlog. The Product Manager and/or Product Owner(s) should share the overall goals and vision but particularly with a big, distributed team, there can be barriers to getting this message across. Documenting the high-level goals and making them available on a collaborative documentation tool makes sure everyone understands the big picture. The high-level goals can be documented in any suitable format, from a wiki page to PowerPoint slides or even a video.
- The organisation needs an accurate and complete description of the overall deliverable to include in its corporate memory
The Agile manifesto values working software over comprehensive documentation. This makes sense: a working piece of code is clearly more valuable to the business than a document describing it. However the latter is not worthless. Team members leave and new people start, teams are restructured and new supplier contracts added. In theory, it should be possible to get a good picture of what the product does from looking at the delivered user stories. In practice, stories morph and split and are often written by an insider for an insider, making it very difficult for a newcomer to figure out what has been delivered.
Some organisations have policies that mandate the approval of detailed requirements document before a project can proceed. However, there’s little value in writing a waterfall-style detailed requirements document for a project which is to be delivered in an Agile manner. Such an approach will create many issues, such as how to deal with change during the project. Agile welcomes change but how do you keep your detailed requirements document up to date? If you do not bother, the requirements document becomes increasingly useless as the project progresses and the inevitable changes are requested. The alternative – to use a change request process within an Agile project – is frankly ridiculous.
I believe there is value in writing a Product Requirements Document (PRD): a lightweight requirements document which defines the business background, scope and high level requirements. This can be written by the Product Owner and/or Product Manager. The PRD should include requirements gathered from all stakeholders but there is no need to have a formal process where all stakeholders sign-off the document. In reality, inviting the appropriate stakeholders to each post-sprint demo is a much better way of getting stakeholder feedback than forcing them to formally approve a document which is often only of marginal interest to them. (As a BA, I am pretty sure that many of the stakeholders I asked to approve a requirements document simply sent their approval without ever reading the document they “approved”…)
The list of high level requirements can be used to populate the initial epics and user stories in the backlog. The epic and story numbers can be then added to the relevant requirement in the PRD. This achieves 2 purposes:
- it ensures completeness since high level requirements without a corresponding epics or user story are highlighted
- it allows a stakeholder to determine which epics and user stories relate to the high-level requirements they are interested in: they can then look at the acceptance criteria on the user stories to ensure lower-lower requirements are captured.
Conclusion
While Agile values working software over comprehensive documentation this doesn’t mean to say that documentation is incompatible with Agile. In the real world, with distributed teams and staff coming and going, I think some form of formal documentation is essential.
What do you think? Please leave your comments below.
Deryck Fay 
Hi Deryck,
Having spent many years and projects delivering requirements in line with a waterfall methodology, the move to agile is welcome. In waterfall, the analysis/requirements phase producing bottomless documents (largely unread!) with reams of changing requirements, stagnates design and dev progression needlessly.
Having been part of a very large program though, using agile for the first time in our project delivery, I can see gaps caused by using the the supplier version of agile. Agile in pure form used in a company with in house development teams, may not experience these gaps. For example:
– Fragmented scoping of user journey/use cases not showing the E2E experience. Pre and post conditions not identified as a result in a way that a classic waterfall requirements phase/doc would do so.
– Functionality requests raised, passed off as ‘data gathering’ and not dealt with in the sprint, i.e. to be tackled separately/later as it’s own data gathering stream. This can cause confusion, unless the sprint SMEs can lead or closely interact with the data gathering stream to ensure alignment on the requirement and supporting data. I seen less issues with this when using waterfall requirements gathering.
Fragmented streams running sprints in parallel with e.g. non-functional, data gathering, reporting needs etc can result in problems for test teams, business impact assessment and training teams – especially on a big program.
It is important to be able to ensure that requirements are easily traceable in whatever methodology is used. I’ve yet to see how agile proves that all requirements have been understood and delivered.
The other challenge to overcome is where a company has some suppliers that work in an agile mode and others that work waterfall only. Where there is a project spanning multiple suppliers with different delivery methods, this needs to be carefully managed.
Again this may be the result of a large scale program. In smaller BAU projects, or companies that are fully setup to run agile, I assume there are less complications with requirements management and delivery of them.
Agile is an ever evolving machine…
LikeLike
Hi Gina
Thanks very much for taking the time to read and comment!
You first point is very valid: for most types of systems, capturing the E2E customer journey is vital. I think the best way to do this is through activity (swimlane) diagrams which are included in, or attached to, the PRD. I’ve sometimes labelled the relevant parts of the journey with references to the user stories which will address it to help the developers and more especially testers understand how the different components hang together. However on a big, complex system this isn’t always feasible. Thanks for highlighting the gap: I’ll add it into the blog post.
In regard to pre and post-conditions not being identified, I’d expect these to included as Acceptance Criteria in the stories. I think the important thing is that in an Agile project with an outsourced partner that the client has ultimate control over the Product Backlog.
I haven’t come across the issue you describe myself about Functionality requests being raised but then being passed off as to be tackled separately/later in its own stream. However if the client has ultimate control over the Product Backlog then such functionality requests are just other stories in the backlog which need to be built.
In regard to the use of Fragmented streams running sprints in parallel, pure Agile doesn’t really suggest how to handle this which is why frameworks like SAFe (https://www.scaledagileframework.com) have grown up on top of Agile. Agile purists may disagree, but I think SAFe is a good approach for a larger organisation which has been brought up on waterfall, as it provides a framework to align multiple teams, and to manage enterprise-level concepts like a product portfolio and enterprise-level architecture.
In regard to traceability, I think one of the strenghts of Agile is that the Product Owner makes the ultimate decision about whether the story meets its Acceptance Criteria and is Done and each completed story is demoed at the end of the sprint. So I’d argue that at a micro-level Agile provides good traceability. Furthermore, if Epics are used properly then the Epic is only complete when all the stories under it are marked as Done. At a macro-level, a technique I have found useful is to include a column in the table of product features within the PRD to include the Epic(s) or stories which will address the feature. This may need to be kept up to date as Epics or stories get split or replaced. I also think that in a truly Agile project, traceability is not as important as in a waterfall project. For Agile it’s important that software is built which delivers business value but not so important to deliver the requirement that Jim-who-used-to-work-in-Finance insisted be included in a DRD 2 years ago, though he’s gone and the business needs have since changed.
Having multiple suppliers who use different methodologies is a big challenge. This leads on to the bigger question of whether Agile is suitable for the project and organisation – a question which I’ll address soon!
LikeLike