Communication, Communication and Communication
When we examine two developers writing a software application in their
spare time we see little that we would call a formal process. However, when
we examine a project with hundreds of developers distributed across multiple
locations working to develop a large software system, process often seems
to be all we can see. Both examples do have process, but the first
is much simpler and very informal. It is most likely maintained wholly
within the minds and interactions of the two developers who, over time,
have learnt to communicate very effectively with each other. In larger
projects, the processes tend to be both much more visible and much more
formal, although you will still find many small ‘processes’ in a large organization
that are hidden, ‘understood’, or part of the tribal knowledge and not recorded
anywhere. In fact the processes (not necessarily productive processes)
that last the longest are those that become habit; ‘it’s just the way we
do things here’. One goal for a process designer is to make their process
straightforward to learn and remember so that following it becomes habit
as quickly as possible.
So what are the fundamental differences between two developers writing
software in a garage and thousand-person, multi-million dollar software
development efforts?
Those who work in the real estate industry tell us that the three
most important aspects of real estate are location, location and location.
The software development process equivalent is communication, communication
and communication. Communication is taking place constantly within a software
development process at every level. In fact, no process (work) can
occur without it! In failed projects, communication, or failure of
it at some level is usually a major contributor to the project's downfall.
If we consider developers as nodes in a communication network all
potentially linked to each other by communications channels, then there
is only one channel between our two developers in their garage.
However, as we add more developers, the number of potential communications
channels grows geometrically.
Between four developers there are six potential communication links.
Between ten there are forty-five potential links and there are 4950 potential
communication links between a hundred individuals on a team.
If not managed, there will either be too much time spent communicating
and nothing will get done or there will be too little communication and
results will not integrate and work together.
As a team grows larger, managing communication becomes increasingly
difficult.
Language
Communication requires language. Within a development team different
sub-teams use different languages. Some of these languages are textual,
others are graphical and others are more mathematical in nature. There are
languages for expressing requirements, defining interfaces, defining database
tables, for communicating analysis and design models, for describing algorithms,
for describing patterns, and a myriad of different programming languages for
communicating instructions to a computer. Much of the work of a development
team involves translating from one language into another.
… and then of course, to add to the mix, there are communication
channels and languages required to communicate with customers, users and
sponsors of the project.
Each time we translate from one language to another there is a potential
to lose information or communicate the wrong information. Errors can occur
when converting statements made during verbal interviews to structured requirements
and diagrammatic models, adding documentation to the models, developing
persistent storage models, and even creating the code. If information is
not accurately transferred at each of these ‘interfaces’, we have the potential
for building the wrong system. Therefore, we want to keep the number of
translations of information to a minimum and to automate as many of the
translations as possible to reduce the chance information is not lost.
To further complicate matters, at each step along the way, the people
involved in transferring the information may not know what they need to
communicate. For example, the user defining the system may not know
what it is that they really need. They may end up describing task steps
or symptoms of a larger problem without actually communicating the needed
functionality. So not only can we make errors in translating the information
from one form or medium to another, we may not even be translating the right
information! Therefore a good process uses numerous feedback loops to provide
validation and verification as often and as early as possible.
Another big problem preventing communication is the fear of being
wrong. If a user, manager, analyst, or developer is so scared of being
seen to have made a mistake that they withhold information, clear communication
is obviously compromised and the error is often compounded over time. It
is right that we should have accountability within a project but our software
development processes should encourage and support an environment of trust
and responsibility rather than one of suspicion and fear of recrimination.
A software development process describes what to communicate, when
to communicate, how to communicate and to whom to communicate (most process
descriptions will not tell you why; this is normally left for someone to
write a book about!).
Complexity
There is a limit to the amount of complexity mere humans can hold within
their heads. As the size of a software system grows, the complexity of that
software system grows ‘at least as fast as the square of the size of the
program’ [Weinberg] and quickly outstrips the relatively fixed capacity of
a human brain. Gerald Weinberg calls this law of software development the
size/complexity dynamic.
The best strategy we have for dealing with such large problems is
decomposition. We break the problem down into smaller and smaller problems
until we have reduced the size of the problems to something we can manage.
We then go about solving each small problem, integrating the solutions
to form the solution for the bigger problem.
Given enough time our two friends in their garage could produce
very large and complex software systems by solving one small problem after
another. Adding more people to the team allows us to solve many of the small
problems in parallel and deliver large systems quicker. The more people we
add, the more we can do in parallel.
Here's the catch: the more people we have working in parallel, the
more we are likely to bump into communication and coordination problems.
If there are significant dependencies between the problems being solved,
then we have to work hard at putting the right communication channels in
place so everyone has the information they need at the right time.
So we are back to managing communication again.
Quality
According to Philip Crosby’s definition [Crosby], quality can be simply
defined as conformance to requirements; how often the software behaves as
required. The question as, Gerald Weinberg [Weinberg], points out is whose
requirements?
We can talk of quality in software development having both an external
and internal face. A user talking about the quality of system talks about
the user interface, the response time, the reliability, and the ease of
use of the system. A developer talking about quality talks about elegance
of design, ease of maintenance and enhancement, compliance to standards, patterns
and conventions.
These two faces are of course related. Low internal quality makes
it much, much harder to maintain high external quality. This is especially
true for incremental or iterative development where after the first iteration
or increment we are building on what we built before.
Recognizing when a design for an iteration is good enough is a very
valuable skill. If we make the design too simple we add to the re-work that
we must do in later iterations when we need to extend that functionality
and when time pressure may be more intense. If we select a more general and
complex design we could be spending time building in flexibility that will
never be used.
It often seems that developers have a natural complexity level;
some developers always seem to make their designs too complex and other always
seem to oversimplify. Recognizing these tendencies in themselves helps developer
make the appropriate trade-off decisions, as does a good knowledge of the
problem domain. Whatever the trade-off decisions, it is important that they
are made visible to others on the team – that communication word again.
The Quality Spectrum
Simply splitting quality into external and internal views is really
too simple a model; users and developers are not the only people with opinions
about a system's quality. A software manager, will look at quality in terms
of ease of maintenance and enhancement, compliance to standards and conventions,
ability to deliver it on time
A project sponsor will look at how the well the system meets their
business requirements. Does it allow them to meet a constantly changing
business environment and be proactive in meeting the challenges that are
ever present in the marketplace?
Add to this the views of testers, technical writers, technical support
personnel, and so on … and we need to look at quality is as a spectrum,
with internal quality at one end, and external quality at the other.
Developers and software managers view the system more in terms of internal
quality. System users and business sponsors tend to view the system
more in terms of external quality.
Building in Quality
A naïve view associates quality with testing of code and testing
with a separate test team that goes to work once coding is complete. This
model has severe problems.
It is a reminiscent of waterfall thinking. A waterfall approach
to software development says do all the analysis first. Then do all the design.
Then do all the coding. Finally test the system.
One of the most documented problems with a waterfall approach over
the last couple of decades is that mistakes made early in the process are
often not found until very late in the process. The cost in finding an
error later rather than sooner varies from study to study but is always
much higher.
“Observe that it is perhaps 100 times as costly to make
a change to the requirements during system testing, as it is to make the
change during requirements definition.” --Richard Fairley, Software Engineering
Concepts, McGraw Hill, 1985
One way to ease this problem is to split the project into iterations
or increments so that the distance in time between analysis and test is reduced.
If we test something earlier we will find the problem earlier and the cost
of fixing it is reduced.
A complimentary approach is to use practices that increase quality
throughout the four activities. In other words, broaden the concept of
quality so that it is more than just the testing of running code. Inspections
are one such technique. Audits and metrics tools are another.
People typically want to do ‘quality’ work. Each individual
developer has his or her own idea about the acceptable level of quality
of a system and that is usually approximately the best quality that they
have achieved in the past.
It is better to ask those developers who have a lower idea of quality
to reach for a higher standard than to ask those with higher standards to
reduce their ideas of acceptable quality. Our self-esteem is linked to the
quality of what we produce. If we are consistently forced to produce what
we consider to be low quality work, we will lose self-esteem, low morale,
and as a result, lower productivity. On the other hand, if recognized as
such, asking developers to produce a higher quality product than they would
naturally do actually enhances their self-esteem … if they can do it.
Even if an organization standardizes on a level of acceptable
internal quality, it may well be lower than that of the individual developers’
ideas of acceptable quality. So at the beginning of a project, the development
team needs to agree on what is an acceptable level of internal quality and
what is not. It, obviously, cannot be lower than that of the organization
(except in unusual circumstances) but may be set higher. This level of quality
is made public in published design and coding standards and enforced primarily
by some sort of design and code inspection (automated source code audits
and source code formatters help enforce code naming and layout standards).
In iterative, object oriented software development where we want
to reuse results, we need to ensure internal quality is built in early
so that we have quality to build on later. If we allow low quality results
at the start we will find ourselves in a vicious cycle of low quality resulting
in more low quality.
A word of caution: Be careful not to confuse quality with optimization.
Great results can be made from combinations of sub-optimal parts. Optimizing
a small part may make no significant difference to the whole. An over
complicated, bug-ridden part, or and incorrect implementation of a requirement,
on the other hand, can be the cause of many problems in other parts.
Process Pain and Relief
In January 2000, a number of people including Peter Coad and Jeff De
Luca were sitting in a room discussing the purpose of a software development
process. The result of that discussion was a statement similar to the following:
“The purpose of a software development process is
to enable and enforce the repeatable delivery of working software in a timely
and cost effective manner supplying accurate and meaningful information
to all key roles inside and outside a project with the minimum of disruption
to developers.”
When a process emphasizes only enforcement, it stifles creativity and
innovation and causes frustration for the developers who feel their ideas
are considered of no worth. On the other side of the coin, when a process
emphasizes only enabling, creativity and innovation are allowed to reign
unbridled causing frustration for managers and team leaders because everyone
is doing their own thing and results do not integrate. This is a
constant balancing act. Any process should be monitored and reviewed for
effectiveness and value, and should be modified when necessary to maintain
its value to the organization.
Similarly, having hundreds of pages of steps to execute in an
over specified process, demoralizes the team members, to the point that
they switch off the minds and blindly follow the steps. At the other extreme,
process descriptions that are too vague, abstract or general cause each
one of us to make up our own way of doing things and work harder than necessary
to achieve the desired results.
The main mechanism for communication within the development
team and between a development team and its client is often large documents.
In a process that emphasizes the production of documents, care must be taken
that the focus is not put on recording results when it should be on getting
the results right; that emphasis is put on achieving quality results instead
of quality formatting of results.
If the overhead of producing progress and status reports is
too high it will not be done well and the information they contain is more
likely to be inaccurate or irrelevant. Without accurate and meaningful information,
managers either have no useful feedback loop from the software process or
even worse, late or inaccurate feedback. Managers cannot take appropriate
action if they do not know what is actually happening on a project. Worse
still, they may take inappropriate action that increases rather than reduces
the pain.
Summary
A good process is well bounded; it provides enough structure
to guide innovation and creativity, constraining it to appropriate times
and places.
A good process clearly defines tasks. The tasks
are focused on results without specifying minutia so that we progress efficiently
and yet still have the freedom to adapt to unusual events or changes in
circumstances.
A good process produces accurate progress and status information
for team leaders, project managers, upper management and the client,
while minimizing the impact on the developer's time.
A good process quickly becomes a matter of habit rather
than a set of conscious acts. Developers do not want to have to consult
a 3000-page process manual every time they reach a new step in an activity.
A good process helps a team maintain quality and manage complexity
.
A good process optimizes communication within and outside
the team.
[Crosby] Philip B. Crosby, Quality is Free, The Art of Making Quality
Certain, Penguin Books U.S.A 1980
[Fairley] Richard Fairley, Software Engineering Concepts, McGraw Hill,
1985
[Weinberg] Weinberg, Quality Software Management vols. 1-4, Dorset House
1992-97
©2001 Giraffe Productions
