Not long ago I published a small repository on GitHub called Red Entities, a simple but efficient query object-mapper currently compatible with SQLite and several MySQL flavors.
I have picked it up again to validate it with the latest version of Node and also because I want to add PostgreSQL and SQL Server compatibility, especially now that I am finishing documenting Mantra Framework for release.
And the curious (and interesting) thing is that, despite having developed this software project myself, it has taken me a little time to take it on again. Although it is well supported by tests and I would even say the design is simple but robust and efficient, and it is not a large library by any means.
It took me some time to remember how the dependency injection mechanism worked, how I had prepared everything to include new connectors for other types of databases and similar things.
Although it is documented at the API level, with enough information for anyone to use it in a Node project, this is one of the major problems we face in software development, especially in projects that are more or less complex and large, and it gets even worse when the design is poor and there is a lot of technical debt accumulated: reading the code, however well structured and developed it may be, does not allow you to easily understand the overall vision of the entire project.
To understand a software project well, code is not enough.
I am not talking about understanding what a class does, or a specific method inside it, or even a small library, but all the information on which the project was developed, with which decisions were made and which is not clearly reflected in the code, since the form and structure of the code are a consequence of the former.
One might think that everything is in the code: classes, modules, configuration files. However, there is much more information that is neither in it nor can be in it and which is essential to understand why the project was built in that way and for what purpose, especially when the moment comes when other developers, different from the originals, have to take it over.
This, precisely, has been one of the worst professional experiences I remember: having to resume projects carried out by others (who were no longer even in the organization for which I was working). I remember one in Java and another in PHP, with thousands of lines of code, several years of development and evolution, a great deal of technical debt, and no kind of support documentation: the code and that’s it.
Weeks of work to understand not only what had been done at that point, but why, without the possibility of asking the authors, just you, the project, and a lot of imagination.
Resuming work done by others is often a real nightmare in software, which, at best, can take a considerable amount of time just to understand the project well enough to modify and evolve it.
But once you think you understand it and can add changes or make modifications, how? With what criteria, consistent with what is already there? Is there any unknown reason why the former developers did something in a specific way that you would have done differently?
What usually happens is this: a new developer arrives to take over a project they do not know and whose authors are no longer there; they strive to understand what is there, how and why, but, as I said, they are missing a lot of information. And if they manage to get the project running in the end, they will do it their own way, probably adding a new design style to the previous one, modifying things without knowing why they were done in that way, perhaps modifying the original design (or adding a new one), violating architectural criteria (without knowing it), thereby increasing the lack of coherence along with other problems.
Multiply this by several different hands and we have the perfect recipe for a good bowl of spaghetti code.
In other words, a very common situation in our sector.
I wonder: can it be avoided? Especially, can it be avoided without needing expensive and exhaustive documentation?
To understand a software project well, it is not enough for the code to be readable, with good test coverage, well organized, with simple classes or methods, without redundancies, clean, and without technical debt. All these things are already known and I talk about them a lot in my books, especially in The Practical Guide to the Agile Programmer.
Especially in large and complex projects, you may find yourself asking questions such as the following:
- What is the architecture of the system?
- What criterion has been used to organize the project assets? Is that criterion coherent throughout the project?
- What is the design intention in this or that area?
- How is a development and testing environment created?
- What workflow is necessary to run all the tests (if they exist, of course)?
- Is there version control? And how is it followed?
- How have certain non-functional requirements such as scalability, constraints, security, etc. been resolved?
- Where is the configuration located?
- Are the libraries up to date?
- Is the project compatible with updating the libraries it depends on, or is there an exception?
- Why is that piece of code commented out?!
- Is there a class diagram somewhere with its relationships? If so, is it up to date?
- Why was this library used instead of that more popular one? Was there a reason for it?
- What usability criteria were followed for the user interface?
- How is the project compiled or started? What dependencies are there?
- If the project is divided into layers or tiers, where are they located in the codebase?
- How does the project scale in a real deployment?
- If there are databases, is the model sufficient to understand their design?
And a very long etcetera. If you look closely, many questions that developers working on a project from the beginning or for a long time know well, but for a newcomer that important information does not exist unless someone tells them. Of course, for someone who has been working on that project for a year, all of the above is trivial; they have it all in their head, lacking only documentation, without realizing the number of decisions, assumptions, and unwritten norms they work with every day, beyond the nature of the code itself.
This difficulty often increases when the project has been developed by the same people for years and they have not cared to simplify it or, even less, document anything, because the knowledge of the project is stored so deeply in the team’s heads that it becomes very difficult for someone new to take it over without an enormous additional effort.
One of my favorite mantras in software says something like this:
“We always have to program so that others understand what we have done and why.”
But I add something more: not only at the code level, but also at the level of the project as a whole.
Without going too far, I have recently seen firsthand the difficulty of taking over a low-level project by new people in a team. It can become so difficult that, at times, it is almost impossible to assume it, to the point that the need to rebuild it from scratch is even considered, if the company can afford it, of course.
It is true that the Agile Manifesto says in one of its points:
“Working software over comprehensive documentation”
But this has been misinterpreted: it does not mean that nothing is documented, but that it is more important to prioritize that the software works than to create excessively exhaustive documentation. But from there to zero documentation… this made sense in an era when those heavy methodologies forced you to present genuine volumes with hundreds of pages of UML diagrams, etc., which, by the way, nobody read.
Is there then a middle path between not documenting anything and over-documenting?
How should a software project be documented correctly so that the above does not happen? In other words, how can the complete story be told without falling into the trap of documenting too much and unnecessarily, and without making it an excessive cost to the project?
Sometimes it is believed that the only useful things to document are the architecture (if someone has taken the trouble to define it or there is an architect role, of course), the APIs, and little else. As I said, all that is necessary but not sufficient.
Below I propose a series of documents that I believe are a minimum for anyone to resume a software project with much less difficulty and fewer headaches (note: only a few pages each at most):
- Context: indicates what the project is about, what clients it is for, etc.
- Functional view: gives a broad overview of the project’s most important features.
- Quality attributes: explains how the system’s non-functional requirements have been resolved (scalability, performance, i18n, l10n, monitoring and maintenance, etc.).
- Constraints: are there constraints? Which ones? How have they been addressed?
- Deployment: explains how to deploy the project in a new environment.
- Data models.
- Principles: what principles have been followed in development? For example, is there an ORM, is it divided into decoupled components? What is the logging/tracing policy or exception management policy? Is SOLID used? How is it implemented in the codebase?
- Configuration: what are the configuration parameters and where are they located?
- Decision log: indicates which design or technology decisions have been most relevant.
- Operations and support: how is the system operated and how is support provided?
- User guide.
- Development and test environment: explains how to create a new development and testing environment from scratch.
- And, of course, the architecture document.
Depending on the type of project, some of this information may not be necessary, but if it exists, it would certainly shed a lot of light on how and why it was implemented that way. I would venture to say that including all of the above for any project can stay within a document of only a few dozen pages, not more.
The learning curve for someone who lands in a project that is already underway is a cost and a risk that managers should take into account. The greater the technical debt and the greater the documentation deficit, the greater the cost and the greater the chance that new incorporations will add even more technical debt.
Now, precisely at the moment I am writing this, I am closing a production version of Mantra Framework, and I realize that including the documentation above in a section that makes sense for the project is, in reality, a relatively simple and inexpensive thing to do. Moreover, by documenting, I have discovered details that I have improved and several inconsistencies that I have also resolved. In other words, to some extent, documenting helps you improve the project.
But we already know: developers always try to leave documentation to other people…
Now seriously: a professional project cannot be considered finished if it does not include at least the minimum documentation described above, which costs much less than we imagine.