I have published a new GitHub repository for a library extracted from all the work I have done so far on Hub de Libros.
It is Red Entities, an object mapper and query builder that makes access to data entities transparent, with some particularities that encouraged me a year ago to build it instead of using other similar projects. For now, it is compatible with several flavors of MySQL and SQLite and tested with Node.js 10, 12, 13, and 14.
Although I have used GitHub a lot and have several repositories published, with Red Entities I have spent a lot of time on the documentation and on the lifecycle needed to publish a project like this and make it available as an NPM package as well.
As I explain in the documentation, there is a design intention behind Red Entities related to the conception of highly scalable projects: projects that are going to grow a lot in every sense and that, in addition, we have no idea where they are heading.
In software there is a principle we do not always take into account: a bounded project, clear, with very clear requirements, which starts and ends and then that is it… can be built with certain design and architecture and the lifecycle and methodology you choose. A not-clear project, MVP, unpredictable requirements, changing business rules… must be approached with a totally different design, conception, and architecture, along with a more open and iterative agile methodology. Subtle to grasp, but with deep consequences in the software generated one way or another.
And this affects, of course, how data persists, how it is used, and how databases evolve: as business rules and functionality evolve, logically, database structures (whatever they are) will also change. To avoid this becoming a nightmare, we need to apply a looser approach, with its drawbacks, of course, but also with the advantages I mention in this post.
Digression…
I know some may find what I am about to say strange, because it goes against certain well-established paradigms even learned during our academic stage:
Relational data models are an obstacle in systems that are going to grow a lot and whose evolution we do not know at all.
If you have a closed project (you know exactly what must be done, the business logic is very well specified, and besides, it is delivered and that is it, with little more to change), then it is perfect to use related data entities as the data repository (1..n, 1..1 relationships, etc.), which impose constraints on how data is used and distributed across different tables, keeping referential integrity in place at all times and facilitating things like cascading deletion, transactions, rollbacks, etc. Even coupling the project to a specific database server technology (SQL Server, PostgreSQL, Mongo, etc.).
However, every time I have seen a project that has grown a lot with functionality unimaginable at the beginning and people wanted to maintain a strongly relational approach to data at all costs, the result has been horrific: an exaggerated number of tables, data redundancy, extreme complexity in migrations, and everything tied together with glue. I remember one project with more than a hundred tables (yes, more than one hundred) and so many lines indicating their relationships that it looked like the London Underground map. Result? Impossible to maintain; any change meant a huge headache, failures in production due to collateral effects impossible to predict, and let us not even talk about performance.
In these cases, a different approach is needed—more like thinking with a different paradigm.
We need radical componentization of the system, where functionality is distributed across small components that talk to each other, each with its own domain of work. High-level functionality is implemented through their orchestration.
By their nature, since each component is small, each one handles a reduced set of data entities over which it basically performs CRUD operations.
And nothing more, because this is the key to building large systems that evolve a lot.
This approach gives us extraordinary freedom to evolve the system (with more components, changes in data models, etc.). In Hub de Libros, the system where I have clearly used this paradigm, in addition to having more than seventy components currently, only two use three different entities (tables) in their data model, with little coupling, and in total the system has six different databases, each serving a different purpose and with a different backup policy.
Of course, what we gain on one side requires paying a price, as always: we lose the wonderful referential integrity. But the price is worth it. It must be maintained programmatically. However, after working on two projects that continue to grow with this paradigm, that is not a relevant problem given the advantages of handling small data models for small components.
Red Entities, the project I am releasing to the community for free use, allows exactly that: defining a schema (a model as a JSON object), implementing it with a single line of code (creating its data structure in MySQL or SQLite), and using an insultingly simple and fast syntax to perform common CRUD operations through what I call selectors, which in some ways reminds me of the effectiveness of jQuery.
I know there are other similar projects and much more ambitious ones, but given the importance of this library in Hub de Libros, I decided to develop it almost from scratch after having worked the previous year on a similar concept but for Redis (Blue Entities) and evolve it so that Hub de Libros would not depend on third-party solutions for such an important point. I have not found, I say, an object mapper and query builder with the characteristics needed to implement components in the way we do in Hub de Libros and in the projects I will work on soon within Mantra Framework.
Without going into details, an important point: updating one schema to another (due to a property change, a new index, or any other change) is trivial, something that currently takes a lot of effort in the life of any project.
I hope it is useful and that some people will be encouraged to use Red Entities in their own projects.