Green Field Project

A long while ago, I discovered the Conventional Commits specification. It’s a simple convention on how to write commit messages. It’s simple, but it’s powerful. It allows us to generate changelogs, version numbers, etc. automatically.

With a defined convention, it is easier for every on in the team to understand what a commit is about. It also makes it easier to generate release notes.

A typical commit message would look like this:

The type can be one of the following:

In the footer, I would add a reference to the JIRA ticket (or any other ticket system) that the commit is related to.

Going one step further, I think the type should also be the prefix for the branch name. Followed by the ticket number, and finally, some words about the feature or problem. This way, we can easily see what the branch is about.

I would use semantic versioning to version the project. It’s a simple convention that allows us to know what kind of changes are in a version just by looking at the version number.

From the semver website:

On the subject of version, they are just numbers; we should not hesitate to increment them, they cost nothing. And we should not try to keep all parts of a project in sync with the version number. It’s ok to have a version 1.0.0 of a library and a version 2.0.0 of the application that uses it.

But, when we deploy, we need keep track of the versions of the different parts of the project. This way, we can easily see what is deployed where.

We need to make sure we take i18n (internalization) into account from the start. We will not simply place string of character for buttons, menus, descriptions, etc. We will use a library that is appropriate to the selected frameworks (frontend and backend).

It is much easier to put in place from the start, even with only one language, that to retro-fit once started.

Also, if we store the information in the backend, like configuration, we shall return all the languages from queries and let the frontend pick the one needed. This is especially true when writing apis.

Most projects are going to need dates at some point or other. We will make sure that the communication between services and between the frontend and backend uses ISO8601 date format from the start.

Also, dates are hard, just google it or take a look at Falsehoods programmers believe about time. So, we should be smart and use libraries to manipulate time and dates.

It will save us from pain in the long run.

Security should not be an afterthought. We should have it in mind as we start the project. We should take the time to define permissions and groups, to determine which endpoints should be secured, which need authentication and authorization and which should be public.

We should also be using the security features of the selected framework, not only for access, but to avoid sql injections, sessions takeover, etc. OWASP Top Ten is a good starting point.

I’ve recently been introduced to the concept of Diataxis (https://dev.to/onepoint/documentation-chaotique-diataxis-a-la-rescousse—​3e9o).

It is a way to categorize and organize the documentation of a project.

It can be seen as a matrix with two axises: the content and the form.

I have not yet used this concept, but I think it is a good way to organize the documentation.

There exists many ways and format to document our future project. Quite often, we will see markdown as a format. Unfortunately, markdown is more limited, and there is a variety of competing flavors for markdown.

So, we should use Asciidoc as the format. It’s a powerful format that can be used to create documentation. It can be used to generate documentation in many formats, like html, pdf, etc. Documentation can be for different outputs, like books, articles, etc.

If we ever need to convert it back to markdown, we can use the following command:

Antora is the single or multi-repository documentation site generator for tech writers who love writing in AsciiDoc.

Antora allows you to write asciidoctor documentation in multiple code repository, and to set up a centralizing project where you can gather the documentation from all your repositories. You can then publish it as a static website for your organisation.

It is a very interesting way to make sure you have a good starting point for all your up-to-date documentation.

From a project start, we make architectural decisions. This article suggests some of them. As time goes by, the people may change projects and the memory of those decisions and why they were taken get lost.

Architectural Decision Records are a way to record them and keep them in a single place.

A few projects exist to facilitate the creation of ADR, but most use markdown. I’m still looking for a good project that would support asciidoctor. For now, adr-j seems a good candidate that supports both markdown and asciidoctor.

Claming to be The world’s fastest framework for building websites, Hugo is a framework that takes a set of markdown or asciidoctor documents and converts them into a static website with theming and nice features.

I’ve started using it with GitHub actions to generate my blog, and I’m happy with it.

I love IntelliJ IDEA by jetbrains. I’ve been using it for a long time (since december 2012).

But in fact, each person should use any IDE they like, on one condition: They should master it. They should know how to use it to its full potential.

If we have junior person in our team, make sure they take time to learn their IDE.

In many projects, we will need some helper services. I would use docker-compose to define and bundle the helper services for the developers. And wrap the actions in a shell script that offer some help and sane default.

This way, we can start the helper services with a single command. We can also stop the helper services with a single command. We can also restart the helper services with a single command.

In our projects, the helper script understands profiles. So a front end developer would start helper services like the database and the backend, while a backend developer would start the database and the front end. And a QA would start everything.

We can also add any other helper service that can be dockerized.

And of course, all the projects, modules or microservices that are part of the project.

The simple reality is pick one, anyone and stick to it.

But, from experience, I would add some other criteria to select it:

As soon as there are (or could be) more than one person working on a project, we will need a way to manage our work, note the tasks that need to be done, etc.

We should use the ticket system that is already in place at the organisation where the project is started. If there is none, many options are available.

When we are building an API, we will need to return error messages. It is nice if we can predefine the format of the error messages and be consistent across all the apis we expose, even if only internally.

I would use the Problem Details for HTTP APIs (RFC 9457) to return error messages. It’s a simple convention that can be used to return error messages. It can be used to return error messages in many formats, like json, xml, etc. It can be used to return error messages in many languages, like java, JavaScript, etc.

One feature to notice is that we can make it so the errors in the logs have a unique UUID that is also returned to the client. This way, We can trace the error in the logs and in the client.

Here is a longer post by A java geek that explains https://blog.frankel.ch/problem-details-http-apis/

There is an implementation ready for Quarkus: https://github.com/quarkiverse/quarkus-resteasy-problem

Communication is key in a project. Either for a quick question, to share a snippet of code, to ask for help, etc. We need a chat system.

Here again, I would use the chat system that is already in place at the organisation where the project is started. If there is none, many options like MS Teams, Slack, etc. are available.

Just make sure we create dedicated channels for different aspects (code review, deployments/devops, fun) of the project. This way, we can keep the conversation focused.

I would identify in the code base examples of good code. This way, when a new developer joins the team, they can see what is considered good code. It can be a simple class, a method, a pattern, etc.

From the beginning, we should have unit tests in place. They are the first line of defense against bugs. They are also a good way to document the code. Start with the unit tests, and then add integration tests when needed.

We don’t have to test libraries. We should test our code, the code that we write.

Code should be tested before it is merged. We should have a pipeline that runs the tests and blocks the merge if the tests fail.

Static analysis is a good way to catch bugs before they happen.

Your IDE is your first line of defence, keep an eye on the warnings it gives you.

Ideally, we would link our IDE to a more robust tool like Sonarqube that can check the code with same configuration as the pipeline for us. It can be done as you code, or, minimally, before the code is committed.

Another way to increase quality code is to review it. It allows us to catch bugs, but also to share knowledge.

Even if the team is small, it is a good practice to have a code review. We should have our pipeline blocks if code is not reviewed.

We should have a build pipeline that runs the tests, the static analysis, and make sure there was code review, etc. It will catch errors that don’t happen on our machine and make sure the build is more robust.

I would use Quarkus as the framework to build the backend the project. It’s a modern Java framework that is pretty mature. It looks like it was built from the start with the developer in mind. And it can create artifacts that are native and fast and tailored for containers.

There is an excellent tutorial to give us an overview of the framework and the associated features. https://quarkus.io/quarkus-workshops/super-heroes/

Quite often, when building a robust backend, we will need different but corresponding models (DTO, pojo, entities) for different parts of the application.

As the information moves from one part of the application to another (from the database to the service, from the service to the controller, from the controller to the client), we will need to map the information from one model to another.

I would use Mapstruct. It’s a powerful product that can be used to map objects from one type to another. The mapping is done at compile time, so it’s fast.

It is pretty useful if we have to map from a DTO to an entity and back. It can match properties by name, or we can define the mapping ourselves. We can also easily define custom transformation methods.

One of the complaints people have over java is writing lots of boilerplate code.

I would use Lombok to alleviate this. It’s a powerful product that can be used to generate the boilerplate code for us. It can be used to generate the boilerplate code for us in many ways, like getters, setters, constructors, including some patterns like builders, equals and hashcode, etc.

For some constructs, using Java Records could be a good alternative.

At some point, we will probably need a relational database to store our data (See Postgresql later on). And then, we will need a way to manage the schema of that database.

I would use Liquibase for that. It’s a mature product that can be used to manage the schema of the database. It can be used to create the schema, update the schema, etc. It can also be used to create some data in the database.

It also supports the concept of contexts. So we can store in the same system different change sets (example data for dev or qa) for different environments, needs or features. This is a powerful feature.

There is even some support for some non-relational/sql databases, like MongoDB, Noe4j, Databricks Data Lakehouses, etc.

Monitoring our application is often a task that is pushed into the future after the features are implemented. But it’s important to start thinking about it early.

I would use OpenTelemetry to monitor the application. It’s a modern framework that can be used to monitor the application. It can be used to monitor the application in production, but also in development. It can be used to monitor the application in a container, but also in a native environment. Many libraries implement the open telemetry specification, so we can use it to monitor the application in many languages.

And we can add our own metrics as well. Let’s say we want to monitor the number of times a specific feature is used. We can add a metric for that. Or if we want to make sure a cron job is completed properly at the expected rate, we can add a metric for that.

An example from the quarkus documentation:

What if I told you " you can put everything into feature flags"?

As soon as our core application exists, we should consider wrapping every feature with a feature flag.

There are the two main reasons for that:

We can also use feature flags to turn off a feature if it’s not working as expected.

If our project needs a relational database, I would use Postgresql. It’s a mature product that can be used to store the data of the project. It’s a powerful product that has many features like transactions, constraints, triggers, etc. It has many built in capabilities, like storing objects in json format, full text search, etc. It also has many extensions, like Postgis, that can be used to store and query geospatial data, Timescale, that can be used to store and query time series data, etc. It is very stable, adheres to standards and has a large community. It is well documented and available on most cloud providers

At some point, we will need to manage users and their access to the application. I would use Keycloak for that. It’s a mature product that can be used to manage users, roles, permissions, etc. We can also set it up to defer the authentication to an external system by using identity providers. There is even a way to migrate our users from an external system to Keycloak.

It is quite possible that our project will have to interact with external services. We will want to test our code without having to rely on actually calling these external services. We can use the service documentation to get the payload format.

I would use Wiremock to replace the services during development. It’s a mature product that can simulate the external services. We can define the responses we want to get from the external services and use Wiremock to simulate the external services.

It even supports randomizing the result or returning timestamps that are always a set period in the past or the future of the call.

We have passwords, too many of them. And we should not store them in clear text.

I would use a password manager to store the passwords. There are many password managers available, like 1Password, LastPass, Bitwarden, etc.

Some, like 1Password, are more than just a password vault, they come with some tools that allow us to securely use the passwords in our applications or on the command line.

Since we are ultimately talking about writing code as a team, we need way to manage our code. I would choose Git as the version control system. Then, we would need a place to store that code. The usual suspects are GitHub, Gitlab, Bitbucket, etc.

I’d be pragmatic and chose whatever is already used at the organisation where the project is started. As long as we can also have pipelines to check, build and package the code, I’m good.

At some point, for sure, we will have to manage secrets in our repository. I would use Sops to encrypt these secrets. This way, I can store them in the git repository without fear that they will be read by people who should not have access.

Make sure we include this early in the process, so that no secrets are ever stored in clear text in our repo.

More info on how to set this up here: https://blog.gitguardian.com/a-comprehensive-guide-to-sops/

Some organisations use Gitlab, other use Github, Bitbucket or even AWS CodeCommit. Whatever your organisation is using, make sure you have a pipeline that can check, build and package the code.

As we are using Gitlab, we will be using the pipelines that can run in gitlab. It’s a powerful tool that can be used to check, build and package the code. It can be used to deploy the code. It can be used to monitor the code. It can be used to roll back the code.

Here are some typical steps that we put in our pipelines:

From the danger website:

We should use Danger to enforce the conventions we set with the team.

We will want to check the quality of our code. Static analysis of our code allows us to catch many bad habits, bugs or security problems.

I would use Sonarqube for that. It’s a mature product that can check our code for bugs, vulnerabilities, code smells, etc. It can also check our code for coverage, duplications, etc.

Most IDEs should have a plugin that allows visualization of the analysis directly in our IDE or before commiting.

I think it is a good guess to think that we will deploy our application in containers. Even more so if our application is not a big monolith, but a set of modules or microservices. Think about doing a front end in React, a backend in Quarkus, a database in Postgresql, etc.

We can use Docker to create the images of our application. We can use Docker to run the containers of our application. And, if the need arises, we can use Kubernetes to deploy our entire application stack.

So, early in the project, we should make sure we have a pipeline that can build the images of our application. We will need to take into consideration the necessary steps to build the images, and what configuration we need to pass to the images. And we will test both the pipeline and the resulting image.

Ideally, we should have a pipeline that builds the images, and push it to a container repository. This way, we can use the same image in all our environments.

I think that making different images for different environments is a bad idea. We should be able to deploy the same image in all our environments. The only difference should be the configuration.

We’ll save ourselves a lot of pain and stress if we start early with this instead of waiting to do it when we are near the User Acceptance Test or worse, the Production date.

We are going to deploy our application into some kind of infrastructure. And we will most probably need the same infrastructure in different environments, like development, qa, staging, uat, production. The best way to make sure each environment is as close as possible to the previous one is to make it reproducible. I would use Terraform to define the infrastructure as code. This way, we can deploy the same infrastructure in each environment.

Another advantage of using Terraform is that we can bring together and synchronize parts of the infrastructure that are in different cloud providers. Let’s say we use GitHub for our code repository, use Amazon Pipelines for our build pipelines, and want to configure Keycloak and Grafana, we can put all that into Terraform states.

This is, I think, easier than using the proprietary configuration of each cloud provider.

Terragrunt is a thin wrapper for Terraform that provides extra tools for keeping your configurations DRY, working with multiple Terraform modules, and managing remote state.

Managing a big infrastructure with Terraform is a bit painful. We probably have one or more of the following, a big state file on AWS S3 bucket, a lot of modules and many environments. Terragrunt can help us manage all that.

I consider this a subset of monitoring. We will probably want to know if, when and where our users are using our application. I would use Plausible for that. It’s a simple product that can be used to monitor our application. It can be used to monitor our application in production, but also in development, in a container, or in a native environment.