Grafana has developed really fast in the past few years. It has been the de facto standard for observability. I have been using grafana and loki for a while and I am really impressed by its performance and the quality of the code. Grafana builds most its backend components in Go. Meanwhile I am also using Go to build stuff. So it seems a good idea to learn some good practice from Grafana’s codebase. Then I found this https://github.com/grafana/grafana/blob/main/contribute/README.md which is a really good guide for building Go applications. I will put some of the points I found useful on coding style, database structure, error handling, etc.

Style guide

Three references are recommended:

Services practice

link

some takeaways:

  • Grafana’s backend services encapsulate and expose application logic through related operations using Wire for dependency injection.
  • Background services which implement registry.BackgroundService interface can be regestered and run in background.
  • Disabled services can be controlled using the registry.CanBeDisabled interface, allowing the service to determine if it should run.
  • Methods of a service should have context.Context as their first argument, especially if they interact with the bus, other services, or the database.

Databases

link

  • Grafana uses SQLlite3 as a default database while also supports MySQL and Postgres.
  • Yes it uses ORM called xorm however between service and orm there is another layer called ‘sqlstore’ this is for keeping service layer dedicated to business logic and ORM layer dedicated to database operations.
  • Use self developed migration tool to manage database schema changes. It seems a natual choice in go world to use some kind of migration scripts/tool to manage database schema changes.

Error handling

Grafana defined their owen [errorutil] (github.com/grafana/grafana/pkg/util/errutil.Error) to handle errors. They key idea is to ensure the error is structured and can be handled in a consistent way.

  • The custom error type called errutil.Error is used to handle errors effectively.errutil.Error provides a structured way to manage errors by carrying information such as error details, localization metadata, log levels, and HTTP status codes.
  • Error messages are organized into message categories to maximize public message structures and minimize specific error details for security reasons.
  • You can set a static message sent to the client when the error occurs or use dynamic messages and more options.
  • You can specify an error source to describe the origin of the error, allowing operators to distinguish between Grafana and non-Grafana errors.

Testing

For testing with go , the recommendation is use standard test library and use testify for assertions. Integration tests are seprated from unit tests for the benefits of CI experience. If you need to mock some dependencies, mockery

As grafana is a web application using go as backend and react as frontend, It is quite important to run e2e tests. For this grafana uses a forked version of cypress to run e2e tests.

Instrumenting

Instrumenting is about logs and metrics.

Logs are crucial for recording events, warnings, and errors, and they should be structured using lowercase names and context. Metrics play a key role in providing real-time insights into application performance, and they should follow naming conventions, especially for high cardinality issues.

Tracing with OpenTelemetry is recommended for tracking application requests and identifying issues within an application.

Wrap up

There are quite a few things such as commenting, dependency management, and all the frontend stuff I didn’t cover which you can find in this folder contribute. I’m grateful to the grafana team for sharing their experience on how to build a large scale application with go. On the other hand, after all, it’s the beauty of open source community.