Golang: Use viper with local config, management with struct, and envrionment variables

Viper is a complete configuration solution for Go applications. It is designed to work within an application, and can handle all types of configuration needs and formats. Also, viper has many features, such as support for different configuration formats (JSON, TOML, YAML, etc.), live watching and re-reading of config files, reading from environment variables, etc.

Here will demostract how to use viper with local config, management with struct, and envrionment variables.

Here is the local yaml config file for the application, the config are mysql database connection parameters:

config/config.yml

mysql:
  host: 127.0.0.1
  port: 3306
  username: default
  password: deafult
  database: database_name

Reading Local Configuration Files

First, we use InitConfig() to set the initial values for Viper, specifying the location of the local config we want to read:

utils/config.go

package utils

import (
	viper "github.com/spf13/viper"
)
func InitConfig() error {

	//set config file as default
	viper.SetConfigName("config")
	viper.SetConfigType("yaml")
	viper.AddConfigPath("config/")
    ...

Next, we can use viper.GetString to directly specify the nested structure key to get the corresponding value:

utils/database.go

package utils

import (
	"fmt"
	"sync"
	viper "github.com/spf13/viper"
	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

var DB *gorm.DB
var o sync.Once
var db *gorm.DB

func InitDB() (*gorm.DB, error) {

	host := viper.GetString("mysql.localhost")
	port := viper.GetString("mysql.port")
	username := viper.GetString("mysql.username")
	password := viper.GetString("mysql.password")
	database := viper.GetString("mysql.database")
	var err error
	dsn := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s?charset=utf8mb4&parseTime=True&loc=Local", username, password, host, port, database)
	...
}

main.go

package main

	"go_api_example/utils"
)

func init() {
	err := utils.InitConfig()
	if err != nil {
		panic("init config error:" + err.Error())
	}
}

func main() {
    Db, err := utils.InitDB()
...
}

Managing Through Structures

Originally, we read directly from the local file. To make the subsequent architectural design more flexible, we adjusted to read our local value through the InfoDbHost struct. The original function was adjusted to the InfoDbHost Method function. We use viper.GetStringMap to get the MySQL values and convert them into a map structure. Then, we use mapstructure.Decode from mapstructure to convert the map into a struct structure. This allows us to get the values using the struct method.

package utils

import (
	"fmt"
	"sync"

	"github.com/go-redis/redis"
	"github.com/mitchellh/mapstructure"
	viper "github.com/spf13/viper"
	"gorm.io/driver/mysql"
	"gorm.io/gorm"
)

type InfoDbHost struct {
	Host     string
	Port     any
	Username string
	Password string
	Database string
}

func (infoDb InfoDbHost) InitDB() (*gorm.DB, error) {
    var err error
    o.Do(func() {
		err = mapstructure.Decode(viper.GetStringMap("mysql"), &infoDb)
		dsn := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s?charset=utf8mb4&parseTime=True&loc=Local", infoDb.Username, infoDb.Password, infoDb.Host, infoDb.Port, infoDb.Database)
		fmt.Println("Init DB once")
        ...
    }
    ...

In the original place where the InitDB() function was directly used, it has been adjusted to specify var InfoDb utils.InfoDb and call the InfoDb.InitDB() method:

main.go

package main

	"go_api_example/utils"
)

func init() {
	err := utils.InitConfig()
	if err != nil {
		panic("init config error:" + err.Error())
	}
}

var InfoDb utils.InfoDb

func main() {
    Db, err := InfoDb.InitDB()
...
}

Using Local as Default, Automatically Apply Environment Variables

In microservice design, environment variables are often used as default values to make CI/CD design more flexible. There are many packages in Go for setting and managing environment variables, but here we’ll only introduce Viper.

In this example, you need to first set up some environment variables. You can use export or Go’s os package to do this, for example:

	// Set environment variables
	os.Setenv("MYSQL_USERNAME", "root")
	os.Setenv("MYSQL_PASSWORD", "root")
	os.Setenv("MYSQL_DATABASE", "database")
	os.Setenv("MYSQL_PORT", "3306")
	os.Setenv("MYSQL_HOST", "localhost")

First, we define a Config structure that includes Mysql. This will serve as our management of environment variables. We use viper.AutomaticEnv() to allow Viper to automatically read environment variables. To align our environment variables with the nested structure, we use _ as a separator in environment variable names. We then use viper.SetEnvKeyReplacer(strings.NewReplacer(".", “_")) to replace the separators (Viper automatically reads all environment variables and the keys will be automatically uppercased). Finally, viper.ReadInConfig() allows environment variables that match the naming format to replace the local config value.

For example: MYSQL_USERNAME will replace mysql.username.

utils/config.go

package utils

import (
	"fmt"
	"os"
	"strings"
	viper "github.com/spf13/viper"
)

type Mysql struct {
	Host     string
	Port     string
	Username string
	Password string
	Database string
}
t
type Config struct {
	Mysql Mysql
}

func InitConfig() error {
	//set config file as default
	viper.SetConfigName("config")
	viper.SetConfigType("yaml")
	viper.AddConfigPath("config/")

	// viper auto read all env variables, the key will auto uppercase
	viper.AutomaticEnv()
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

	err := viper.ReadInConfig()
	if err != nil {
		panic(fmt.Errorf("fatal error config file: %w", err))
	}
	return err
}

Here, we adjust the InfoDb struct to use the Mysql struct, which makes it easier to access and use the values in Viper.

The viper.Unmarshal function in Go’s Viper library is used to unmarshal the configuration data into a struct. This means it takes the configuration data that Viper has loaded and decodes it into a struct

In this example, Viper.Unmarshal decodes the configuration data into the Config struct. The Port and Host fields in the struct map to the port and host keys in the configuration data.

utils/database.go

package utils

var DB *gorm.DB
var o sync.Once
var db *gorm.DB

type InfoDb struct {
	Mysql Mysql
}

func (infoDb InfoDb) InitDB() (*gorm.DB, error) {
	var err error
	o.Do(func() {
		if err := viper.Unmarshal(&infoDb); err != nil {
			panic(fmt.Errorf("unable to decode into struct, %v", err))
		}
		dsn := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s?charset=utf8mb4&parseTime=True&loc=Local", infoDb.Mysql.Username, infoDb.Mysql.Password, infoDb.Mysql.Host, infoDb.Mysql.Port, infoDb.Mysql.Database)
		...
	})
	return db, err
}

Treatment the environment variable with prefix

Sometimes we need to add a different prefix to environment variables for local development or multiple tenant usage.

For example, with a prefix MYP for all variables like:

	// Set environment variables
	os.Setenv("MYSQL_USERNAME", "username")
	os.Setenv("MYSQL_PASSWORD", "password")
	os.Setenv("MYSQL_DATABASE", "database")
	os.Setenv("MYSQL_PORT", "3306")
	os.Setenv("MYSQL_HOST", "localhost")

Here we can use the viper SetEnvPrefix to specify our environment variable prefix:

package utils

import (
	viper "github.com/spf13/viper"
)
func InitConfig() error {

	//set config file as default
	viper.SetConfigName("config")
	viper.SetConfigType("yaml")
	viper.AddConfigPath("config/")

	// viper auto read all env variables, the key will auto uppercase
	viper.AutomaticEnv()

	//Set prefix of env variables
	viper.SetEnvPrefix("MYP")

	//Replace the environment variables _ to .
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

	err := viper.ReadInConfig()
	if err != nil {
		panic(fmt.Errorf("fatal error config file: %w", err))
	}
	return err

Conculsion

In conclusion, here we provides a comprehensive guide on how to use the Viper library in Go for configuration management. It covers how to read local configuration files, manage configurations through structures, and use local configurations as defaults while automatically applying environment variables.

By following this guide, you can effectively manage configurations in their Go applications, making their code more maintainable and their applications more flexible and robust.