This chapter is about how you can start a new project, organize a source tree, and manage the packages you need to develop your programs. A well designed project structure is important because when other developers work on your project or try to use components from it, they can quickly and easily find what they are looking for. This chapter will first answer some of the questions you may have when you are starting a new project. Then, we will look at how you can use the Go package system, work with standard library and third-party packages, and make it easy for other developers to useyour packages.

This chapter includes thefollowing recipes:

• Creatinga module
• Creating asource tree
• Building andrunning programs
• Importingthird-party packages
• Importing specific versionsof packages
• Using internal packages to reduceAPI surface
• Using a local copy ofa module
• Workspaces
• Managing the versions ofyour module

First, a few words about modules and packages would be helpful. Apackage is a cohesive unit of data types, constants, variables, and functions. You build and test packages, not individual files or modules. When you build a package, the build system collects and also builds all dependent packages. If the package name ismain, building it will result in an executable. You can run themain package without producing a binary (more specifically, the Go build system first builds the package, produces the binary in a temporary location, and runs it). To use another package, you import it. Modules help with organizing multiple packages and the resolution of package references within a project. Amodule issimply a collection of packages. If you import a package into your program, the module containing that package will be added togo.mod, and a checksum of the contents of that module will be added togo.sum. Modules also help you to manage versions ofyour programs.

All files of a package are stored under a single directory on the filesystem. Every package has a name declared using thepackage directive, shared by all source files in it. The package name usually matches the directory name containing the files, but this is not necessarily so. For example, themain package is not usually under a directory namedmain/. The directory of the package determines the package’s “import path.” You import another package into your current package using theimport <importPath> statement. Once you import a package, you use the names declared in that package using its package name (which is not necessarily thedirectory name).

A module name points to the location where the module contents are stored in a version control system on the Internet. At the time of writing, this is not a hard-and-fast requirement, so you can actually create module names that do not follow this convention. This should be avoided to prevent potential future incompatibilities with the build system. Your module names should be part of the import paths for the packages of those modules. In particular, module names whose first component (the part before the first/) does not have. are reserved for thestandard library.

These concepts are illustrated inFigure 1.1.

Figure 1.1 – Modules and packages

1. The module name declared ingo.mod is the repository path where the module canbe found.
2. The import path inmain.go defines where the imported package can be found. The Go build system will locate the package using this import path, and then it will locate the module containing thepackage by scanning the parent directories of the package path. Once the module is found, it will be downloaded to themodule cache.
3. The package name defined in the imported module is the package name you use to access the symbols of that package. This can be different from the last component of the import path. In our example, the package name isexample, but the import path for this packageisgithub.com/bserdar/go-recipes-module.
4. TheExample function is located in theexample package.
5. Theexample package also imports another package contained in the same module. The build system will identify this package to be part of the same module and resolve the references, using the downloaded version ofthe module.

You will need a recent version of Go on your computer to build and run the examples in this chapter. The examples in this book were tested usingGo version 1.22. The code from this chapter can be foundathttps://github.com/PacktPublishing/Go-Recipes-for-Developers/tree/main/src/chp1.

When you start working on a new project, the first thing to do is to create a module for it. A module is how Gomanages dependencies.

How to do it...

1. Create a directory to store anew module.
2. Under that directory, usego mod init <moduleName> to create the new module. Thego.mod file marks the root directory of a module. Any package under this directory will be a part of this module unless that directory also has ago.mod file. Although such nested modules are supported by the build system, there is not much to be gainedfrom them.
3. To import a package in the same module, usemoduleName/packagePath. WhenmoduleName is the same as the location of the module on the internet, there are no ambiguities about what you arereferring to.
4. For the packages under a module, the root of the module is the closest parent directory containing ago.mod file. All references to other packages within a module will be looked up in the directory tree under themodule root.
5. Start by creating a directory to store the project files. Your current directory can be anywhere on the filesystem. I have seen people use a common directory to store their work, such as$HOME/projects (or\user\myUser\projects in Windows). You may choose to use a directory structure that looks like the module name, such as$HOME/github.com/mycompany/mymodule (or\user\myUser\github.com\mycompany\mymodule in Windows). Depending on your operating system, you may find a moresuitable location.

Warning

Do not work under thesrc/ directory of your Go installation. That is the source code for the Gostandard library.

Tip

You should not have an environment variable,GOPATH; if you have to keep it, do not work under it. This variable was used by an older mode of operation (Go version <1.13) that is now deprecated in favor of the Gomodule system.

Throughoutthis chapter, we will be using a simple program that displays a form in a web browser and stores the entered information ina database.

After creating the module directory, usego mod init. The following commands will create awebform directory underprojects and initialize a Gomodule there:

$ cd projects$ mkdir webform$ go mod init github.com/examplecompany/webform

This will create ago.mod file in this directory that lookslike this:

module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webformgo 1.21.0

Use a name that describes where your module can be found. Always use a URL structure such as the<host>.<domain>/location/to/module format (e.g.,github.com/bserdar/jsonom). In particular, the first component of the module name should have a dot (.) (the Go build systemchecks this).

So, even though you can name the module something such aswebform ormywork/webform, donot do so. However, you can use something such asworkspace.local/webform. When in doubt, use the coderepository name.

Once you have a new module, it is time to decide how you are going to organize thesource files.

How to do it...

There are several established conventions, depending onthe project:

• Use a standard layout, suchashttps://github.com/golang-standards/project-layout.
• A library with a narrow focus can put all the exported names at the module root, with implementation details optionally stored under internal packages. A module that produces a single executable with relatively few or no reusable components can also use the flatdirectory structure.

For a project like ours that produces an executable, the structure laid out inhttps://github.com/golang-standards/project-layout fits. So, let’s followthat template:

webform/  go.mod  cmd/    webform/      main.go  web/    static/  pkg/    ...  internal/    ...  build/    ci/    package/  configs/

Here, thecmd/webform directory will contain themain package. As you can see, this is one instance where the package name does not match the directory it is in. The Go build system will create executables using the directory name, so when you build themain package undercmd/webform, you get an executable namedwebform. If you have multiple executables built within a single module, you can accommodate them by creating a separatemain package under a directory matching the program name, under thecmd/ directory.

Thepkg/ directory will contain the exported packages of the program. These are packages that can be imported and reused inother projects.

If you have packages that are not usable outside this project, you should put them under theinternal/ directory. The Go build system recognizes this directory and does not allow you to import packages underinternal/ from other packages that are outside the directory containing theinternal/ directory. With this setup, all the packages of ourwebform program will have access to the packages underinternal/, but it will be inaccessible to packages importingthis module.

Theweb/ directory will contain any web-related assets. In this example, we will have aweb/static directory containing static web pages. You can also addweb/templates to store server-side templates if youhave any.

Thebuild/package directory should have packaging scripts and configuration for cloud, container, packaging systems (dep,rpm,pkg, etc.).

Thebuild/ci directory should have continuous integration tool scripts and configurations. If the continuous integration tool you are using requires its files to be in a certain directory other than this, you can create symbolic links, or simply put those files where the tool needs them insteadof/build/ci.

Theconfigs/ directory should contain the configuration file templates anddefault configurations.

You can also see projects that have themain package under the module root, eliminating thecmd/ directory. This is a common layout when the module has onlyone executable:

webform/  go.mod  go.sum  main.go  internal/    ...  pkg/    ...

Then there are modules without anymain package. These are usually libraries that you can import into your projects. For example,https://github.com/google/uuid contains the popular UUID implementation using a flatdirectory structure.

Now thatyou have a module and a source tree with some Go files, you can build or runyour program.

How to do it...

• Usego build to build thecurrent package
• Usego build ./path/to/package to build the package in thegiven directory
• Usego build <moduleName> to builda module
• Usego run to run the currentmain package
• Usego run ./path/to/main/package to build and run themain package in thegiven directory
• Usego run <moduleName/mainpkg> to build and run the module’s main under thegiven directory

Let’s write themain function that starts an HTTP server. The following snippetiscmd/webform/main.go:

package mainimport (    "net/http")func main() {    server := http.Server{        Addr:    ":8181",        Handler: http.FileServer(http.Dir("web/static")),    }    server.ListenAndServe()}

Currently,main only imports the standard library’snet/http package. It starts a server that serves the files under theweb/static directory. Note that for this to work, you have to run the program from themodule root:

$ go run ./cmd/webform

Always run themain package; avoidgo run main.go. This will runmain.go, excluding any other files in the main package. It will fail if you have other.go files that contain helper functions in themain package.

If you run this program from another directory, it will fail to find theweb/static directory; because itis arelative path, it is resolved relative to thecurrent directory.

When yourun a program viago run, the program executable is placed in a temporary directory. To build the executable, usethe following:

$ go build ./cmd/webform

This will create a binary in the current directory. The name of the binary will be determined by the last segment of the main package – in this case,webform. To build a binary with a different name, usethe following:

$ go build -o wform ./cmd/webform

This will build a binarycalledwform.

Most projects willdepend on third-party libraries that must be imported into them. The Go module system managesthese dependencies.

How to do it...

1. Find the import path of the package you need to use inyour project.
2. Add the necessary imports to the source files you use in theexternal package.
3. Use thego get orgo mod tidy command to add the module togo.mod andgo.sum. If the module was not downloaded before, this step will also downloadthe module.

Tip

You can usehttps://pkg.go.dev to discover packages. It is also the place to publish documentation for the Go projectsyou publish.

Let’s add a database to our program from the previous section so that we can store the data submitted by the web form. For this exercise, we will use theSQLite database.

Changethecmd/webform/main.go file to import the database package and add the necessary databaseinitialization code:

package mainimport (    "net/http"    "database/sql"    _ "modernc.org/sqlite"    "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/    webform/pkg/commentdb")func main() {    db, err := sql.Open("sqlite", "webform.db")    if err != nil {        panic(err)    }    commentdb.InitDB(db)    server := http.Server{        Addr:    ":8181",        Handler: http.FileServer(http.Dir("web/static")),    }    server.ListenAndServe()}

The_ "modernc.org/sqlite" line imports theSQLite driver into the project. The underscore is theblank identifier, meaning that thesqlite package is not directly used by this file and is only included for its side effects. Without the blank identifier, the compiler would complain that the import was not used. In this case, themodernc.org/sqlite package is a database driver, and when you import it, itsinit() functions will register the required driver with thestandard library.

The nextdeclaration imports thecommentdb package from our module. Note that the complete module name is used to import the package. The build system will recognize the prefix of this import declaration as the current module name, and it will translate it to a local filesystem reference, which, in this case,iswebform/pkg/commentdb.

On thedb, err := sql.Open("sqlite", "webform.db") line, we use thedatabase/sql package function,Open, to start aSQLite database instance.sqlite names the database driver, which was registered by the imported_ "modernc.org/sqlite".

Thecommentdb.InitDB(db) statement will call a function from thecommentdbpackage .

Now, let’s see whatcommentdb.InitDB looks like. This is thewebform/pkg/commentdb/initdb.go file:

package commentdbimport (    "context"    "database/sql")const createStmt=`create table if not exists comments (email TEXT,comment TEXT)`func InitDB(conn *sql.DB) {    _, err := conn.ExecContext(context.Background(), createStmt)    if err != nil {        panic(err)    }}

As you can see, this function creates the database tables if they have not beencreated yet.

Note the capitalization ofInitDB. If the first letter of a symbol name declared in a package is a capital letter, that symbol is accessible from other packages (i.e., it isexported). If not, the symbol can only be used within the package it is declared (i.e., it isnot exported). ThecreateStmt constant is not exported and will be invisible toother packages.

Let’s buildthe program:

$ go build ./cmd/webform  cmd/webform/main.go:7:2: no required module provides package modernc.org/sqlite; to add it:      go get modernc.org/sqlite

You can rungo get modernc.org/sqlite to add a module to your project. Alternatively, you can runthe following:

$ go get

That will get all the missing modules. Alternatively, you can runthe following:

$ go mod tidy

go mod tidy will download all missing packages, updatego.mod andgo.sum with updated dependencies, and remove references to any unused modules.go get will only downloadmissing modules.

Sometimes, you need a specific version of a third-party package because of API incompatibilities or a particular behavior youdepend on.

How to do it...

• To get a specific version of a package, specify theversion label:

  $ go get modernc.org/sqlite@v1.26.0

• To get the latest release of a specific major version of a package,use this:

  $ go get gopkg.in/yaml.v3

  Alternatively,use this:

  $ go get github.com/ory/dockertest/v3

• To import the latest available version,use this:

  $ go get modernc.org/sqlite

• You can also specify a different branch. The following will get a module from thedevel branch, if thereis one:

  $ go get modernc.org/sqlite@devel

• Alternatively, you can get aspecific commit:

  $ go get modernc.org/sqlite@a8c3eea199bc8fdc39391d5d261eaa3577566050

As you can see, you can get a specific revision of a module using the@revision convention:

$ go get modernc.org/sqlite@v1.26.0

The revision part of the URL is evaluated by the version control system, which, in this case, isgit, so any validgit revision syntax canbe used.

Tip:

You can find which revision control systems are supported by checking out thesrc/cmd/go/alldocs.go file under yourGo installation.

That alsomeans you canuse branches:

$ go get modernc.org/sqlite@master

Tip

Thehttps://gopkg.in service translates version numbers to URLs compatible with the Go build system. Refer to the instructions on that website on how touse it.

Themodule cache is a directory where the Go build system stores downloaded module files. This section describes how to work with themodule cache.

How to do it...

The module cache is, by default, under$GOPATH/pkg/mod, which is$HOME/go/pkg/mod whenGOPATH isnot set:

• By default, the Go build system creates read-only files under the module cache to preventaccidental modifications.
• To verify that the module cache is not modified and reflects the original versions of modules,use this:

  go mod verify

• To clean up the module cache,use this:

  go clean -modcache

The authoritative source for information about the module cache is the Go ModulesReference (https://go.dev/ref/mod)

Notevery piece of code is reusable. Having a smaller API surface makes it easier for others to adapt and use your code. So, you should not export APIs that are specific toyour program.

How to do it...

Createinternal packages to hide implementation details from other packages. Anything under aninternal package can only be imported from the packages under the package containing thatinternal package – that is, anything undermyproject/internal can only be imported from the packagesundermyproject.

In our example, we placed the database access code into a package where it can be accessed by other programs. However, it does not make sense to expose the HTTP routes to others, as they are specific to this program. So, we will put them under thewebform/internal package.

This is theinternal/routes/routes.go file:

package routesimport (    "database/sql"    "github.com/gorilla/mux"    "net/http")func Build(router *mux.Router, conn *sql.DB) {    router.Path("/form").        Methods("GET").HandlerFunc(func(w http.ResponseWriter, r         *http.Request) {        http.ServeFile(w, r, "web/static/form.html")    })    router.Path("/form").        Methods("POST").HandlerFunc(func(w http.ResponseWriter, r         *http.Request) {        handlePost(conn, w, r)    })}func handlePost(conn *sql.DB, w http.ResponseWriter, r *http.Request) {    email := r.PostFormValue("email")    comment := r.PostFormValue("comment")    _, err := conn.ExecContext(r.Context(), "insert into comments     (email,comment) values (?,?)",    email, comment)    if err != nil {        http.Error(w, err.Error(), http.StatusInternalServerError)        return    }    http.Redirect(w, r, "/form", http.StatusFound)}

Then, we change themain.go file to use theinternal package:

package mainimport (    "database/sql"    "net/http"    "github.com/gorilla/mux"    _ "modernc.org/sqlite"    "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/    webform/internal/routes"    "github.com/PacktPublishing/Go-Recipes-for-Developers/src/chp1/    webform/pkg/commentdb")func main() {    db, err := sql.Open("sqlite", "webform.db")    if err != nil {        panic(err)    }    commentdb.InitDB(db)    r := mux.NewRouter()    routes.Build(r, db)    server := http.Server{        Addr:    ":8181",        Handler: r,    }    server.ListenAndServe()}

Sometimes, youwill work on multiple modules, or you download a module from a repository, make some changes to it, and then want to use the changed version instead of the version available onthe repository.

How to do it...

Use thereplace directive ingo.mod to point to the local directory containinga module.

Let’s return to our example – suppose you want to make some changes to thesqlite package:

1. Clone it:

   $ ls  webform$ git clone git@gitlab.com:cznic/sqlite.git$ ls  sqlite  webform

2. Modify thego.mod file under your project to point to the local copy of the module.go.mod becomesthe following:

   module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webformgo 1.22.1replace modernc.org/sqlite => ../sqliterequire (    github.com/gorilla/mux v1.8.1    modernc.org/sqlite v1.27.0)...

3. Youcan now make changes in thesqlite module on your system, and those changes will be built intoyour application.

Sometimes you need to work with multiple interdependent modules. A convenient way to do this is by defining a workspace. A workspace is simply a set of modules. If one of the modules within a workspace refers to a package in another module in the same workspace, it is resolved locally instead of that module being downloaded overthe network.

How to do it...

1. Tocreate a workspace, you have to have a parent directory containing all yourwork modules:

   $ cd ~/projects$ mkdir ws$ cd ws

2. Then, start a workspaceusing this:

   $ go work init

   This will create ago.work file inthis directory.

3. Place the module you are working on intothis directory.

   Let’sdemonstrate this using our example. Let’s say we have the followingdirectory structure:

   $HOME/  projects/    ws/       go.work       webform       sqlite

   Now, we want to add the two modules,webform andsqlite, to the workspace. To do that,use this:

   $ go work use ./webform$ go work use ./sqlite

   These commands will add the two modules to your workspace. Anysqlite reference from thewebform module will now be resolved to use the local copy ofthe module.

Gotooling uses the semantic versioning system. This means that the version numbers are of theX.Y.z form, broken downas follows:

• X is incremented for major releases that are not necessarilybackward compatible.
• Y is incremented for minor releases that are incrementalbut backward-compatible
• z is incremented forbackward-compatible patches

You can learn more about semantic versioningathttps://semver.org.

How to do it...

• To publish a patch or minor version, tag the branch containing your changes with the newversion number:

  $ git tag v1.0.0$ git push origin v1.0.0

• If you want to publish a new release that has an incompatible API with the previous releases, you should increment the major versions of that module. To release a new major version of your module, use anew branch:

  $ git checkout -b v2

  Then, change your module name ingo.mod to end with/v2, and update all references in the source tree to use the/v2 version ofthe module.

For example, let’s say you released the first version of thewebform module,v1.0.0. Then, you decided you would like to add new API endpoints. This would not be a breaking change, so you simply increment the minor version number –v1.1.0. But then it turns out some of the APIs you added were causing problems, so you removed them. Now, that is a breaking change, so you should publishv2.0.0 with it. How can youdo that?

The answer is, you use a new branch in the version control system. Create thev2 branch:

$ git checkout -b v2

Then, changego.mod to reflect thenew version:

module github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform/v2go 1.22.1require (  ...)

If there are multiple packages in the module, you have to update the source tree so that any references to packages within that module also use thev2 version.

Commit and push thenew branch:

$ git add go.mod$ git commit -m "New version"$ git push origin v2

To use the new version, you now have to import thev2 version ofthe packages:

import "github.com/PacktPublishing/Go-Recipes-for-Developers/chapter1/webform/v2/pkg/commentdb"

This chapter focused on the concepts and mechanics of setting up and managing Go projects. It is by no means an exhaustive reference, but the recipes presented here should give you the basics of using the Go buildsystem effectively.

The definitive guide for Go modules is the Go ModulesReference (https://go.dev/ref/mod).

Check out theManaging dependencies link (https://go.dev/doc/modules/managing-dependencies) for a detailed discussion ondependency management.

In the next chapter, we will start working withtextual data.