Finding and Importing Packages in Go
What Else to Include in Your Repo
Recently I found myself working for a client who needed some things done in Go. With a lot of my time lately has been devoted to Ruby related projects, I quickly encountered situations in Go where I needed a more specialized tool than the basic tools provided by Go, like some of the things I was used to using with Ruby. No problem, Go has lots of community tools available to be downloaded through the Go package site. From there you can download whatever you like. You can simply search for something that you need, or a specific user that you know has a package that you want to download and the site will take you to a page with details about that package. Once you have selected the package that you want to use you simply import it into the code you are working on. This is as simple as including the following line:
import "github.com/gmr5311/jsondig"
Don’t forget that you will still need to download the package also. This is another simple step that a lot of IDEs will take care of for you. If they don’t, you just need to use the terminal command
go get github.com/gmr5311/jsondig
which will download it for you. In the above example you would be pulling a package called jsondig from a Github repo owned by gmr5311 (that’s me, if you didn’t already know that). This import command is generally included as part of the README file in the repo to help guide users that might be interested in using this tool that you created.
So far this is all very basic. Most people learn about these simple steps by referencing Go’s docs, and if you haven’t already looked through the resources there you really should. They have some fantastic tutorials and information. I find myself referencing that site all the time while working. Now that we have covered these basics, what if you want to create a tool all your own?
I find myself encountering the package creation step most often during the refactor phase of test driven development(TDD). For those of you who are not familiar with this process, the short description is that you start your work by writing a test that describes what you want to have happen. Once you have written the basics of what you are planning to do you run the test and verify that it is failing. It should be failing because at this point you haven’t fixed the bug or created the feature yet. This is the “red” phase of the process because your tests are not yet passing. Next you create the functionality needed to make the test pass. This is the “green” phase of the process. Next you evaluate if you have reached your objective. Many times I find that my test originally didn’t cover all the edge cases that I would need to consider. Sometimes I don’t even know what the final arguments will be for functions that I create. This is totally fine, the point is that you should be updating your test as you go. You really should only be creating new code as a result of a failing test. When this process is completed and your tests are passing and the objective has been reached it’s time for the final step, “refactor”. This is when you evaluate the code that you have written and try to clean it up. No one writes perfect code the first time through so you should never neglect this step. It can be quite tempting to create that pull request once the first two steps are done, but if you don’t go back and refactor you are adding code debt to the project which will hurt your project in the long run.
During the refactor step you are looking for ways to DRY out the code so you don’t have so much duplication. You also might be looking for long functions that can be broken down into smaller steps. Some of those smaller functions you might try to make a bit more generic so they can be applied later to other parts of the code. It is here that I find that I’m working on creating a new package.
In the example code that I mentioned above I found that I had a variety of different json strings being returned to me from various http requests. As I was writing my tests I created constants that represented the string that I expected to come back and verified that the response that I received matched. This worked perfectly until I changed the order of the fields of my structs. At that point the strings came back out of order so the test failed. Obviously this was a problem. The test was failing but not for the reason that it should have been. I started working on a better way to check to make sure the information that I needed was in the string that I was receiving.
The first step I took was to create a brand new sandbox project that I could experiment with. I keep one on hand all the time and just change out the code to make it easy. This also allows me to save the information or undo changes if I ever need to leave and come back. Go offers their own playground where you can also do this, but it doesn’t have the formatting tools that I like in my IDE so I only use the playground when I’m looking into smaller questions. From here I created my little sample string and started the TDD process to look for my values.
When everything was done I had a simple function that took in a string, a key, and an expected value for that key and returned true or false to represent its presence in the original string. It wasn’t much but it was a good first step. My goal with this tool was to help make my life better, but there is no reason that it can’t help other people too, right? To that end I decided to publish it and make it open source for anyone interested in using it or contributing to it.
At this point I had all the basics that I needed so I published my new tool to Github. The next important step is to make sure your tool is versioned. This allows users to see the state of your package and helps Go’s get tool to make sure that it has the right version of your package should it need a specific version. The best way to handle versioning is by using Semantic Versioning. This type of versioning uses three numbers separated by a . that indicate the major, minor, and patch version. The first number is your major version. This starts at 0 while you are in the development phase of production and will change to 1 when it is ready for production. After that it should only go up if you have changed something that will make it no longer backwards compatible. This would be some kind of breaking change to the code. The second number is your minor version. This also starts at 0 and would increase for each time you add new functionality to the package. The final number is the patch number and this generally would increment for each bug fix you do.
With this in mind I went to the terminal and added my tag with the command git tag v0.0.1 and sent this new tag to Github with git push origin v0.0.1. Then I went to Github and created my new release using the link on the right side of the page, followed the prompts and it was done!
I was so excited to have my new package published and versioned that I couldn’t wait to see it show up on Go’s package site. I went straight over and searched for it only to find that it was not there. You see, just publishing to Github and providing it with a version does not mean that it will show up. Go needs to update the site and it does this when someone uses the go get command to fetch your package.
Now I was ready, I started up a new project, imported my package, ran
go get github.com/gmr5311/jsondig
to make sure it was pulled from Github and not my local machine and started testing. It still worked just like it had before, but it still didn’t show up on the package site. This was really just because it takes time for the site to update, but I wanted to make sure. To really verify I used another command to tell Go to list my package on their site which was
GOPROXY=proxy.golang.org go list -m github.com/gmr5311/jsondig@v0.0.1.
After running that command and waiting about 5 minutes my package was listed on the site! Remember if you are going to use this command to replace the github.com part with the name and version number of your repo.
Go’s package site really is fantastic. Once your package has been published and is available for everyone to use the site will populate a page for your package with the details that you provided in your code. The page for my little package can be found here as an example for what we will talk about next. The first section of the page provides details about the current latest version, when it was published, the license type, and how many times it has been imported. After that comes a section that tells you if the package uses a go mod file, if you can redistribute this code based on the license type, if it has tagged versions, and if it is a stable release. Once your package has reached major version 1 it will be considered a stable release. All this information helps to tell potential users what they can do with your code at a glance.
Next comes details that Go gets from your repo. First is your README. This is a very important document that should tell your users how to use this new package that you have created. It gets included on your package page exactly as it is written in your repo with links added for every heading in the README, so make sure to pay attention to your formatting.
After the README section comes the Documentation section. This section starts out with an overview which will be the package documentation that you provide for your package. In case you didn’t know, this documentation comes in the form of a code comment directly above the package declaration line in your code. This should give some basic information about why you might be using this package. I’ve seen some people create a documentation file in their repo that they use for all their package documentation and simply end the file with the package declaration. It might look something like this:
// Package sample is a package to show you how a package file
// might look.
//
// You have to make sure that the file ends with the package
// declaration like it does here.
//
// The file might be called something like doc.go
//
// Remember that everything that belongs to your package must
// be in the same folder.
//
package sample
After the overview section is the Index, which is just links to everything else that follows. This can be very important for packages that are very large because navigation on the page can be challenging otherwise.
After the Index comes four sections called Constants, Variables, Functions, and Types. Exactly like each section sounds, these are the constants, variables, functions, and types that you declared for your package. It doesn’t matter what file they are found in, they will show up in the list. Here you will see the declaration line for each item followed by the documentation that you provided before each item’s declaration. In Go anything that is exported starts with a capital letter and you should make sure that any exported element has documentation to support it. These are the things that people will be interacting with so quality documentation will matter here. With this documentation showing up on the package site as well, the quality of your documentation could impact if someone will import and use your package or if they will contribute to your project, so try not to cut corners here.
The final section is the Source Files. This is the list of .go files that are in your repo, excluding test files. Clicking on any of them will take you directly to the repo with that file.
By this point you have your repo up on Github and available for everyone on the Go package site, what else is there? Well, there are a few things that might help your project pick up some steam. For one thing you might consider setting up some automated test runners. You should already have some tests set up as part of the TDD process, so you can configure Github to run them for you any time you push something to the repo. Some options that are available for this are circleci, Github Actions, or TravisCI. Each has their own pros and cons, but that is a topic for another time. There can be costs associated with using these tools, but if your repo is public they often will allow you to use them for free. I’ve used circleci before, but I wanted to try out Github Actions for this project so I could learn a bit more about it. The details there are also a topic for another time. In the end I set Actions with a very simple approach. Here is the yml file that I used:
name: jsondig tests
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.17
- name: Test
run: make test
You’ll notice that the command to run my tests uses make. I set up a simple make script to lint and test at the same time which looks like this:
.DEFAULT_GOAL := lint
NAME := $(shell basename $(CURDIR))
clean:
@echo "Cleaning ${NAME}..."
@go clean -i ./...
@rm -rf bin
test: clean
@echo "Testing ${NAME}..."
@go test ./... -cover -race
lint: test
@echo "Linting ${NAME}..."
@go vet ./...
@golangci-lint run #https://golangci-lint.run/usage/install/
The make file just lives in the main directory labeled Makefile and the action file lives in .github/workflows/test.yml. This is enough to allow the tests to run any time I open a pull request on the main branch or any time anything is pushed to the main branch.
Other things to make sure you include in your repo is a LICENSE file. This tells both Github and the Go package site what people are allowed to do with your code. I used an MIT license, but you should research the various license types to find the one that best matches what you want to do.
The last thing that I might recommend is a CODEOWNERS file. You can set up protections that prevent people from doing things to your repo that you haven’t authorized, and this is one of those things. With code owners you can provide rules about who is required to review and approve a pull request before it is merged and other things like that. Creating the file is only the first step and more details about code owners can be found here.
Creating a package that I was able to make available for everyone to use was a very rewarding experience. What I created is very simple and really doesn’t do much right now, but it was an important first step to building something more. I imagine one day having a package that can manipulate json strings in a variety of ways and provide a more robust way to collect information out of them for testing or anything else you can think of. So many people out there have other ideas that could impact programmer’s lives in a positive way but may not know how to get their code out there. My goal in this post was to help to demystify that process a little bit and share with you the journey that I went on to learn it for myself. Let me know if you have any questions or comments that I can address, or if you have ideas of other blockers that you would benefit from learning more about. I look forward to hearing from you!