
Posted on • Originally published atmeetgor.com
Golang: JSON, YAML, TOML (config) File Reading.
Reading specific file types (JSON, YAML, TOML)
In the previous post, we have seen how to read files in golang, in this extended post of that part, we will look into reading some specific files used for configuration and storing data like JSON, YAML, TOML, CSV, etc.
We will see how to read files and get individual pieces in the files. We'll use packages likeos
,ioutil
andencoding
to perform reading operations on file and file objects.
Reading a JSON File
Golang has built-in support for reading JSON files, but still, we can and need to have low-level controls on how to parse and extract content from the file.
Let's say we have ajson
file namedblog.json
, we can use theencoding/json package to convert the JSON data into a GO object (an object that is native and understandable to go). TheUnmarshal function is used to convert the slice of bytes from the file, into a map object.
{"title":"Golang Blog Series","date":"22nd October 2022","tags":["go","files"],"words":1500,"published":true}
The above is a simple JSON file, this file has a few types of key-value pairs like string, list, integer, and boolean. But we can also have nested objects and a list of those nested objects.
packagemainimport("encoding/json""log""os")funcmain(){f,err:=os.ReadFile("blog.json")iferr!=nil{log.Println(err)}vardatamap[string]interface{}json.Unmarshal([]byte(f),&data)log.Println(data)fork,v:=rangedata{log.Println(k,":",v)}}
I have removed the time stamp from the logs below so as to clearly see the output. We can usefmt
to print the simple things while keeping consistent with the rest of the snippets in the series.
$ go run json.gomap[date:22nd October 2022 published:true tags:[go files] title:Golang Blog Series words:1500]published : truetitle : Golang Blog Seriesdate : 22nd October 2022tags : [go files]words : 1500
The file is read using theos.ReadFile method, that takes in a string as a path to the file and returns a slice of bytes or an error if there was an issue in reading the file. The parsed slice of byte is than passed as the first argument to theUnmarshal
method in theencoding/json
package. The second parameter is the output reference where the parsed JSON will be stored or returned. The function does not return the parsed content instead returns an error if there arose any while parsing the JSON content.
As we can see we have got a map ofstring
with aninterface
. The interface is used because the value of the key can be anything. There is no fixed value like astring
,int
,bool
, or a nestedmap
,slice
. Hence we have mapped the JSON object as a map ofstring
with aninterface
. The type of the value is identified with the interface it has attached to it. Let's take a look what is the type of each value in the map.
published : truebooltitle : Golang Blog Seriesstringdate : 22nd October 2022stringtags : [go files][]interface {}words : 1500float64
Here, we can see it has correctly identified the string type of the fields like bool in case of true or false, a string for string type of values, the fourth field however has a list interface attached to it. The default precedence forfloat64
over integer is the reason the1500
value is of typefloat64
.
Reading a YAML File
Though there is no standard package for parsing/unmarshalingYAML
files in golang, it's quite easy to use a third-party package and use it to read YAML files.
The packagegopkg.in/yaml.v3 is used for encoding and decoding YAML files. We'll be just using it for decoding a YAML file by reading it and converting the file object into native Go objects like maps, lists, strings, etc.
The below steps can be used for setting up the package and installing the YAML package locally.
go mod init <your_project_package_name>go get gopkg.in/yaml.v3
This should create two files namelygo.mod
andgo.sum
with the dependency of thegopkg.in/yaml.v3
package.
title:"GolangBlogSeries"date:"22ndOctober2022"tags:["go","files"]published:falsewords:1500
The above file is a simple YAML config, we'll follow similar kind of examples for the dummy files used in the examples.
packagemainimport("log""os"yaml"gopkg.in/yaml.v3")funcmain(){f,err:=os.ReadFile("blog.yaml")iferr!=nil{log.Fatal(err)}vardatamap[string]interface{}err=yaml.Unmarshal(f,&data)iferr!=nil{log.Fatal(err)}log.Println(data)fork,v:=rangedata{log.Println(k,":",v)}}
$ go run yaml.gomap[date:22nd October 2022 published:false tags:[go files] title:Golang Blog Series words:1500]published : falsewords : 1500title : Golang Blog Seriesdate : 22nd October 2022tags : [go files]
The above code and output demonstrate the usage of theyaml.v3
package for reading a YAML file.
Firstly, we read the file into a single-string object with theos.ReadFile() method. The method will return a[]byte
(slice of byte) or an error. If there is an error, we simply log or panic out of the program, else we can use theyaml.Unmarshal method to convert the string/slice of the byte into a map or a pre-defined struct. In this example, we just keep things simple by storing the file content asmap [string, interface{}]
, i.e. a map ofstring
and aninterface
. The key for YAML can be only a string or an integer. It can't have unrestricted data types like the value can have. Though if you want to be unrestrictive, you can use a map ofmap[interface{}]interface{}
to make the key any shape you like to have.
So, we have created a variable calleddata
as a map ofstring
andinterface{}
, basically key can be a string and the value can be any type of interface depending on the parsed literally from the file object. TheUnmarshal
function takes in two parameters, the first being the slice of byte i.e. the file contents, and the second being the output variable. Now, the method does not return the parsed YAML, it can return an error if there arose, so we need to set the second parameter as a pointer to the object into which we want to store the parsed YAML.
In the example, we have calledUnmarshal(f, &data)
which will fetch the contents from the slice of bytesf
and output the parsed YAML from the slice of bytes into the memory location ofdata
and hence using&data
indicating the pointer to the variable(fetch the memory address).
So, that is how we obtain the map of keys and values from the YAML config, thereafter, you can iterate on the map, access the keys and values, type caste them as per requirement, and basically have control over what processing needs to be done to the parsed YAML content.
published : falseboolwords : 1500inttitle : Golang Blog Seriesstringdate : 22nd October 2022stringtags : [go files][]interface {}
I have just printed the types of the values in the above output aslog.Printf("%T", v)
, we can see the types are being correctly recognized and being parsed. The last object is indeed aslice
so it has an interface of the slice(array) attached to it.
Reading a TOML file
YAML and TOML are almost identical, though toml has more restrictive configuration options, but is more readable than YAML, as YAML can get complicated pretty quickly. Though both of them have their pros and cons, YAML is used everywhere in the DevOps world, configs, whereas TOML is the format of choice for python packaging, and static site generation configs.
Let's see how we can use golang to read TOML files.
$ go mod init <your_project_package_name>$ go get github.com/pelletier/go-toml
The above commands are used for setting up a golang package or project and installing thego-toml package. Once the above commands are done executing, it will generatego.mod
andgo.sum
files used for storing dependencies and packages installed for the project locally.
[blog]name='techstructive-blog'tags=['go','django','vim']author='meet gor'active=true[author]name='Meet Gor'github='mr-destructive'twitter='meetgor21'posts=80
The above is the sample fileblog.toml
which we will use to read in the go script below. The toml file has a similar structure as we have seen in the previous examples. We have different data types like string, boolean, integer, and list.
packagemainimport("log""os"toml"github.com/pelletier/go-toml")funcmain(){f,err:=os.ReadFile("blog.toml")iferr!=nil{log.Fatal(err)}vardatamap[interface{}]interface{}err=toml.Unmarshal(f,&data)log.Println(data)iferr!=nil{log.Fatal(err)}fork,v:=rangedata{log.Println(k,":",v)switcht:=v.(type){casemap[string]interface{}:fora,b:=ranget{log.Println(a,":",b)}}}}
$ go run toml.gomap[author:map[github:mr-destructive name:Meet Gor posts:80 twitter:meetgor21] blog:map[active:true author:meet gorname:techstructive-blog tags:[go django vim]]]blog : map[active:true author:meet gor name:techstructive-blog tags:[go django vim]]name : techstructive-blogtags : [go django vim]author : meet goractive : trueauthor : map[github:mr-destructive name:Meet Gor posts:80 twitter:meetgor21]name : Meet Gorgithub : mr-destructivetwitter : meetgor21posts : 80
So, in the above example and output, the YAML file was read and the key-value pairs inside them were read. The first thing we do is read the fileblog.toml
withioutil
package, with theReadFile
function. The function takes in the string as the path to the file to be read and returns a slice of byte. We use this slice of byte as a parameter to theUnmarshal method. The second paramter for theUnmarshal
is the output variable(usually a pointer to a variable), we have created a map ofinterface{]
with aninterface
as we see there can be nested keys which hold the name of the config.
The variabledata
is a map ofinterface{}
to aninterface{}
, and we parse the memory address to thedata
variable to theUnmarshal
method. Thereby the parsedTOML
content is stored in the data variable.
name : techstructive-blogstringtags : [go django vim][]interface{}author : meet gorstringactive : trueboolname : Meet Gorstringgithub : mr-destructivestringtwitter : meetgor21stringposts : 80int64
The above is a verbose output for the type of the values as parsed by golang, we havestring
,bool
,int64
, and aslice
(list with interface{} attached with it). Only types likestring
,bool
,int
,float64
can be parsed from the Unmarshal function, other than these types, the type will have an interface attached to it.
In such cases, where the type of value is not among the 4 types(string, bool, int float), we can use a pre-defined struct to parse the content from the file. Though it would require a strict structure and predictable response from the parsed file.
Reading CSV file
We can even read a CSV file in golang, we have seen in the previous post, we have used custom delimiters in the parsing of the file.
id,name,posts,exp21,jim,23,233,kevin,39,145,james,70,256,chris,89,3
The above file is a sample csv file, though the size is too small, we can use it as an example.
packagemainimport("encoding/csv""log""os")funcmain(){f,err:=os.Open("temp.csv")check_error(err)reader:=csv.NewReader(f)n,err:=reader.ReadAll()check_error(err)for_,line:=rangen{for_,text:=rangeline{log.Println(text)}}}
$ go run main.goidnamepostsexp21jim23233kevin39145james70256chris893
The CSV package has aNewReader method that accepts anio.Reader
and returns aReader
object. After parsing the reader, we use theReadAll method to return a 2d string or an error if there exists an error while parsing the content. You can get a detailed explanation of the CSV parsing and reading in theprevious post.
Reading CSV from URL
The CSV file can also be read from the URL, the content of the file is aresponse.Body
in place of the file object reference, in the previous example, theos.Open() method returns aos.File object.
We use thehttp.Get(string)
method to get the response from the URL for reading the CSV file present on the web.
packagemainimport("encoding/csv""log""net/http")funcmain(){url:="https://github.com/woocommerce/woocommerce/raw/master/sample-data/sample_products.csv"response,err:=http.Get(url)iferr!=nil{log.Println(err)return}deferresponse.Body.Close()reader:=csv.NewReader(response.Body)n,err:=reader.ReadAll()iferr!=nil{log.Println(err)}for_,line:=rangen{for_,text:=rangeline{log.Println(text)}}}
$ go run csv.go<feff>IDTypeSKUNamePublishedIs featured?Visibility in catalogShort descriptionDescriptionDate sale price startsDate sale price ends.........
So, that's how we can read a CSV file from the URL. By fetching the CSV URLhttps://github.com/woocommerce/woocommerce/raw/master/sample-data/sample_products.csv
from thehttp.Get method, this will get us theresponse.Body that contains the actual CSV file content. The response than can be parsed to thecsv.NewReader(*Os.File).ReadAll()
i.e.reader.ReadAll(). The function returns a multidimensional slice[][]slice
that can be iterated and parsed as per requirement.
Reading XML file
XML is the de facto standard for RSS feeds, it is widely used in many places and is still all over the web. We'll see an example to read an XML file locally, but as we saw in the above example, we can also read an RSS link from the web.
Just like CSV, we haveencoding/xml, and the standard library has all the functions used for parsing the XML files.
We will be using a local XML file calledrss.xml
, and reading the contents from the tags in the file.
<?xml version="1.0" encoding="UTF-8" ?><channel><title>Meet Gor</title><description>Techstructive Blog Feed</description><item><title>Why and How to make and use Vim as a text editor and customizable IDE</title><link>https://www.meetgor.com/vim-text-editor-ide</link></item><item><title>Setting up Vim for Python</title><link>https://www.meetgor.com/vim-for-python</link></item></channel>
The above example is a short part of my blog'srss feed. I have just trimmed the unwanted part and will be just using the tags that we want to fetch.
packagemainimport("encoding/xml""log""os")typeChannelstruct{XMLNamexml.Name`xml:"channel"`Titlestring`xml:"title"`Descriptionstring`xml:"description"`Item[]Item`xml:"item"`}typeItemstruct{XMLNamexml.Name`xml:"item"`Titlestring`xml:"title"`Linkstring`xml:"link"`}funccheck_error(errerror){iferr!=nil{log.Fatal(err)}}funcmain(){f,err:=os.ReadFile("rss.xml")check_error(err)deferf.Close()d:=Channel{}err=xml.Unmarshal(f,&d)check_error(err)for_,item:=ranged.Item{log.Println(item.Title)}}
$ go run xml.go{{ channel} Meet Gor Techstructive Blog Feed [{{ item} Why and How to make and use Vim as a text editor and customizable IDE https://www.meetgor.com/vim-text-editor-ide} {{ item} Setting up Vim for Python https://www.meetgor.com/vim-for-python}]}Why and How to make and use Vim as a text editor and customizable IDESetting up Vim for Python
The above example uses a couple ofstruct
likeChannel
andItem
that stores the tag data liketitle
,description
,link
, etc. Unlike the JSON, YAML, and toml files, XML can't parse the content directly we need a structure to parse into. And that's fine as XML is not much dynamic in terms of config, we usually have standard tags and elements which can be pre-defined in a struct type.
In this example, the RSS feed has aChannel
tag withtitle
,description
, anditem
.
NOTE: Use Title case for the fields of the structs. It will make the fields public, I spent a few hours debugging that really :)
So, we define theChannel
struct with fields likeTitle
as a string which is a tag in the file asxml:"title"
. This means the title in the tag of the XML will be stored in the field as a string in the attribute name asTitle
. Similarly, we have fields likeDescription
andItem[]
this is a list or multiple ofitem
tags that might be present in the XML file. TheXMLName
is used for identifying the parent tag for the struct, so we usechannel
for the first struct as it is the first tag appearing of the hierarchy in the XML file.
So, we create an object for the root structure asChannel{}
(an empty object instantiated). Thexml.Unmarshal
function is parsed with the content of the file asdata
which is a slice of byte as we have seen in the previous examples. The slice of byte is then used in theUnmarshal
method as the first parameter and the reference of an emptyChannel
object as the second parameter. The second parameter will be to store the parsed XML content from the file.
I have a few examples on the GitHub repository covering the reading of files from a URL for the CSV, and XML files. But, this concept in the example, can be applied to JSON, YAML, and other file formats as well.
That's it from this part. Reference for all the code examples and commands can be found in the100 days of Golang GitHub repository.
Conclusion
So, that's it from this post, we covered how to read specific configuration files likeJSON
,CSV
,YAML
,TOML
, andXML
. We saw how to read a local file and also touched on the concept to read contents from a file on the web with a URL. We also saw how we can use pre-defined structs to parse content from a file, especially for XML.
Thank you for reading. If you have any queries, questions, or feedback, you can let me know in the discussion below or on my social handles. Happy Coding :)series: "['100-days-of-golang']"
Top comments(1)
For further actions, you may consider blocking this person and/orreporting abuse