- Notifications
You must be signed in to change notification settings - Fork60
Decorated Syntax Tree - manipulate Go source with perfect fidelity.
License
dave/dst
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Thedst package enables manipulation of a Go syntax tree with high fidelity. Decorations (e.g.comments and line spacing) remain attached to the correct nodes as the tree is modified.
Thego/ast package wasn't created with source manipulation as an intended use-case. Comments arestored by their byte offset instead of attached to nodes, so re-arranging nodes breaks the output.Seethis Go issue for more information.
Consider this example where we want to reverse the order of the two statements. As you can see thecomments don't remain attached to the correct nodes:
code:=`package afunc main(){var a int // foovar b string // bar}`fset:=token.NewFileSet()f,err:=parser.ParseFile(fset,"",code,parser.ParseComments)iferr!=nil {panic(err)}list:=f.Decls[0].(*ast.FuncDecl).Body.Listlist[0],list[1]=list[1],list[0]iferr:=format.Node(os.Stdout,fset,f);err!=nil {panic(err)}//Output://package a////func main() {//// foo//var b string//var a int//// bar//}
Here's the same example usingdst:
code:=`package afunc main(){var a int // foovar b string // bar}`f,err:=decorator.Parse(code)iferr!=nil {panic(err)}list:=f.Decls[0].(*dst.FuncDecl).Body.Listlist[0],list[1]=list[1],list[0]iferr:=decorator.Print(f);err!=nil {panic(err)}//Output://package a////func main() {//var b string // bar//var a int // foo//}
Parsing a source file todst and printing the results after modification can be accomplished withseveralParse andPrint convenience functions in thedecoratorpackage.
For more fine-grained control you can useDecoratorto convert fromast todst, andRestorerto convert back again.
Comments are added at decoration attachment points.See herefor a full list of these points, along with demonstration code of where they are rendered in theoutput.
The decoration attachment points have convenience functionsAppend,Prepend,Replace,ClearandAll to accomplish common tasks. Use the full text of your comment including the// or/**/markers. When adding a line comment, a newline is automatically rendered.
code:=`package mainfunc main() {println("Hello World!")}`f,err:=decorator.Parse(code)iferr!=nil {panic(err)}call:=f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)call.Decs.Start.Append("// you can add comments at the start...")call.Decs.Fun.Append("/* ...in the middle... */")call.Decs.End.Append("// or at the end.")iferr:=decorator.Print(f);err!=nil {panic(err)}//Output://package main////func main() {//// you can add comments at the start...//println /* ...in the middle... */ ("Hello World!") // or at the end.//}
TheBefore property marks the node as having a line space (new line or empty line) before the node.These spaces are rendered before any decorations attached to theStart decoration point. TheAfterproperty is similar but rendered after the node (and after anyEnd decorations).
code:=`package mainfunc main() {println(a, b, c)}`f,err:=decorator.Parse(code)iferr!=nil {panic(err)}call:=f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr)call.Decs.Before=dst.EmptyLinecall.Decs.After=dst.EmptyLinefor_,v:=rangecall.Args {v:=v.(*dst.Ident)v.Decs.Before=dst.NewLinev.Decs.After=dst.NewLine}iferr:=decorator.Print(f);err!=nil {panic(err)}//Output://package main////func main() {////println(//a,//b,//c,//)////}
The common decoration properties (Start,End,Before andAfter) occur on all nodes, and can beaccessed with theDecorations() method on theNode interface:
code:=`package mainfunc main() {var i inti++println(i)}`f,err:=decorator.Parse(code)iferr!=nil {panic(err)}list:=f.Decls[0].(*dst.FuncDecl).Body.Listlist[0].Decorations().Before=dst.EmptyLinelist[0].Decorations().End.Append("// the Decorations method allows access to the common")list[1].Decorations().End.Append("// decoration properties (Before, Start, End and After)")list[2].Decorations().End.Append("// for all nodes.")list[2].Decorations().After=dst.EmptyLineiferr:=decorator.Print(f);err!=nil {panic(err)}//Output://package main////func main() {////var i int // the Decorations method allows access to the common//i++ // decoration properties (Before, Start, End and After)//println(i) // for all nodes.////}
While debugging, it is often useful to have a list of all decorations attached to a node. Thedstutil package provides a helper functionDecorations whichreturns a list of the attachment points and all decorations for any node:
code:=`package main// main comment// is multi linefunc main() {if true {// fooprintln( /* foo inline */ "foo")} else if false {println /* bar inline */ ("bar")// bar after} else {// empty block}}`f,err:=decorator.Parse(code)iferr!=nil {panic(err)}dst.Inspect(f,func(node dst.Node)bool {ifnode==nil {returnfalse}before,after,points:=dstutil.Decorations(node)varinfostringifbefore!=dst.None {info+=fmt.Sprintf("- Before: %s\n",before)}for_,point:=rangepoints {iflen(point.Decs)==0 {continue}info+=fmt.Sprintf("- %s: [",point.Name)fori,dec:=rangepoint.Decs {ifi>0 {info+=", "}info+=fmt.Sprintf("%q",dec)}info+="]\n"}ifafter!=dst.None {info+=fmt.Sprintf("- After: %s\n",after)}ifinfo!="" {fmt.Printf("%T\n%s\n",node,info)}returntrue})//Output://*dst.FuncDecl//- Before: NewLine//- Start: ["// main comment", "// is multi line"]////*dst.IfStmt//- Before: NewLine//- After: NewLine////*dst.ExprStmt//- Before: NewLine//- Start: ["// foo"]//- After: NewLine////*dst.CallExpr//- Lparen: ["/* foo inline */"]////*dst.ExprStmt//- Before: NewLine//- End: ["\n", "\n", "// bar after"]//- After: NewLine////*dst.CallExpr//- Fun: ["/* bar inline */"]////*dst.BlockStmt//- Lbrace: ["\n", "// empty block"]
TheBefore andAfter properties cover the majority of cases, but occasionally a newline needs tobe rendered inside a node. Simply add a\n decoration to accomplish this.
Re-using an existing node elsewhere in the tree will panic when the tree is restored toast. Instead,use theClone function to make a deep copy of the node before re-use:
code:=`package mainvar i /* a */ int`f,err:=decorator.Parse(code)iferr!=nil {panic(err)}cloned:=dst.Clone(f.Decls[0]).(*dst.GenDecl)cloned.Decs.Before=dst.NewLinecloned.Specs[0].(*dst.ValueSpec).Names[0].Name="j"cloned.Specs[0].(*dst.ValueSpec).Names[0].Decs.End.Replace("/* b */")f.Decls=append(f.Decls,cloned)iferr:=decorator.Print(f);err!=nil {panic(err)}//Output://package main////var i /* a */ int//var j /* b */ int
Thedstutil package is a fork ofgolang.org/x/tools/go/ast/astutil,and provides theApply function with similar semantics.
The decorator can automatically manage theimport block, which is a non-trivial task.
UseNewDecoratorWithImportsandNewRestorerWithImportsto create an import aware decorator / restorer.
During decoration, remote identifiers are normalised -*ast.SelectorExpr nodes that representqualified identifiers are replaced with*dst.Ident nodes with thePath field set to the path ofthe imported package.
When adding a qualified identifier node, there is no need to use*dst.SelectorExpr - just add a*dst.Ident and setPath to the imported package path. The restorer will wrap it in a*ast.SelectorExpr where appropriate when converting back to ast, and also update the importblock.
To enable import management, the decorator must be able to resolve the imported package forselector expressions and identifiers, and the restorer must be able to resolve the name of apackage given it's path. Several implementations for these resolvers are provided, and the bestmethod will depend on the environment.See below for more details.
TheLoad convenience function usesgo/packages to load packages and decorate all loaded ast files, with import management enabled:
// Create a simple module in a temporary directorydir,err:=tempDir(map[string]string{"go.mod":"module root","main.go":"package main\n\n func main() {}",})deferos.RemoveAll(dir)iferr!=nil {panic(err)}// Use the Load convenience function that calls go/packages to load the package. All loaded// ast files are decorated to dst.pkgs,err:=decorator.Load(&packages.Config{Dir:dir,Mode:packages.LoadSyntax},"root")iferr!=nil {panic(err)}p:=pkgs[0]f:=p.Syntax[0]// Add a call expression. Note we don't have to use a SelectorExpr - just adding an Ident with// the imported package path will do. The restorer will add SelectorExpr where appropriate when// converting back to ast. Note the new Path field on *dst.Ident. Set this to the package path// of the imported package, and the restorer will automatically add the import to the import// block.b:=f.Decls[0].(*dst.FuncDecl).Bodyb.List=append(b.List,&dst.ExprStmt{X:&dst.CallExpr{Fun:&dst.Ident{Path:"fmt",Name:"Println"},Args: []dst.Expr{&dst.BasicLit{Kind:token.STRING,Value:strconv.Quote("Hello, World!")},},},})// Create a restorer with the import manager enabled, and print the result. As you can see, the// import block is automatically managed, and the Println ident is converted to a SelectorExpr:r:=decorator.NewRestorerWithImports("root",gopackages.New(dir))iferr:=r.Print(p.Syntax[0]);err!=nil {panic(err)}//Output://package main////import "fmt"////func main() { fmt.Println("Hello, World!") }
The decorator exposesDst.Nodes andAst.Nodes which map betweenast.Node anddst.Node. Thisenables systems that refer toast nodes (such asgo/types) to be used:
code:=`package mainfunc main() {var i inti++println(i)}`// Parse the code to ASTfset:=token.NewFileSet()astFile,err:=parser.ParseFile(fset,"",code,parser.ParseComments)iferr!=nil {panic(err)}// Invoke the type checker using AST as inputtypesInfo:= types.Info{Defs:make(map[*ast.Ident]types.Object),Uses:make(map[*ast.Ident]types.Object),}conf:=&types.Config{}if_,err:=conf.Check("",fset, []*ast.File{astFile},&typesInfo);err!=nil {panic(err)}// Create a new decorator, which will track the mapping between ast and dst nodesdec:=decorator.NewDecorator(fset)// Decorate the *ast.File to give us a *dst.Filef,err:=dec.DecorateFile(astFile)iferr!=nil {panic(err)}// Find the *dst.Ident for the definition of "i"dstDef:=f.Decls[0].(*dst.FuncDecl).Body.List[0].(*dst.DeclStmt).Decl.(*dst.GenDecl).Specs[0].(*dst.ValueSpec).Names[0]// Find the *ast.Ident using the Ast.Nodes mappingastDef:=dec.Ast.Nodes[dstDef].(*ast.Ident)// Find the types.Object corresponding to "i"obj:=typesInfo.Defs[astDef]// Find all the uses of that objectvarastUses []*ast.Identforid,ob:=rangetypesInfo.Uses {ifob!=obj {continue}astUses=append(astUses,id)}// Find each *dst.Ident in the Dst.Nodes mappingvardstUses []*dst.Identfor_,id:=rangeastUses {dstUses=append(dstUses,dec.Dst.Nodes[id].(*dst.Ident))}// Change the name of the original definition and all usesdstDef.Name="foo"for_,id:=rangedstUses {id.Name="foo"}// Print the DSTiferr:=decorator.Print(f);err!=nil {panic(err)}//Output://package main////func main() {//var foo int//foo++//println(foo)//}
There are two separate interfaces defined by theresolver packagewhich allow the decorator and restorer to automatically manage the imports block.
The decorator uses aDecoratorResolver which resolves the package path of any*ast.Ident. This iscomplicated by dot-import syntax (see below).
The restorer uses aRestorerResolver which resolves the name of any package given the path. Thisis complicated by vendoring and Go modules.
WhenResolver is set onDecorator orRestorer, thePath property must be set to the localpackage path.
Several implementations of both interfaces that are suitable for different environments areprovided:
Thegotypespackage provides aDecoratorResolver with full dot-import compatibility. However it requires fullexport data for all imported packages, so theUses map fromgo/types.Info is required. Thereare several methods of generatinggo/types.Info. Usinggolang.org/x/tools/go/packages.Load isrecommended for full Go modules compatibility. See thedecorator.Loadconvenience function to automate this.
Thegoast packageprovides a simplifiedDecoratorResolver that only needs to scan a single ast file. This is unableto resolve identifiers from dot-imported packages, so will panic if a dot-import is encountered inthe import block. It uses the providedRestorerResolver to resolve the names of all importedpackages. If noRestorerResolver is provided, theguess implementation is used.
Thegopackagespackage provides aRestorerResolver with full compatibility with Go modules. It usesgolang.org/x/tools/go/packages to load the package data. This may be very slow, and uses thegocommand line tool to query package data, so may not be compatible with some environments.
Thegobuildpackage provides an alternativeRestorerResolver that uses the legacygo/build system to loadthe imported package data. This may be needed in some circumstances and provides better performancethango/packages. However, this is not Go modules aware.
Theguess andsimple packagesprovide simpleRestorerResolver implementations that may be useful in certain circumstances, orwhere performance is critical.simple resolves paths only if they occur in a provided map.guess guesses the package name based on the last part of the path.
Here's an example of supplying resolvers for the decorator and restorer:
code:=`package mainimport "fmt"func main() {fmt.Println("a")}`dec:=decorator.NewDecoratorWithImports(token.NewFileSet(),"main",goast.New())f,err:=dec.Parse(code)iferr!=nil {panic(err)}f.Decls[1].(*dst.FuncDecl).Body.List[0].(*dst.ExprStmt).X.(*dst.CallExpr).Args= []dst.Expr{&dst.CallExpr{Fun:&dst.Ident{Name:"A",Path:"foo.bar/baz"},},}res:=decorator.NewRestorerWithImports("main",guess.New())iferr:=res.Print(f);err!=nil {panic(err)}//Output://package main////import (//"fmt"////"foo.bar/baz"//)////func main() {//fmt.Println(baz.A())//}
To control the alias of imports, use aFileRestorer:
code:=`package mainimport "fmt"func main() {fmt.Println("a")}`dec:=decorator.NewDecoratorWithImports(token.NewFileSet(),"main",goast.New())f,err:=dec.Parse(code)iferr!=nil {panic(err)}res:=decorator.NewRestorerWithImports("main",guess.New())fr:=res.FileRestorer()fr.Alias["fmt"]="fmt1"iferr:=fr.Print(f);err!=nil {panic(err)}//Output://package main////import fmt1 "fmt"////func main() {//fmt1.Println("a")//}
For more information on exactly how the imports block is managed, read through thetestcases.
Consider this file...
package mainimport (."a")funcmain() {B()C()}
B andC could be local identifiers from a different file in this package,or from the imported packagea. If only one is froma and it is removed, we should remove theimport when we restore toast. Thus the resolver needs to be able to resolve the package usingthe full info fromgo/types.
This package is well tested and used in many projects. The API should be considered stable going forward.
Feel free to create anissue or chat in the#dst Gophers Slack channel.
For further developing or contributing todst, check outthese notes.
Thanks very much tohawkinsw for taking on the task of adding generics compatibility todst.
About
Decorated Syntax Tree - manipulate Go source with perfect fidelity.
Resources
License
Contributing
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Contributors4
Uh oh!
There was an error while loading.Please reload this page.