Share
Embed
Building a go tool to modify struct tags
Struct field tags are an important part of encode/decode types, especially when using packages such as encoding/json. However, modifying tags is repetitive, cumbersome and open to human errors. We can make it easy to modify tags with an automated tool that is written for this sole purpose.
Fatih Arslan
July 13, 2017
Transcript
232 slides
-
Building a go tool to modify struct tags Fatih Arslan
Software Engineer @DigitalOcean -
type Example struct { StatusID int64 Foo string Bar bool
Server struct { Address string TLS bool } DiskSize int64 Volumes []string } -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } -
So, what's a struct tag? type Example struct { Foo
string `json:"foo"` } -
type Example struct { Foo string "json:\"foo\"" } quotes instead
of backticks (works, but not fun to deal with) -
`json:"foo,omitempty" option (not part of the spec) space separation key
value xml:"bar"` backtick -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"tls" xml:"tls"` -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"server"` -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"disk_size"` -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } -
type Example struct { StatusID int64 `json:"status_id"` Foo string `json:"foo"
xml:"foo"` Bar bool `json:"bar" xml:"bar"` Server struct { Address string `json:"address"` TLS bool `json:"tls",xml:"tls"` } `json="server"` DiskSize int64 `json:disk_size` Volumes []string `"json":"volumes"` } `json:"volumes"` -
Old implementation • From the cursor search backwards for the
first 'struct {' literal • Once found, do a forward search for the right hand brace } • Get the line numbers between the two braces • For each line, get the first identifier, convert it to camel_case word and then append to the same line • A combination of Vim's search() function and regex is being used -
No Formatting type Example struct { Bar string `json:"bar"` Foo
bool `json:"foo"` } type Example struct { Bar string `json:"bar"` Foo bool `json:"foo"` } want have -
In-line comments type Example struct { Bar string `json:"bar"` //
comment for bar Foo bool `json:"foo"` } type Example struct { Bar string // comment for bar `json:"bar"` Foo bool `json:"foo"` } want have -
Nested structs type Example struct { Bar bool `json:"bar"` Server
struct { `json:"server"` Address string `json:"address"` } } type Example struct { Bar bool `json:"bar"` Server struct { Address string `json:"address"` } `json:"server"` } have want -
Duplicate tags type Example struct { Bar string `json:"bar"` `xml:"bar"`
Foo bool `xml:"foo"` } have type Example struct { Bar string `json:"bar" xml:"bar"` Foo bool `xml:"foo"` } want -
more quirks • not able to remove tags • not
able to add or remove options • field with interface{} types don't work • comment with braces ({ and }) don't work • ... -
3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr
*ast.BasicLit *ast.BasicLit 5 - 7 -
3 + (7 - 5) *ast.BinaryExpr *ast.BasicLit 3 + *ast.BinaryExpr
*ast.BasicLit *ast.BasicLit 5 - 7 -
type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType
*ast.FieldList *ast.Field -
type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType
*ast.FieldList *ast.Field []*ast.Ident Foo -
type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType
*ast.FieldList *ast.Field *ast.Ident string -
type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType
*ast.FieldList *ast.Field *ast.BasicLit `json:"foo"` (a.k.a: struct tag) -
type Example struct { Foo string `json:"foo"` } *ast.TypeSpec *ast.StructType
*ast.FieldList *ast.Field []*ast.Ident *ast.Ident *ast.BasicLit -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { fmt.Printf("Field: %s\n", field.Names[0].Name) fmt.Printf("Tag: %s\n", field.Tag.Value) } return false }) Field: Foo Tag: `json:"foo"` -
package main import ( "fmt" "reflect" ) func main() {
tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) } -
package main import ( "fmt" "reflect" ) func main() {
tag := reflect.StructTag(`json:"foo"`) value := tag.Get("json") fmt.Printf("value: %q\n", value) } $ go run main.go value: "foo" -
Issues with reflect.StructTag • can't detect if the tag is
malformed (only go vet knows that) • doesn't know the semantics of options (i.e: omitempty) • doesn't return all existing tags • modifying existing tags is not possible -
Parse and list all tags tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`)
if err != nil { panic(err) } // iterate over all key-value pairs for _, t := range tags.Tags() { fmt.Printf("tag: %+v\n", t) } $ go run main.go tag: json:"foo,omitempty" tag: xml:"foo" -
Get a single Tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) if
err != nil { panic(err) } jsonTag, err := tags.Get("json") if err != nil { panic(err) } fmt.Println(jsonTag) // Output: json:"foo,omitempty" fmt.Println(jsonTag.Key) // Output: json fmt.Println(jsonTag.Name) // Output: foo fmt.Println(jsonTag.Options) // Output: [omitempty] -
Change existing tag jsonTag, err := tags.Get(`json:"foo,omitempty"`) if err !=
nil { panic(err) } jsonTag.Name = "bar" jsonTag.Options = nil tags.Set(jsonTag) fmt.Println(tags) json:"bar" -
Add new tag tags, err := structtag.Parse(`json:"foo,omitempty" xml:"foo"`) hclTag :=
&structtag.Tag{ Key: "hcl", Name: "gopher", Options: []string{"squash"}, } // add new tag tags.Set(hclTag) fmt.Println(tags) json:"foo,omitempty" xml:"foo" hcl:"gopher,squash" -
Add/remove options tags, err := structtag.Parse(`json:"foo" xml:"bar,comment"`) if err !=
nil { panic(err) } tags.AddOptions("json", "omitempty") tags.DeleteOptions("xml", "comment") fmt.Println(tags) // json:"foo,omitempty" xml:"bar" -
Both issues are fixed now Parse struct (using go/parser) Parse
struct tag (using fatih/structtag) -
func main() { var cfg config node = cfg.parse() start,
end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result -
func main() { var cfg config node = cfg.parse() start,
end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result -
cfg := &config{ file: *flagFile, line: *flagLine, structName: *flagStruct, offset:
*flagOffset, output: *flagOutput, write: *flagWrite, clear: *flagClearTags, clearOption: *flagClearOptions, transform: *flagTransform, sort: *flagSort, override: *flagOverride, } $ gomodifytags --file example.go ... Use flags to set configuration -
Things we need: 1. What content to process 2. Where
and what to output 3. Which struct to modify 4. Which tags to modify -
Things we need: type config struct { file string modified
io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify -
Things we need: type config struct { file string modified
io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify -
Things we need: type config struct { file string modified
io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify -
Things we need: type config struct { file string modified
io.Reader output string write bool offset int structName string line string start, end int remove []string add []string override bool transform string sort bool clear bool addOpts []string removeOpts []string clearOpt bool } 1. What content to process 2. Where and what to output 3. Which struct to modify 4. Which tags to modify -
package main type Example struct { Foo string } $
gomodifytags -file example.go -struct Example -add-tags json -
func main() { var cfg config node = cfg.parse() start,
end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result -
gomodifytags parse parse go/parser package main type Example struct {
Foo string } *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident tag -
func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet()
var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content -
func (c *config) parse() (ast.Node, error) { c.fset = token.NewFileSet()
var contents interface{} if c.modified != nil { archive, err := buildutil.ParseOverlayArchive(c.modified) if err != nil { return nil, fmt.Errorf("failed to parse -modified archive: %v", err) } fc, ok := archive[c.file] if !ok { return nil, fmt.Errorf("couldn't find %s in archive", c.file) } contents = fc } return parser.ParseFile(c.fset, c.file, contents, parser.ParseComments) } Parse content (cont.) -
package main type Example struct { Foo string } parse
gomodifytags parse -
package main type Example struct { Foo string } parse
find gomodifytags -
func main() { var cfg config node = cfg.parse() start,
end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result -
parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit
select struct that matches our criteria ... -
package main type Example struct { Foo string } type
Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -
package main type Example struct { Foo string } type
Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -
package main type Example struct { Foo string } type
Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server -
package main type Example struct { Foo string } type
Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 struct name: Server start line: 8 end line: 10 -
package main type Example struct { Foo string } type
Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163) -
package main type Example struct { Foo string } type
Server struct { Name string Port| int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 offset byte: 96 (of 163) start line: 8 end line: 10 -
package main type Example struct { Foo string } type
Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 8 end line: 10 .. or specify explicit lines -
package main type Example struct { Foo string } type
Server struct { Name string Port int EnableLogs bool } type Person struct { Name string } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 start line: 9 end line: 9 .. or specify explicit lines -
parse find gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit
select struct that matches our criteria ... -
gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit ... and
get the start and end lines start line: 8 end line: 10 -
func (c *config) findSelection(node ast.Node) (int, int, error) { if
c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection -
func (c *config) findSelection(node ast.Node) (int, int, error) { if
c.line != "" { return c.lineSelection(node) } else if c.offset != 0 { return c.offsetSelection(node) } else if c.structName != "" { return c.structSelection(node) } return 0, 0, errors.New("-line, -offset or -struct is not passed") } Find selection -
// collectStructs collects and maps structType nodes to their positions
func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType, 0) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } -
// collectStructs collects and maps structType nodes to their positions
func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType } -
// collectStructs collects and maps structType nodes to their positions
func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } -
// collectStructs collects and maps structType nodes to their positions
func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } type structType struct { name string node *ast.StructType } -
// collectStructs collects and maps structType nodes to their positions
func collectStructs(node ast.Node) map[token.Pos]*structType { structs := make(map[token.Pos]*structType) collectStructs := func(n ast.Node) bool { t, ok := n.(*ast.TypeSpec) if !ok { return true } structName := t.Name.Name x, ok := t.Type.(*ast.StructType) if !ok { return true } structs[x.Pos()] = &structType{ name: structName, node: x, } return true } ast.Inspect(node, collectStructs) return structs } -
var encStruct *ast.StructType for _, st := range collectStructs() {
if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
if st.name == c.structName { encStruct = st.node } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line } Struct selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection -
var encStruct *ast.StructType for _, st := range collectStructs() {
structBegin := c.fset.Position(st.node.Pos()).Offset structEnd := c.fset.Position(st.node.End()).Offset if structBegin <= c.offset && c.offset <= structEnd { encStruct = st.node break } } start = c.fset.Position(encStruct.Pos()).Line end = c.fset.Position(encStruct.End()).Line Offset selection -
package main type Example struct { Foo string } parse
find gomodifytags find -
package main type Example struct { Foo string } parse
find modify gomodifytags -
func main() { var cfg config node = cfg.parse() start,
end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result -
gomodifytags start and end lines we're going to modify start
line: 8 end line: 10 -
gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit our parsed
AST start line: 8 end line: 10 -
gomodifytags *ast.TypeSpec *ast.StructType *ast.FieldList *ast.Field []*ast.Ident *ast.Ident ast.BasicLit struct we're
going to modify start line: 8 end line: 10 *ast.StructType -
gomodifytags *ast.StructType 6 7 8 9 10 11 12 *ast.Field
*ast.Field *ast.Field *ast.Field *ast.Field -
type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType
*ast.Field []*ast.Ident *ast.Ident *ast.BasicLit -
type Example struct { Foo string `json:"foo" } gomodifytags *ast.StructType
*ast.Field Foo string `json:"foo"` -
type Example struct { Foo string `json:"foo"` } gomodifytags *ast.StructType
*ast.Field Foo string `json:"foo"` -
type Example struct { Foo string `json:"bar" } gomodifytags *ast.StructType
*ast.Field Foo string `json:"foo" `json:"bar"` type Example struct { Foo string `json:"foo"` } -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = `json:"bar"` } return false }) After finding a field, replace the value -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = `json:"bar"` } return false }) Tag: `json:"foo"` Tag: `json:"bar"` After finding a field, replace the value -
gomodifytags *ast.StructType *ast.Field DiskSize string json:"disk_size" process() type Example struct
{ DiskSize string } type Example struct { DiskSize string `json:"disk_size"` } -
src := `package main type Example struct { Foo string`
+ " `json:\"foo\"` }" fset := token.NewFileSet() file, err := parser.ParseFile(fset, "demo", src, parser.ParseComments) if err != nil { panic(err) } ast.Inspect(file, func(x ast.Node) bool { s, ok := x.(*ast.StructType) if !ok { return true } for _, field := range s.Fields.List { // found field! field.Tag.Value = process(field) } return false }) tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) } -
gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field
select field between start and end lines ... Modify overview -
gomodifytags *ast.StructType 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field
... and rewrite selected fields tags 8 9 10 *ast.Field *ast.Field *ast.Field *ast.Field *ast.Field rewrite Modify overview -
package main type Example struct { Foo string } parse
find modify gomodifytags modify -
func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr
error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr } -
func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr
error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr } -
func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr
error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr } -
for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line
if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... } -
for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line
if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... } -
for _, f := range x.Fields.List { line := c.fset.Position(f.Pos()).Line
if !(c.start <= line && line <= c.end) { continue } if f.Tag == nil { f.Tag = &ast.BasicLit{} } // ... } -
for _, f := range x.Fields.List { fieldName := ""
if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res } -
for _, f := range x.Fields.List { fieldName := ""
if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res } -
for _, f := range x.Fields.List { fieldName := ""
if len(f.Names) != 0 { fieldName = f.Names[0].Name } if f.Names == nil { ident, ok := f.Type.(*ast.Ident) if !ok { continue // anonymous field } fieldName = ident.Name } res, err := c.process(fieldName, f.Tag.Value) if err != nil { rewriteErr = err return true } f.Tag.Value = res } -
func (c *config) process(fieldName, tagVal string) (string, error) { var
tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } -
func (c *config) process(fieldName, tagVal string) (string, error) { var
tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } -
func (c *config) process(fieldName, tagVal string) (string, error) { var
tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } -
func (c *config) process(fieldName, tagVal string) (string, error) { var
tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) } -
func (c *config) process(fieldName, tagVal string) (string, error) { var
tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } tags = c.removeTags(tags) tags, err = c.removeTagOptions(tags) if err != nil { return "", err } tags = c.clearTags(tags) tags = c.clearOptions(tags) tags, err = c.addTags(fieldName, tags) if err != nil { return "", err } tags, err = c.addTagOptions(tags) if err != nil { return "", err } if c.sort { sort.Sort(tags) } -
func (c *config) process(fieldName, tagVal string) (string, error) { var
tag string if tagVal != "" { var err error tag, err = strconv.Unquote(tagVal) if err != nil { return "", err } } tags, err := structtag.Parse(tag) if err != nil { return "", err } // process tags ... res := tags.String() if res != "" { res = quote(tags.String()) } return res, nil } -
func (c *config) rewriteFields(node ast.Node) (ast.Node, error) { var rewriteErr
error rewriteFunc := func(n ast.Node) bool { x, ok := n.(*ast.StructType) if !ok { return true } for _, f := range x.Fields.List { // process each field // ... } return true } ast.Inspect(node, rewriteFunc) return node, rewriteErr } -
package main type Example struct { Foo string } parse
find modify format gomodifytags -
func main() { var cfg config node = cfg.parse() start,
end = cfg.findSelection(node) rewritten = cfg.rewrite(node, start, end) out = cfg.format(rewritten) fmt.Println(out) } 1. Fetch configuration settings 2. Parse content 3. Find selection 4. Modify the struct tag 5. Output the result -
gomodifytags ast.Node (modified) go/format ... and output the result to
stdout package main type Example struct { Foo string `json:"foo"` } -
package main type Server struct { Name string Port int
EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } package main type Server struct { Name string `json:"name"` Port int `json:"port"` EnableLogs bool `json:"enable_logs"` BaseDomain string `json:"base_domain"` Credentials struct { Username string `json:"username"` Password string `json:"password"` } `json:"credentials"` } Input Output $ gomodifytags -file example.go -struct Server -add-tags json -
gomodifytags ast.Node (modified) go/format Also has support for custom JSON
output { "start": 3, "end": 5, "lines": [ "type Example struct {", " Foo string `json:\"foo\"`", "}" ] } -
package main type Server struct { Name string Port int
EnableLogs bool BaseDomain string Credentials struct { Username string Password string } } { "start": 3, "end": 12, "lines": [ "type Server struct {", " Name string `xml:\"name\"`", " Port int `xml:\"port\"`", " EnableLogs bool `xml:\"enable_logs\"`", " BaseDomain string `xml:\"base_domain\"`", " Credentials struct {", " Username string `xml:\"username\"`", " Password string `xml:\"password\"`", " } `xml:\"credentials\"`", "}" ] } Input JSON Output $ gomodifytags -file example.go -struct Server -add-tags json -format json -
Why stdout? • by default, it means "dry-run" • immediate
feedback • tools can redirect the output • composable (gomodifytags myfile.go > newfile.go) -
func (c *config) format(file ast.Node) (string, error) { switch c.output
{ case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result -
func (c *config) format(file ast.Node) (string, error) { switch c.output
{ case "source": // return formatted source case "json": // return only changes in json default: return "", fmt.Errorf("unknown output mode: %s", c.output) } } Output the result -
var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err
!= nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file -
var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err
!= nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file -
var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err
!= nil { return "", err } if c.write { err = ioutil.WriteFile(c.file, buf.Bytes(), 0) if err != nil { return "", err } } return buf.String(), nil Source file -
var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err
!= nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output -
var buf bytes.Buffer err := format.Node(&buf, c.fset, file) if err
!= nil { return "", err } var lines []string scanner := bufio.NewScanner(bytes.NewBufferString(buf.String())) for scanner.Scan() { lines = append(lines, scanner.Text()) } if c.start > len(lines) { return "", errors.New("line selection is invalid") } ... JSON output -
out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 :
c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.) -
out := &output{ Start: c.start, End: c.end, Lines: lines[c.start-1 :
c.end], } o, err := json.MarshalIndent(out, "", " ") if err != nil { return "", err } return string(o), nil JSON output (cont.) -
package main type Example struct { Foo string } parse
find modify gomodifytags format format -
package main type Example struct { Foo string } parse
find modify gomodifytags format Overview -
package main type Example struct { Foo string } -file
example.go -struct Example -add-tags json gomodifytags -
package main type Example struct { Foo string } parse
find gomodifytags -
package main type Example struct { Foo string } parse
find modify gomodifytags -
package main type Example struct { Foo string } parse
find modify format gomodifytags -
package main type Example struct { Foo string } package
main type Example struct { Foo string `json:"foo"` } parse find modify format gomodifytags -
Built from the ground up for editors • selecting structs
based on offset or range of lines • write result to stdout or file • json output for easy parsing • support for unsaved buffers • individual cli flags for all features -
Supported editors • vim with vim-go • atom with go-plus
(thanks Zac Bergquist) • vscode with vscode-go (thanks Ramya Rao) • emacs - WIP (https://github.com/syohex/emacs-go-add-tags/issues/ 6) -
Recap • A single tool to rule all editors •
Easy maintenance • Free of bugs due stdlib tooling (go/parser family) • Happy and productive users