Syntax Tree (RubyConf Mini)

Syntax Tree

Kevin Newton @kddnewton

Syntax Tree

What is Syntax Tree?

Syntax Tree concepts

Parser Syntax Tree concepts

Parser Syntax tree Syntax Tree concepts

Parser Syntax tree Visitor Syntax Tree concepts

Parser · Syntax tree · Visitor

Parser · Syntax tree · Visitor Parser

Matz is nice so we are nice. Parser · Syntax
tree · Visitor

Lexical analysis Parser · Syntax tree · Visitor

Matz is nice so we are nice. Parser · Syntax
tree · Visitor

Matz is nice so we are nice . Parser ·
Syntax tree · Visitor

is nice so we are nice . Matz Parser ·

Matz is nice we are nice . so Parser ·

Matz is nice so we are . nice Parser ·

Semantic analysis Parser · Syntax tree · Visitor

Parser · Syntax tree · Visitor is nice are nice

Grammar Parser · Syntax tree · Visitor

Grammar Parser · Syntax tree · Visitor verb-phrase : VERB
ADJECTIVE

Parser · Syntax tree · Visitor is nice are nice

Parser · Syntax tree · Visitor is nice verb phrase
verb phrase are nice

Parser · Syntax tree · Visitor Matz is nice verb
phrase we verb phrase are nice

ADJECTIVE

ADJECTIVE   subject-phrase : NOUN verb-phrase

Parser · Syntax tree · Visitor Matz is nice verb
phrase we verb phrase are nice

Parser · Syntax tree · Visitor Matz is nice subject
phrase subject phrase verb phrase we verb phrase are nice

Parser · Syntax tree · Visitor Matz is nice so
subject phrase subject phrase verb phrase we verb phrase are nice

ADJECTIVE   subject-phrase : NOUN verb-phrase

ADJECTIVE   subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase

subject phrase subject phrase verb phrase we verb phrase are nice

subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

. subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

verb-phrase : VERB ADJECTIVE   subject-phrase : NOUN verb-phrase  
subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase Grammar Parser · Syntax tree · Visitor

subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase   sentence : subordinating-conjunction PERIOD Grammar Parser · Syntax tree · Visitor

. subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

. sentence subord. conjunction subject phrase subject phrase verb phrase we verb phrase are nice

Parser · Syntax tree · Visitor Syntax tree

Parser · Syntax tree · Visitor verb-phrase : VERB ADJECTIVE
subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD

Parser · Syntax tree · Visitor class VerbPhrase attr_reader :verb,
:adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD

subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

Parser · Syntax tree · Visitor class SubjectPhrase attr_reader :noun,
:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end verb-phrase : VERB ADJECTIVE subject-phrase : NOUN verb-phrase subordinating-conjunction :   subject-phrase CONJUNCTION subject-phrase sentence : subordinating-conjunction PERIOD class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end

:verb_phrase def initialize(noun:, verb_phrase:) @noun = noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end

class SubjectPhrase attr_reader :noun, :verb_phrase def initialize(noun:, verb_phrase:) @noun =
noun @verb_phrase = verb_phrase end end class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end end class SubordinatingConjunction attr_reader :left, :conjunction, :right def initialize(left:, conjunction:, right:) @left = left @conjunction = conjunction @right = right end end class Sentence attr_reader :phrase, :punctuation def initialize(phrase:, punctuation:) @phrase = phrase @punctuation = punctuation end end class Verb attr_reader :value def initialize(value:) @value = value end end class Adjective attr_reader :value def initialize(value:) @value = value end end class Noun attr_reader :value def initialize(value:) @value = value end end class Conjunction attr_reader :value def initialize(value:) @value = value end end class Period attr_reader :value def initialize(value:) @value = value end end Parser · Syntax tree · Visitor

class VerbPhrase attr_reader :verb, :adjective def initialize(verb:, adjective:) @verb =
verb @adjective = adjective end end Parser · Syntax tree · Visitor

:adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def deconstruct_keys(keys) { verb: verb, adjective: adjective } end def ==(other) other in VerbPhrase[ verb: ^(verb), adjective: ^(adjective) ] end end

:adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def copy(verb: self.verb, adjective: self.adjective) VerbPhrase.new(verb: verb, adjective: adjective) end end

:adjective def initialize(verb:, adjective:) @verb = verb @adjective = adjective end def accept(visitor) visitor.visit_verb_phrase(self) end def child_nodes [verb, adjective] end end

Visitor Parser · Syntax tree · Visitor

verb @adjective = adjective end def accept(visitor) visitor.visit_verb_phrase(self) end def child_nodes [verb, adjective] end end Parser · Syntax tree · Visitor

class VerbPhrase def accept(visitor) visitor.visit_verb_phrase(self) end end Parser · Syntax
tree · Visitor

class VerbPhrase def accept(visitor) visitor.visit_verb_phrase(self) end end   class Visitor
def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

class VerbPhrase def accept(visitor) visitor.visit_verb_phrase(self) end end   class Visitor
def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

class Visitor def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node)
end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

class Visitor def visit(node) node.accept(self) end def visit_adjective(node) end def
visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) # ... end end node.accept(Visitor.new) Parser · Syntax tree · Visitor

Parser · Syntax tree · Visitor class Visitor def visit(node)
node.accept(self) end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) node.child_nodes.each { |node| visit(node) } end end node.accept(Visitor.new)

node.accept(self) end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) node.child_nodes.each { |node| visit(node) } end end

node.accept(self) end def visit_child_nodes(node) node.child_nodes.each { |node| visit(node) } end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end def visit_verb_phrase(node) visit_child_nodes(node) end end

node.accept(self) end def visit_child_nodes(node) node.child_nodes.each { |node| visit(node) } end def visit_adjective(node) end def visit_conjunction(node) end def visit_noun(node) end def visit_verb(node) end def visit_punctuation(node) end def visit_sentence(node) end def visit_subordinating_conjunction(node) end def visit_subject_phrase(node) end alias visit_verb_phrase visit_child_nodes end

node.accept(self) end def visit_child_nodes(node) node.child_nodes.each { |node| visit(node) } end alias visit_adjective visit_child_nodes alias visit_conjunction visit_child_nodes alias visit_noun visit_child_nodes alias visit_verb visit_child_nodes alias visit_punctuation visit_child_nodes alias visit_sentence visit_child_nodes alias visit_subordinating_conjunction   visit_child_nodes alias visit_subject_phrase visit_child_nodes alias visit_verb_phrase visit_child_nodes end

Parser · Syntax tree · Visitor

Parser · Syntax tree · Visitor class SentenceCountVisitor < Visitor
attr_reader :count def initialize @count = 0 end def visit_sentence(node) @count += 1 super end end visitor = SentenceCountVisitor.new node.accept(visitor) puts visitor.count

Parser · Syntax tree · Visitor class PrettyPrintVisitor < Visitor
attr_reader :q def initialize @q = PP.new(+"") end def visit_sentence(node) q.group(2, "sentence(", ")") do visit(node.phrase) visit(node.punctuation) end end end visitor = PrettyPrintVisitor.new node.accept(visitor) visitor.q.flush puts visitor.q.output

Parser · Syntax tree · Visitor class FormatterVisitor < Visitor
attr_reader :q def initialize @q = PP.new(+"") end def visit_sentence(node) q.group do q.format(node.phrase) q.format(node.punctuation) end end end visitor = FormatterVisitor.new node.accept(visitor) visitor.q.flush puts visitor.q.output

What is Syntax Tree?

Syntax Tree

Object layer representing the result of parsing Ruby Syntax Tree

Object layer representing the result of parsing Ruby Tools to
interact with and manipulate that layer Syntax Tree

Build a syntax tree Syntax Tree

Build a syntax tree Format the syntax tree Syntax Tree

Build a syntax tree Format the syntax tree Create a
CLI Syntax Tree

CLI Create a language server (LSP) Syntax Tree

CLI Create a language server (LSP) Translate the syntax tree Syntax Tree

Build · Format · CLI · LSP · Translate Syntax
Tree

Build a syntax tree Build · Format · CLI ·
LSP · Translate

Define each node Build · Format · CLI · LSP
· Translate

Build · Format · CLI · LSP · Translate class
VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end

VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields

VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments

VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments #accept/#child_nodes

VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments #accept/#child_nodes #deconstruct/   #deconstruct_keys

VCall < Node attr_reader :value, :location, :comments def initialize(value:, location:, comments: []) @value = value @location = location @comments = comments end def accept(visitor) visitor.visit_vcall(self) end def child_nodes [value] end alias deconstruct child_nodes def deconstruct_keys(keys) { value: value, location: location, comments: comments } end end Named fields #location/#comments #accept/#child_nodes #deconstruct/   #deconstruct_keys Immutable

Define each node Build · Format · CLI · LSP
· Translate

Ripper::EVENTS.count # => 190 Define each node Build · Format
· CLI · LSP · Translate

Ripper::EVENTS.count # => 190 SyntaxTree::Node.descendants.count   # => 162 Define
each node Build · Format · CLI · LSP · Translate

Ripper::EVENTS.count # => 190 SyntaxTree::Node.descendants.count   # => 162 qwords_new/qwords_add
-> SyntaxTree::QWords Define each node Build · Format · CLI · LSP · Translate

How we use ripper Build · Format · CLI ·
LSP · Translate

var_ref : user_variable { /*%%%*/ if (!($$ = gettable(p, $1,
&@$))) $$ = NEW_BEGIN(0, &@$); /*% if (id_is_var(p, get_id($1))) { $$ = dispatch1(var_ref, $1); } else { $$ = dispatch1(vcall, $1); } %*/ } | keyword_variable { /*%%%*/ if (!($$ = gettable(p, $1, &@$))) $$ = NEW_BEGIN(0, &@$); /*% %*/ /*% ripper: var_ref!($1) %*/ } ; Build · Format · CLI · LSP · Translate

class Foo def bar baz end end Build · Format

· CLI · LSP · Translate class Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) [:class, [const, superclass, bodystmt]] end def on_def(name, params, bodystmt) [:def, [name, params, bodystmt]] end end

Parser < Ripper def on_const(value) = [:@const, value] def on_ident(value) = [:@ident, value] def on_kw(value) = @stack << [:@kw, value] def on_const_ref(const) = [:const_ref, const] def on_vcall(ident) = [:vcall, ident] def on_class(const, superclass, bodystmt) kw = @stack.rfind { |token| token in [:@kw, "class"] } [:class, [@stack.delete(kw), const, superclass, bodystmt]] end def on_def(name, params, bodystmt) kw = @stack.rfind { |token| token in [:@kw, "def"] } [:def, [@stack.delete(kw), name, params, bodystmt]] end end

Build · Format · CLI · LSP · Translate ->
(positional; local1, local2) {}

Build · Format · CLI · LSP · Translate ->
(positional; local1, local2) {} range = (location.start_char + 1)...location.end_char locals = lambda_locals(source[range]) Paren.new( lparen: params.lparen, contents: LambdaVar.new( params: params.contents, locals: locals, location: location ), location: params.location, comments: params.comments )

Foo # comment def bar baz end end

Foo def bar # comment baz end end

Foo def bar baz # comment end end

Build · Format · CLI · LSP · Translate #
comment class Foo def bar baz end end

Foo # comment def bar baz end end

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ ]

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, ]

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, ]

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, # comment3, ]

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, # comment3, # comment4, ]

comment1 class Foo # comment2 # comment3 def bar # comment4 baz # comment5 end end [ # comment1, # comment2, # comment3, # comment4, # comment5 ]

Build · Format · CLI · LSP · Translate [
# comment1, # comment2, # comment3, # comment4, # comment5 ] (program (statements ((class (const_ref (const "Foo")) (bodystmt (statements ((void_stmt), (def (ident "bar") (params) (bodystmt (statements ((vcall (ident "baz")))))))))))))

Build · Format · CLI · LSP · Translate (program
(statements ((comment "# comment1"), (class (const_ref (const "Foo") ((comment "# comment2"))) (bodystmt (statements ((void_stmt), (comment "# comment3"), (def (ident "bar") (params ((comment "# comment4"))) (bodystmt (statements ((vcall (ident "baz") ((comment "# comment5"))))))))))))))

Walking the tree Build · Format · CLI · LSP
· Translate

Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

VCall < Node def accept(visitor) visitor.visit_vcall(self) end end class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

VCall < Node def accept(visitor) visitor.visit_vcall(self) end end # Visit an ARef node. alias visit_aref visit_child_nodes # Visit an ARefField node. alias visit_aref_field visit_child_nodes # Visit an Alias node. alias visit_alias visit_child_nodes # Visit an ArgBlock node. alias visit_arg_block visit_child_nodes # Visit an ArgParen node. alias visit_arg_paren visit_child_nodes # Visit an ArgStar node. alias visit_arg_star visit_child_nodes # Visit an Args node. alias visit_args visit_child_nodes # Visit an ArgsForward node. alias visit_args_forward visit_child_nodes # Visit an ArrayLiteral node. alias visit_array visit_child_nodes # Visit an AryPtn node. alias visit_aryptn visit_child_nodes # Visit an Assign node. alias visit_assign visit_child_nodes # Visit an Assoc node. alias visit_assoc visit_child_nodes # Visit an AssocSplat node. alias visit_assoc_splat visit_child_nodes # Visit a Backref node. alias visit_backref visit_child_nodes class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

VCall < Node def accept(visitor) visitor.visit_vcall(self) end end alias visit_vcall visit_child_nodes # Visit a VoidStmt node. alias visit_void_stmt visit_child_nodes # Visit a When node. alias visit_when visit_child_nodes # Visit a While node. alias visit_while visit_child_nodes # Visit a WhileMod node. alias visit_while_mod visit_child_nodes # Visit a Word node. alias visit_word visit_child_nodes # Visit a Words node. alias visit_words visit_child_nodes # Visit a WordsBeg node. alias visit_words_beg visit_child_nodes # Visit a XString node. alias visit_xstring visit_child_nodes # Visit a XStringLiteral node. alias visit_xstring_literal visit_child_nodes # Visit a Yield node. alias visit_yield visit_child_nodes # Visit a Yield0 node. alias visit_yield0 visit_child_nodes # Visit a ZSuper node. alias visit_zsuper visit_child_nodes # Visit an EndContent node. alias visit___end__ visit_child_nodes class Visitor def visit(node) node&.accept(self) end def visit_child_nodes(node) node.child_nodes.map do |node| visit(node) end end end

RegexpVisitor < Visitor def visit_regexp_literal(node) # do something with regexp literal node here node.options # call super to continue walking the tree super end end

JSONVisitor < Visitor def visit_aref(node) node(node, "aref") do field("collection", node.collection) field("index", node.index) comments(node) end end def visit_aref_field(node) node(node, "aref_field") do field("collection", node.collection) field("index", node.index) comments(node) end end def visit_alias(node) node(node, "alias") do field("left", node.left) field("right", node.right) comments(node) end end

Format the syntax tree Build · Format · CLI ·
LSP · Translate

Algorithm Build · Format · CLI · LSP · Translate

prettyprint Algorithm Build · Format · CLI · LSP ·
Translate

prettyprint Build group, text, breakable to layout content Algorithm Build
· Format · CLI · LSP · Translate

prettyprint Build group, text, breakable to layout content Break outermost
groups to fit to print width Algorithm Build · Format · CLI · LSP · Translate

Build · Format · CLI · LSP · Translate [foo1,
foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2]]

foo1, foo2, [bar1, bar2, bar3, bar4], foo3, foo4, [baz1, baz2] ]

foo1, foo2, [ bar1, bar2, bar3, bar4 ], foo3, foo4, [baz1, baz2] ]

breakGroup([group([text(["["]), indent([breakable, text(["foo1", ","]), breakable, text(["foo2", ","]), breakable, group([text(["["]), indent([breakable,
text(["bar1", ","]), breakable, text(["bar2", ","]), breakable, text(["bar3", ","]), breakable, text(["bar4"])]), breakable, text(["]"])]), text([","]), breakable, text(["foo3", ","]), breakable, text(["foo4", ","]), breakable, ... Build · Format · CLI · LSP · Translate

I made too many additions to prettyprint Build · Format
· CLI · LSP · Translate 😅😅😅

prettyprint Build · Format · CLI · LSP · Translate

prettier_print Build · Format · CLI · LSP · Translate

Build · Format · CLI · LSP · Translate if
foo bar += baz end

Build · Format · CLI · LSP · Translate bar
+= baz if foo

Build · Format · CLI · LSP · Translate if
foo bar else baz end

Build · Format · CLI · LSP · Translate foo
? bar : baz

Build · Format · CLI · LSP · Translate here_is_a_method_call(1,
2, <<~HEREDOC this is the content of the heredoc HEREDOC )

Build · Format · CLI · LSP · Translate here_is_a_method_call(1,
2, <<~HEREDOC) this is the content of the heredoc HEREDOC

Build · Format · CLI · LSP · Translate def
foo return [1, 2, 3].map { |element| element * 2 } # comment end

Build · Format · CLI · LSP · Translate def
foo return( [1, 2, 3].map do |element| element * 2 end ) # comment end

If < Node def format(q) if node.consequent || node.statements.empty? q.group { format_break(q, force: true) } else q.group do q .if_break { format_break(q, force: false) } .if_flat do Parentheses.flat(q) do q.format(node.statements) q.text(" if ") q.format(node.predicate) end end end end end end

Create a CLI Build · Format · CLI · LSP
· Translate

Build · Format · CLI · LSP · Translate stree
ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree check [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Check that the given files are formatted as syntax tree would format them stree debug [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Check that the given files can be formatted idempotently stree doc [--plugins=...] [-e SCRIPT] FILE Print out the doc tree that would be used to format the given files stree expr [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the first expression of the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree json [--plugins=...] [-e SCRIPT] FILE Print out the JSON representation of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree help Display this help message stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files stree version Output the current version of syntax tree stree write [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Read, format, and write back the source of the given files --plugins=... A comma-separated list of plugins to load. --print-width=NUMBER The maximum line width to use when formatting. -e SCRIPT Parse an inline Ruby string.

ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files

Build · Format · CLI · LSP · Translate $
stree ast -e '1+2'

stree ast -e '1+2' (program (statements ((binary (int "1") + (int "2")))))

stree format -e '1+2'

stree format -e '1+2' 1 + 2

stree match -e '1+2'

stree match -e '1+2' SyntaxTree::Program[ statements: SyntaxTree::Statements[ body: [ SyntaxTree::Binary[ left: SyntaxTree::Int[value: "1"], operator: :+, right: SyntaxTree::Int[value: "2"] ] ] ] ]

stree search Int -e '1+2'

stree search Int -e '1+2' script:1:0: 1+2 script:1:2: 1+2

stree search Int -e '1+2'

stree search 'Int[value: "2"]' -e '1+2'

stree search 'Int[value: "2"]' -e '1+2'     script:1:2: 1+2

stree search <(stree match -e '1+2') \   -e '1+2'

stree search <(stree match -e '1+2') \   -e '1+2'     script:1:0: 1+2

ast [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the AST corresponding to the given files stree format [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Print out the formatted version of the given files stree match [--plugins=...] [-e SCRIPT] FILE Print out a pattern-matching Ruby expression that would match the given files stree lsp [--plugins=...] [--print-width=NUMBER] Run syntax tree in language server mode stree search PATTERN [-e SCRIPT] FILE Search for the given pattern in the given files stree write [--plugins=...] [--print-width=NUMBER] [-e SCRIPT] FILE Read, format, and write back the source of the given files

Create a language server Build · Format · CLI ·
LSP · Translate

Language server protocol Build · Format · CLI · LSP
· Translate

Language server protocol Runs a JSON RPC server behind an
editor Build · Format · CLI · LSP · Translate

editor textDocument/didChange textDocument/didOpen textDocument/didClose Build · Format · CLI · LSP · Translate

editor textDocument/didChange textDocument/didOpen textDocument/didClose textDocument/formatting Build · Format · CLI · LSP · Translate

editor textDocument/didChange textDocument/didOpen textDocument/didClose textDocument/formatting textDocument/inlayHint Build · Format · CLI · LSP · Translate

Build · Format · CLI · LSP · Translate

Shopify/ruby-lsp Build · Format · CLI · LSP · Translate

Shopify/ruby-lsp Document highlight Build · Format · CLI · LSP
· Translate

Shopify/ruby-lsp Document highlight Document symbols Build · Format · CLI
· LSP · Translate

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Build · Format

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Formatting Build ·
Format · CLI · LSP · Translate

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Formatting Selection ranges
Build · Format · CLI · LSP · Translate

Shopify/ruby-lsp Document highlight Document symbols Folding ranges Formatting Selection ranges
Semantic highlighting Build · Format · CLI · LSP · Translate

Translate the syntax tree Build · Format · CLI ·
LSP · Translate

Translate the syntax tree seattlerb/ruby_parser Build · Format · CLI
· LSP · Translate

RubyParserVisitor < Visitor def visit_binary(node) left = visit(node.left) right = visit(node.right) case node in { operator: :and } s(:and, left, right) in { operator: :!~ } s(:not, s(:call, left, :=~, right)) else s(:call, left, node.operator, right) end end # ... end

Translate the syntax tree seattlerb/ruby_parser Build · Format · CLI
· LSP · Translate

Translate the syntax tree seattlerb/ruby_parser whitequark/parser Build · Format ·
CLI · LSP · Translate

ParserVisitor < Visitor def visit_binary(node) left = visit(node.left) right = visit(node.right) case node in { operator: :and } s(:and, [left, right]) in { operator: :or } s(:or, [left, right]) else s(:send, [left, node.operator, right]) end end # ... end

Translate the syntax tree seattlerb/ruby_parser whitequark/parser Build · Format ·
CLI · LSP · Translate

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Build · Format

YARVVisitor < Visitor def visit_binary(node) case node.operator when :"&&" # ... when :"||" # ... else visit(node.left) visit(node.right) builder.send( node.operator, 1, VM_CALL_ARGS_SIMPLE ) end end # ... end

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Build · Format

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Ruby Build ·

MutationVisitor < Visitor def visit_binary(node) node.copy( left: visit(node.left), right: visit(node.right) ) end end

Build · Format · CLI · LSP · Translate mutator
= SyntaxTree.mutation do |mutator| query = <<~RUBY IfNode[predicate: Assign | OpAssign] | UnlessNode[predicate: Assign | OpAssign] RUBY mutator.mutate(query) do |node| node.copy(predicate: SyntaxTree::Paren.new( lparen: SyntaxTree::LParen.default, contents: node.predicate, location: node.predicate.location )) end end mutated = node.accept(mutator)

Translate the syntax tree seattlerb/ruby_parser whitequark/parser YARV Ruby Build ·

Syntax Tree

Parser Syntax Tree

Parser Syntax tree Syntax Tree

Parser Syntax tree Visitor Syntax Tree

Syntax Tree

Syntax Tree Build a syntax tree

Syntax Tree Build a syntax tree Format the syntax tree

Create a CLI

Create a CLI Create a language server (LSP)

Create a CLI Create a language server (LSP) Translate the syntax tree

Thank you!

Syntax Tree github.com/ruby-syntax-tree Kevin Newton @kddnewton

Syntax Tree (RubyConf Mini)

Syntax Tree (RubyConf Mini)

More Decks by Kevin Newton

Featured

Transcript