Quick Overview of SwiftParser

わいわいswiftc #40

Transcript

1. Quick Overview of SwiftParser 1 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
2. Swift Compiler 2 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
3. Swift Compiler 3 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
4. Parsers → libParse in C++ for Swift → SwiftSyntax in Swift for Swift Macros 4 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
5. libParse 5 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
6. SwiftSyntax + ASTGen 6 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
7. Swift Macros Swift Macros implementations will be executed in a sandbox 7 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
8. Sandbox for Swift Macros 8 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
9. SwiftSyntax 9 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
10. SwiftSyntax SwiftSyntax is a set of Swift libraries for parsing, inspecting, generating, and transforming Swift source code. 10 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
11. SwiftSyntax 11 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
12. Mainline SwiftSyntax → SwiftParser for natively parsing source code → SwiftOperators for folding SwiftSyntax trees containing operators → SwiftSyntaxBuilder for generating Swift code with a result builder-style interface → SwiftSyntaxMacros for providing syntactic macro 12 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
13. SwiftParser The SwiftParser framework implements a parser that accepts Swift source text as input and produces a SwiftSyntax syntax tree. 13 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
14. SwiftParser 14 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
15. Parser.parse import SwiftParser import SwiftSyntax let sourceText = """ let a = 1 """ // Parse the source code in sourceText into a syntax tree let sourceFile: SourceFileSyntax = Parser.parse(source: sourceText) // Visualize the complete syntax tree. dump(sourceFile) 15 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
16. Source text let a = 1 16 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
17. Syntax tree // let a = 1 SourceFile CodeBlockItemList CodeBlockItem VariableDecl let PatternBindingList PatternBinding IdentifierPattern a InitializerClause = IntegerLiteralExpr 1 17 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
18. Lexer 18 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
19. Token // let a = 1 [ Lexer.Lexeme(tokenKind: .keyword, ...), Lexer.Lexeme(tokenKind: .identifier, ...), Lexer.Lexeme(tokenKind: .equal, ...), Lexer.Lexeme(tokenKind: .integerLiteral, ...), ] 19 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
20. TokenKind(RawTokenKind) public enum TokenKind: Hashable { case arrow case atSign case backslash case backtick case binaryOperator(String) case colon case comma ... } 20 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
21. Keyword public enum Keyword: UInt8, Hashable { case accesses case actor case addressWithNativeOwner case addressWithOwner case any case `Any` case `as` ... 21 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
22. How to lex number 22 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
23. Lexing Basics → peek(at:) → is(offset:, at), is(offset:, notAt) → advance(matching:), advance(if:), advance(while:) 23 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
24. lexNormal private mutating func lexNormal( sourceBufferStart: Lexer.Cursor, preferRegexOverBinaryOperator: Bool ) -> Lexer.Result { switch self.peek() { case UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), UInt8(ascii: "8"), UInt8(ascii: "9"): return self.lexNumber() ... } ... } 24 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
25. lexNormal private mutating func lexNormal( sourceBufferStart: Lexer.Cursor, preferRegexOverBinaryOperator: Bool ) -> Lexer.Result { switch self.peek() { case UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), UInt8(ascii: "8"), UInt8(ascii: "9"): return self.lexNumber() ... } ... } 25 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
26. lexNormal private mutating func lexNormal( sourceBufferStart: Lexer.Cursor, preferRegexOverBinaryOperator: Bool ) -> Lexer.Result { switch self.peek() { case UInt8(ascii: "0"), UInt8(ascii: "1"), UInt8(ascii: "2"), UInt8(ascii: "3"), UInt8(ascii: "4"), UInt8(ascii: "5"), UInt8(ascii: "6"), UInt8(ascii: "7"), UInt8(ascii: "8"), UInt8(ascii: "9"): return self.lexNumber() ... } ... } 26 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
27. lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 27 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
28. lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 28 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
29. lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 29 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
30. lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 30 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
31. lexNumber mutating func lexNumber() -> Lexer.Result { // Handle a leading [0-9]+, lexing an integer or falling through if we have a // floating point value. self.advance(while: { $0.isDigit || $0 == Unicode.Scalar("_") }) if self.is(at: ".") { if self.peek(at: 1) == nil { // If there are no more digits following the '.', we don't have a float // literal. return Lexer.Result(.integerLiteral) } ... } else if self.isAtEndOfFile || self.is(notAt: "e", "E") { ... return Lexer.Result(.integerLiteral) } ... return Lexer.Result(.floatingLiteral) } 31 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
32. Parser 32 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
33. Syntax tree // let a = 1 SourceFile CodeBlockItemList CodeBlockItem VariableDecl let PatternBindingList PatternBinding IdentifierPattern a InitializerClause = IntegerLiteralExpr 1 33 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
34. SyntaxNodes → SyntaxDeclNodes → SyntaxExprNodes → SyntaxNodes → SyntaxPatternNodes → SyntaxStmtNodes → SyntaxTypeNodes 34 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
35. SyntaxKind public enum SyntaxKind: CaseIterable { case token case accessorBlock case accessorDeclList case accessorDecl case accessorEffectSpecifiers case accessorParameters case actorDecl case arrayElementList ... } 35 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
36. How to parse integer literal 36 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
37. Parsing Basics → peek(isAt:) → at(anyIn:), at(prefix:) → eat → consume(if:), consume(ifAnyIn:), consume(ifPrefix:, as) 37 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
38. parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 38 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
39. parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 39 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
40. parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 40 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
41. parsePrimaryExpression mutating func parsePrimaryExpression( pattern: PatternContext, forDirective: Bool, flavor: ExprFlavor ) -> RawExprSyntax { switch self.at(anyIn: PrimaryExpressionStart.self) { case (.integerLiteral, let handle)?: let literal = self.eat(handle) return RawExprSyntax( RawIntegerLiteralExprSyntax( literal: literal, arena: self.arena ) ) ... } ... } 41 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)
42. Recap → Much readable than C++ → Swift Macros → Performance comparison with libParse 42 — Quick Overview of SwiftParser, Yusuke Kita (@kitasuke)