Encrypting the Internet with Go @ GopherCon 2017

For a few months this year, a lot of your Internet requests might have gone through Go. That's because at Cloudflare we picked crypto/tls to build our TLS 1.3 implementation.

Video: https://www.youtube.com/watch?v=CB_VfgwPmxQ

1. Encrypting the Internet with Go Filippo Valsorda
3. TLS 1.2 ECDHE Client Hello Supported cipher suites Client Server Server Hello Chosen cipher suite Key share Certificate & signature Key share Finished Finished HTTP GET HTTP Answer
4. Client Hello Supported AEAD / groups / signatures Key share Server Hello Chosen AEAD Key share Finished Certificate & signature Finished HTTP GET HTTP Answer TLS 2 Client Server 4 Vista 2017 1.3
5. Client Hello Session Ticket (PSK) Key share Server Hello Key share Finished HTTP GET HTTP Answer Finished 0-RTT Client Server
6. crypto/tls += TLS 1.3
7. package tls // import "crypto/tls" type Conn struct { // Has unexported fields. } A Conn represents a secured connection. It implements the net.Conn interface. func Client(conn net.Conn, config *Config) *Conn func Server(conn net.Conn, config *Config) *Conn
8. func (c *Conn) Handshake() error
9. func (c *Conn) Read(b []byte) (n int, err error) { if err = c.Handshake(); err != nil { return }
10. tls.Conn net.Conn Handshake() Initial Handshake
11. tls.Conn net.Conn Handshake() cipher Late Handshake
12. net.Conn cipher Handshake Complete tls.Conn
13. func (c *Conn) serverHandshake() error { msg, err := c.readHandshake() if err != nil { return false, err }
14. func (c *Conn) readHandshake() (interface{}, error) { for c.hand.Len() < 4 { if err := c.readRecord(recordTypeHandshake); err != nil { return nil, err } } data := c.hand.Bytes() n := int(data[1])<<16 | int(data[2])<<8 | int(data[3]) for c.hand.Len() < 4+n { if err := c.readRecord(recordTypeHandshake); err != nil { return nil, err } }
15. TLS record layer Type Version Length Ciphertext Type Padding Plaintext payload
16. func (hc *halfConn) decrypt(b *block) (ok bool) { payload := b.data[recordHeaderLen:] switch c := hc.cipher.(type) { case cipher.Stream: c.XORKeyStream(payload, payload) case aead: explicitIVLen := c.explicitNonceLen() nonce := payload[:explicitIVLen] payload = payload[explicitIVLen:] payload, err = c.Open(payload[:0], nonce, payload, nil) if err != nil { return false, 0, alertBadRecordMAC } b.resize(recordHeaderLen + len(payload)) }
17. func (c *Conn) readRecord(want recordType) error { switch typ { case recordTypeAlert: c.in.setErrorLocked(&net.OpError{ Op: "remote error", Err: alert(data[1])}) case recordTypeApplicationData: c.input = b b = nil case recordTypeHandshake: c.hand.Write(data) }
18. func (c *Conn) readHandshake() (interface{}, error) { var m handshakeMessage switch data[0] { case typeClientHello: m = new(clientHelloMsg) case typeServerHello: m = new(serverHelloMsg) case typeNewSessionTicket: m = new(newSessionTicketMsg) case typeCertificate: m = new(certificateMsg) ... m.unmarshal(data)
19. func (c *Conn) serverHandshake() error { clientFinished, ok := msg.(*finishedMsg) if !ok { c.sendAlert(alertUnexpectedMessage) return unexpectedMessageErr(clientFinished, msg) }
20. type handshakeStatus int const ( handshakeRunning handshakeStatus = iota discardingEarlyData readingEarlyData waitingClientFinished readingClientFinished handshakeConfirmed ) TLS 1.3 handshake states
21. 0-RTT API 0-RTT data is different, because it might be replayed. The server needs: • To know what part of the data is 0-RTT • To know when it’s safe to use the 0-RTT data
22. type Config struct { Accept0RTTData bool Option 1: just a Config knob No way of knowing which part is the 0-RTT data.
23. func (*Conn) HandshakeWith0RTT() (io.Reader, error) Option 2: a separate function Breaks io.Reader! Imagine wrapping a tls.Conn in gzip.Reader.
24. type Config struct { Is0RTTSafe func(io.Reader) bool Option 3: a check function Subtly breaks io.Reader. You’d have to manually parse HTTP.
25. func (c *Conn) ConnectionState() ConnectionState type ConnectionState struct { HandshakeConfirmed bool Option 4: a ConnectionState field Ok, but how do I wait for it to be “confirmed”?
26. func (c *Conn) ConfirmHandshake() error Option 5: ConfirmHandshake You don’t actually know which part exactly is 0-RTT, but you know if what you read so far is safe, and can wait otherwise. (Sadly, it must buffer in the slow path.)
27. TLSConnContextKey = &contextKey{"tls-conn"} Exposing it to the HTTP handler Like ServerContextKey and LocalAddrContextKey. The HTTP handler needs access to the live ConnectionState and the ConfirmHandshake method.
28. Other API changes None. TLS 1.3 is about sane defaults, so I’d like to see how far we get with just that. Full diff: https://gist.github.com/FiloSottile/37d6516af411582e2aa35a981bf12102
29. Write() Read() Close() Handshake() ConfirmHandshake()
30. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex
31. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex blocks
32. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex blocks locks
33. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex blocks locks
34. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex c.confirmMutex blocks might lock locks
35. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex c.confirmMutex blocks might lock locks locks
36. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex c.confirmMutex blocks might lock locks locks
37. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex c.confirmMutex blocks might lock locks unblocks locks
38. Write() Read() Close() Handshake() ConfirmHandshake() c.in.Mutex c.out.Mutex c.handshakeMutex c.confirmMutex blocks might lock locks https://golang.org/cl/33776 unblocks locks
40. IP=$(docker inspect -f '{{ .NetworkSettings.IPAddress }}' tris-localserver) docker run --rm tls-tris:$2 $IP:3443 | tee output.txt # rejecting 0-RTT grep "Hello TLS 1.3" output.txt | grep "resumed" | grep -v "0-RTT" docker run --rm tls-tris:$2 $IP:4443 | tee output.txt # accepting 0-RTT grep "Hello TLS 1.3" output.txt | grep "resumed" | grep "0-RTT" docker run --rm tls-tris:$2 $IP:5443 | tee output.txt # confirming 0-RTT grep "Hello TLS 1.3" output.txt | grep "resumed" | grep "0-RTT confirmed” Interoperability testing
41. .PHONY: GOROOT GOROOT: GOROOT/$(GOENV)/.ok_$(GOROOTINFO) @rm -f GOROOT/$(GOENV)/pkg/*/crypto/tls.a GOROOT/$(GOENV)/.ok_$(GOROOTINFO): rm -rf GOROOT/$(GOENV) cp -r "$(shell $(GO) env GOROOT)/src" GOROOT/$(GOENV)/src cp -r "$(shell $(GO) env GOROOT)/pkg/include" GOROOT/$(GOENV)/pkg/include cp -r "$(shell $(GO) env GOROOT)/pkg/tool" GOROOT/$(GOENV)/pkg/tool rm -r GOROOT/$(GOENV)/src/crypto/tls ln -s ../../../../.. GOROOT/$(GOENV)/src/crypto/tls GOROOT="$(CURDIR)/GOROOT/$(GOENV)" $(GO) install -v std @touch "$@" Patching the standard library
42. GO ?= go GOENV := $(shell $(GO) env GOHOSTOS)_$(shell $(GO) env GOHOSTARCH) GOROOTINFO := $(shell $(GO) version | cut -d' ' -f 3)_$(GOENV) .PHONY: GOROOT GOROOT: GOROOT/$(GOENV)/.ok_$(GOROOTINFO) @rm -f GOROOT/$(GOENV)/pkg/*/crypto/tls.a GOROOT/$(GOENV)/.ok_$(GOROOTINFO): rm -rf GOROOT/$(GOENV) mkdir -p GOROOT/$(GOENV)/pkg cp -r "$(shell $(GO) env GOROOT)/src" GOROOT/$(GOENV)/src cp -r "$(shell $(GO) env GOROOT)/pkg/include" GOROOT/$(GOENV)/pkg/include cp -r "$(shell $(GO) env GOROOT)/pkg/tool" GOROOT/$(GOENV)/pkg/tool rm -r GOROOT/$(GOENV)/src/crypto/tls ln -s ../../../../.. GOROOT/$(GOENV)/src/crypto/tls GOROOT="$(CURDIR)/GOROOT/$(GOENV)" $(GO) install -v std ifeq ($(shell go env CGO_ENABLED),1) GOROOT="$(CURDIR)/GOROOT/$(GOENV)" $(GO) install -race -v std endif @touch "$@" # Note: when changing this, if it doesn't change the Go version # (it should), you need to run make clean. GO_COMMIT := 98882e950cbb48ce7aad4b5b9972601d3438fbe5 .PHONY: go go: go/.ok_$(GO_COMMIT)_$(GOENV) go/.ok_$(GO_COMMIT)_$(GOENV): rm -rf go/.ok_*_$(GOENV) go/$(GOENV) mkdir -p go git clone --branch 1.8 --single-branch --depth 25 https://github.com/cloudflare/go go/$(GOENV) cd go/$(GOENV) && git checkout $(GO_COMMIT) cd go/$(GOENV)/src && GOROOT_BOOTSTRAP="$(shell $(GO) env GOROOT)" ./make.bash @touch "$@" .PHONY: clean clean: rm -rf GOROOT go Patching the standard library
43. #!/usr/bin/env bash set -e BASEDIR=$(cd "$(dirname "$0")" && pwd) make --quiet -C "$BASEDIR" go >&2 GOENV="$(go env GOHOSTOS)_$(go env GOHOSTARCH)" export GOROOT="$BASEDIR/go/$GOENV" make --quiet -C "$BASEDIR" GOROOT GO="$BASEDIR/go/$GOENV/bin/go" >&2 export GOROOT="$BASEDIR/GOROOT/$GOENV" exec $BASEDIR/go/$GOENV/bin/go "$@" Patching the standard library
44. crypto/tls in production
45. type Config struct { GetCertificate func(*ClientHelloInfo) (*Certificate, error) GetCertificate
46. Keyless certificates
47. type Certificate struct { Certificate [][]byte PrivateKey crypto.PrivateKey // crypto.Signer / crypto.Decrypter OCSPStaple []byte SignedCertificateTimestamps [][]byte Leaf *x509.Certificate } Keyless and GetCertificate See github.com/cloudflare/gokeyless/client
48. type Config struct { NextProtos []string ClientAuth ClientAuthType Accept0RTTData bool Where GetCertificate can’t go
49. type Config struct { GetConfigForClient func(*ClientHelloInfo) (*Config, error) Enter GetConfigForClient https://golang.org/issues/15707
50. type ClientHelloInfo struct { CipherSuites []uint16 ServerName string SupportedCurves []CurveID SupportedPoints []uint8 + SignatureSchemes []uint16 + SupportedProtos []string + SupportedVersions []uint16 + Conn net.Conn } With some extra ClientHelloInfo https://golang.org/issues/17430
51. // A SessionTicketWrapper provides a way to securely incapsulate // session state for storage on the client. type SessionTicketWrapper interface { Wrap(cs *ConnectionState, content []byte) ([]byte, error) Unwrap(chi *ClientHelloInfo, ticket []byte) ([]byte, bool) Clone() SessionTicketWrapper } Last remaining: session tickets https://golang.org/issues/19199
52. Only use assembly crypto To get fast and constant time crypto, you currently need assembly implementations. On amd64: • crypto/elliptic.P256
 https://blog.cloudflare.com/go-crypto-bridging-the-performance-gap/ • golang.org/x/crypto/curve25519 • crypto/aes (especially fast with GCM) • golang.org/x/crypto/chacha20poly1305
53. Only use assembly crypto &tls.Config{ // Causes servers to use Go's default ciphersuite preferences, // which are tuned to avoid attacks. Does nothing on clients. PreferServerCipherSuites: true, // Only use curves which have assembly implementations CurvePreferences: []tls.CurveID{ tls.CurveP256, tls.X25519, // Go 1.8 only }, }
54. Cloudflare OpenSSL nginx nginx
55. Cloudflare OpenSSL Go nginx nginx Client Hello, file descriptor
56. Cloudflare Go nginx
57. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob)
58. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn])
59. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0])
60. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0]) file := os.NewFile(uintptr(fds[0]), “")
61. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0]) file := os.NewFile(uintptr(fds[0]), “") fileconn, err := net.FileConn(file) file.Close()
62. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0]) file := os.NewFile(uintptr(fds[0]), “") fileconn, err := net.FileConn(file) file.Close() tcp, ok := fileconn.(*net.TCPConn)
63. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0]) file := os.NewFile(uintptr(fds[0]), “") fileconn, err := net.FileConn(file) file.Close() tcp, ok := fileconn.(*net.TCPConn) if tcp.RemoteAddr() == nil || tcp.RemoteAddr().(*net.TCPAddr) == nil { continue }
64. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0]) file := os.NewFile(uintptr(fds[0]), “") fileconn, err := net.FileConn(file) file.Close() tcp, ok := fileconn.(*net.TCPConn) if tcp.RemoteAddr() == nil || tcp.RemoteAddr().(*net.TCPAddr) == nil { continue } return io.MultiReader(bytes.NewBuffer(hello), tcp)
65. File descriptor passing n, oobn, _, _, err := c.ReadMsgUnix(hello, oob) cmsgs, err := syscall.ParseSocketControlMessage(oob[0:oobn]) fds, err := syscall.ParseUnixRights(&cmsgs[0]) file := os.NewFile(uintptr(fds[0]), "") fileconn, err := net.FileConn(file) file.Close() three days of my life tcp, ok := fileconn.(*net.TCPConn) if tcp.RemoteAddr() == nil || tcp.RemoteAddr().(*net.TCPAddr) == nil { continue } return io.MultiReader(bytes.NewBuffer(hello), tcp)
66. And now… net/http
67. Timeouts http: Accept error: accept tcp [::]:80: accept: too many open files; retrying in 1s
68. Timeouts srv := &http.Server{ ReadTimeout: 5 * time.Second, WriteTimeout: 10 * time.Second, IdleTimeout: 120 * time.Second, TLSConfig: tlsConfig, Handler: serveMux, } log.Println(srv.ListenAndServeTLS("", ""))
69. Timeouts func (c *conn) readRequest(ctx context.Context) (*response, error) { if d := c.server.ReadTimeout; d != 0 { c.rwc.SetReadDeadline(time.Now().Add(d)) } if d := c.server.WriteTimeout; d != 0 { defer func() { c.rwc.SetWriteDeadline(time.Now().Add(d)) }() }
70. Timeouts in Go 1.8 You really want Go 1.8, possibly 1.9. • fixed ReadTimeout never resetting in H/2 #16450 • neutered WriteTimeout never resetting in H/2 #18437
 (reintroduced and fixed in Go 1.9) • introduced a proper IdleTimeout #14204
71. Still, timeouts are server-wide So you can’t change them based on path (maybe a streaming endpoint?) or authentication. Should be addressed in 1.10. Come discuss the designs! https://golang.org/issues/16100
72. • TCP-level feature, involving a “ping” on the wire • Not a deadline for completion, just a requirement for the other party to answer the ping • Useless against malicious DoS (“slowloris”) • Enabled only by ListenAndServe[TLS], hard-coded at 3m What about TCP keep-alives?
73. Keeping an eye on open connections var idle = make(map[net.Conn]struct{}) // NOTE: protect with a sync.RWMutex func connState(conn net.Conn, state http.ConnState) { switch state { case http.StateNew: // -> StateNew case http.StateActive: if _, ok := idle[conn]; ok { // StateIdle -> StateActive delete(idle, conn) } else { // StateNew -> StateActive } case http.StateIdle: idle[conn] = struct{}{} // StateActive -> StateIdle case http.StateHijacked: // StateActive -> StateHijacked case http.StateClosed: if _, ok := idle[conn]; ok { // StateIdle -> StateClosed delete(idle, conn) } else { // StateNew, StateActive -> StateClose }
74. http.Server and tls.Conn func (c *conn) serve(ctx context.Context) { // [...] if tlsConn, ok := c.rwc.(*tls.Conn); ok { http.Server type-asserts tls.Conn, which is why we can’t use an external TLS package.
75. http.Server and tls.Conn func (c *conn) serve(ctx context.Context) { // [...] if tlsConn, ok := c.rwc.(*tls.Conn); ok { if d := c.server.ReadTimeout; d != 0 { c.rwc.SetReadDeadline(time.Now().Add(d)) } if d := c.server.WriteTimeout; d != 0 { c.rwc.SetWriteDeadline(time.Now().Add(d)) } First, it sets the timeouts.
76. http.Server and tls.Conn func (c *conn) serve(ctx context.Context) { // [...] if tlsConn, ok := c.rwc.(*tls.Conn); ok { // [...] if err := tlsConn.Handshake(); err != nil { c.server.logf("http: TLS handshake error from %s: %v”, c.rwc.RemoteAddr(), err) return } Then, it runs Handshake and logs the error if any. … so, how do we drop a connection? Well…
77. http.Server and tls.Conn func (c *conn) serve(ctx context.Context) { // [...] defer func() { if err := recover(); err != nil { c.server.logf("http: panic serving %v: %v", c.remoteAddr, err) } c.close() c.setState(c.rwc, StateClosed) }() panic(nil), of course! (Go 1.8 added ErrAbortHandler, thankfully.)
78. http.Server and tls.Conn func (c *conn) serve(ctx context.Context) { // [...] if tlsConn, ok := c.rwc.(*tls.Conn); ok { // [...] if proto := c.tlsState.NegotiatedProtocol; validNPN(proto) { if fn := c.server.TLSNextProto[proto]; fn != nil { h := initNPNRequest{tlsConn, serverHandler{c.server}} fn(c.server, tlsConn, h) } return } And finally it invokes the HTTP/2 handler!
79. When: • Server.TLSNextProto is nil (not empty map) • Server.TLSConfig is set and ListenAndServeTLS is used
 or
 tls.Config.NextProtos includes "h2" Server.TLSNextProto is automatically set (and used if the connection is HTTPS and the client offered H/2). When is HTTP/2 auto-enabled?
80. net/http in production
81. To connect the HTTP server with the Cloudflare backend, we used a modified httputil.ReverseProxy: • Dial overridden to always open a connection to nginx • A sizeable pool of connections to nginx • Websockets • H/2 Push • 0-RTT with confirmation httputil.ReverseProxy
82. Dial and pool to nginx proxy := &httputil.ReverseProxy{ Transport: &http.Transport{ MaxIdleConns: 200, MaxIdleConnsPerHost: 200, IdleConnTimeout: 10 * time.Second, Dial: func(string, string) (net.Conn, error) { return net.DialTimeout(nginxNet, nginxAddr, timeout) }, }, }
83. HTTP/2 Push pu, ok := rw.(http.Pusher) linkHeader := res.Header.Get("Link") for i, ph := range derivePushHints(linkHeader) { po := &http.PushOptions{ Header: make(http.Header) } for _, k := range pushPromiseHeaders { v := res.Request.Header.Get(k) if v != "" { po.Header.Set(k, v) } } pu.Push(ph.uri, po) }
84. 0-RTT! func director(req *http.Request) { tlsConn := req.Context().Value(http.TLSConnContextKey).(*tls.Conn) if !tlsConn.ConnectionState().HandshakeConfirmed { // Request came in 0-RTT data if req.Method == "GET" && req.URL.RawQuery == "" { req.Header.Set("CF-0RTT-Unique", connState.Fingerprint) } else { if err := tlsConn.ConfirmHandshake(); err != nil { panic(err) } } }
85. We didn’t break the Internet!
86. Chrome Field Test Firefox Nightly Cloudflare Launch
88. TLS 1.3 code: audited, upstreaming https://go-review.googlesource.com/q/branch:+dev.tls
89. TLS 1.3: hung :-(
90. crypto/tls and net/http: awesome
91. https://golang.org/issues/9671 https://github.com/cloudflare/tls-tris https://blog.cloudflare.com/tag/tls-1-3/ https://blog.gopheracademy.com/advent-2016/ exposing-go-on-the-internet/ Thank you!
92. Filippo Valsorda hi@filippo.io @FiloSottile Olga Shalakhina artwork under CC 3.0 license based on Renee French under Creative Commons 3.0 Attributions.