HTTP Tunneling in Go

HTTP Tunneling can be used in situations similar to the requirements of using VPN, for example, to be able to access a localhost server from outside. This talk will introduce the specific implementation of HTTP tunneling and give audiences a deep knowledge of TCP and HTTP level implementation in Go.

Specifically, this talk discusses the idea of HTTP Tunneling in Go.

Transcript

1. HTTP Tunneling in Go Kazuki Higashiguchi April 23, 2022 @ Go Conference 2022 Spring
2. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1. HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
3. About Me Kazuki Higashiguchi Backend Engineer at Autify. ID: @hgsgtk
4. Autify Solution No-code AI-powered software testing automation services for Web/Mobile applications AI automatically maintenance test scripts No code unlocks automation at scale
5. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1. HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
6. HTTP tunneling • Create a network link between two computers in conditions of restricted network connectivity. e.g. firewalls, NATs, ACLs • Use a protocol of higher level (HTTP) to transport a lower level protocol (TCP)
7. HTTP CONNECT method • Start bidirectional communications with the requested resource e.g. access HTTPS websites, HTTP proxy server HTTP(S) Client HTTP(S) Server Proxy (Gateway)
8. HTTP tunneling with CONNECT Quoted from HTTP: The Definitive Guide / 8.5 Tunnels 1. Client sends CONNECT request 2. Gateway opens TCP connection to the server 3. Gateway returns HTTP ready message to the client 4. Start bidirectional communication of raw packets of data
9. A case study: elazarl/goproxy elazarl/goproxy: An HTTP proxy library for Go $ curl -I -x http://localhost:8080 https://example.com HTTP/1.0 200 OK A basic sample code:
10. Tunneling implementation in elazarl/goproxy https://github.com/elazarl/goproxy/blob/0bfa7c564b 5ffbece2f0832eb4e714fb971f70f3/https.go#L95
11. Tunneling implementation in elazarl/goproxy 1 2 3
12. Step 1: Take over the proxy connection http.Hijacker interface allows an HTTP handler to take over the connection.
13. Step 2: Open TCP connections Opens TCP connection using net.Dial Returns HTTP ready message: 200 Connection established
14. Step 3: Streaming with io.Copy
15. Streaming: TLS handshake over TCP connection • TLS handshake is done using the streaming • Application data is sent and receive after TLS handshake
16. Key takeaways • HTTP Tunneling uses a protocol of higher level (HTTP) to transport a lower level protocol (TCP) • The basic flow of HTTP tunneling using the HTTP Connect method • http.Hijack and io.Copy are key functions to implement HTTP tunneling in Go
17. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1. HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
18. Establish bidirectional connections WebSocket server WebSocket client Handshake WebSocket Conn WebSocket Conn HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server
19. Bidirectional connections • Long/short polling • WebSockets • Server-Sent Events • HTTP/2 Push • etc
20. WebSockets • A mechanism for low-cost, full-duplex communication on Web • Designed to work over HTTP (compatible with the HTTP protocol) • Handshake -> Messages -> Close
21. WebSocket client and server using gorilla/websocket Client: gorilla/websocket: an WebSocket implementation in Go Server:
22. websocket.Conn implementation websocket.Conn has net.Conn in the unexported field. The return value of http.Hijack will be set to the field conn. https://github.com/gorilla/websocket/blob/eeb877e8a1fd26c449b7af3cb1d71879287d724c/server.go#L125
23. Agenda HTTP(S) Client WebSocket server WebSocket client HTTP(S) Server 1. HTTP Tunneling 2. Establish bidirectional connection using WebSocket 3. HTTP Tunneling over WebSocket
24. HTTP Tunneling over WebSocket HTTP(S) Client HTTP(S) Server WebSocket server WebSocket client WebSocket Conn WebSocket Conn TCP Conn TCP Conn $ curl -Lv -x http://websocket-server https://target.local WebSocket server and WebSocket client jointly act as a proxy(gateway). https://target.local http://websocket-server streaming streaming
25. How HTTP Tunneling over WebSocket
26. How HTTP Tunneling with WebSocket WebSocket server tells the address of destination to WebSocket client. WebSocket client open a TCP connection to the destination server.
27. How HTTP Tunneling with WebSocket Bidirectional message over an established WebSocket connection.
28. WebSocket server implementation A sample code available on github.com/hgsgtk/go-snippets
29. WebSocket server implementation 1 2 3
30. Step 1: Write messages to WebSocket connection Send a TCP handshake request with the destination address.
31. Distinguish several types of messages This design transfers several kinds of message over a single WebSocket connection • Request to open TCP connections • Notify to establish TCP connections • Streaming raw packets of data • etc
32. A case study: Chrome Devtools Protocol CDP defines the JSON protocol, which is used to communicate in the WebSocket connection. Request Members: id, method, params Response Members: id, result, (error)
33. Define own protocol for WebSocket communication A small example of methods • Handshake: Request to open TCP connections • Completed: Notify to establish TCP connections • Communication: Streaming raw packets of data
34. Step 2: Take over the proxy connection 2 Take over the connection using http.Hijacker.
35. Step 3: Streaming 3 Use io.Copy to combine WebSocket connection and TCP connection.
36. Notes on net.Conn returned from http.Hijack • http.Hijacker returns net.Conn and bufio.ReadWriter. [Code comment on http.Hijack] The returned bufio.Reader may contain unprocessed buffered data from the client. • Reading net.Conn alone may cause data to be dropped. • Read buffers from bufio.Reader.
37. Wrap websocket.Conn to use in io.Copy: io.Writer Pack incoming packets to defined JSON format ﾝu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>���C� ��m$�X�Ax�Y%^��J�v�[�o�U�a���U�t �u��5��k��m��&|Ǜq����)�z7����?�… { "method": 3, "data": "ﾝu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>� ��C���m$�X�Ax�Y%^��J�v� [�o�U�a���U�t �u��5��k��m��&|Ǜq����) �z7����?�… " }
38. Wrap websocket.Conn to use in io.Copy: io.Reader Read packets from a decoded JSON message ﾝu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>���C� ��m$�X�Ax�Y%^��J�v�[�o�U�a���U�t �u��5��k��m��&|Ǜq����)�z7����?�… { "method": 3, "data": "ﾝu��(gT��!"�. ��ҠF��0�Q��Yl/��L�E8/���c<��>� ��C���m$�X�Ax�Y%^��J�v� [�o�U�a���U�t �u��5��k��m��&|Ǜq����) �z7����?�… " }
39. WebSocket client implementation
40. Open a TCP connection to the given destination Dial to the destination address conn, err := net.Dial("tcp", “autify.com:443”)
41. Key takeaways • HTTP Tunneling over WebSocket • Define own protocol for WebSocket communication and create a wrap struct for streaming with io.Copy • Should read both net.Conn and bufio.ReadWriter returned from http.Hijacker
42. Resources • RFC 6455 The WebSocket Protocol • IETF: Tunneling TCP based protocols through Web proxy servers • Mmdn web doc: Proxy servers and tunneling • Mmdn web doc: CONNECT • Wikipedia: HTTP Tunnel • Wikipedia: WebSocket • HTTP: The Definitive Guide / 8.5 Tunnels • Cloudflare: What happens in a TLS handshake? | SSL handshake • Chrome DevTools Protocol
43. Resources • elazarl/goproxy • gorilla/websocket • github.com/hgsgtk/go-snippets/gocon2022spring/tcpws
44. Thank you for your watching We are taking demo requests https://autify.com/ Autify for Web Autify for Mobile