Websockets and Torrents - A match made in PHP

Websockets are a technology allowing bi-directional communication between the server and the client and are not often the topic of focus when it comes to writing standard web applications due to their complexity and learning curve required. This talk is the story of my first real project involving sending torrent data from the server to the browser so I could build my own Torrent Web GUI.

What is an event loop? How does it work? How about authentication when using both HTTP and Websocket requests? How do I make my code non-blocking? These are some of the topics covered from both a theoretical and a practical standpoint. I also talk about security and privacy and how not to get caught when setting up a server designed solely for streaming Ubuntu downloads.

Transcript

1. @J7mbo
2. Two words about myself @J7mbo James Mallison
3. @J7mbo
4. Websockets and Torrents A match made with PHP! (and not nodeJS) @J7mbo • The story of my interest in websockets • Implementing the technology for arbitrary (torrent) data • Problems I encountered and solved • A Demo!! • What I learned
5. Legit speeds @J7mbo
6. @J7mbo
7. Transmission - Worst Torrent GUI Ever @J7mbo
8. (I was a graduate…) @J7mbo
9. • Browser makes AJAX request • Server performs CLI call • Server returns Json Response AJAX Request Browser Server Command line Json Response Browser Server AJAX to Server @J7mbo
10. @J7mbo
11. @J7mbo
12. • Browser makes AJAX request • Server SSHs to VPS over TOR • SSH over TOR using Torify • VPS performs CLI call • VPS returns Json Response • Server returns Json Response Browser My Server SSH to VPS Json Response Browser My Server Command line AJAX Request Ajax with SSH to VPS @J7mbo TOR (torify)
13. @J7mbo
14. @J7mbo
15. • Browser sends ws:// or wss:// • Server responds with HTTP/ 1.1 Switching Protocols • Open and persistent connection • Connection can be closed by either server or client at any time. Websockets @J7mbo
16. Request to Server Response to Client END Http VS Websockets • Send a request • HALF DUPLEX • Wait for a response • Rinse, repeat Http Event Loop WS Connection Started Send response Send response Send another response Receive message Websockets • Initialise connection • FULL DUPLEX • Send / receive at any time • Close connection @J7mbo
17. Duplex wtf? @J7mbo Half Duplex In a conversation, only one person can talk at once The other person has to listen and wait to give a response Full Duplex In a conversation, everyone’s able to talk and listen at once (Like Internals) Everyone is able to understand what each other is saying (Not Like Internals)
18. @J7mbo Event Loop
19. • Not just while(){} loops • Use interrupt driven I/O instead of polling • IDI/O flag is set on CPU, process suspended and buffer filled • Flag changes, callback executed (Still requires flag checks, not magic) • Useful for IPC • PHP’s PCNTL similar to this (signal based) @J7mbo Event Loops
20. PCNTL Extension @J7mbo
21. @J7mbo Ratchet
22. • Code flow controlled by user • Allows software to react • User calls .subscribe() in JS, onSubscribe() is called in the PHP event handler @J7mbo Event Driven? @J7mbo
23. The Web Application Messaging Protocol @J7mbo WAMP Events
24. onOpen() @J7mbo
25. onOpen() @J7mbo
26. onSubscribe() @J7mbo
27. onSubscribe() @J7mbo
28. Websocket Event Handler + Loop HTTP or Websocket? ProxyPass Http Ws Apache + PHP Client makes a request Enable mod_proxy • ProxyPass /wss/ wss://127.0.0.1:9000/ @J7mbo
29. Event Loop .subscribe(func, topic) 1 onSubscribe($conn, $topic) • store connection in array • start timer for connection • store timer against connection 2 High level functionality + events Timer hit • Get torrent data via CLI • This takes ages… … … 3 ‘Broadcast’ data back to topic • $topic->broadcast($data) 4 Client calls .unsubscribe(func, topic) • Event handler onUnsubscribe() called • Stop timer for connection • Remove from internal connections array 5 @J7mbo
30. Event Loop Problem 1 - Topic ‘Broadcast’ data back to topic • $topic->broadcast($data) 4 Option 1: $topic = ‘torrents’ • - Not on a per-user or per-connection basis • - Everyone subscribed to ‘torrents’ gets the data pushed to them • - Open same page, data gets sent to both pages twice Option 2: $topic = { ‘topic’ => ‘torrents’, ‘userId’ => 245 } • Every subscription is now unique per user id • - Same user can open same page and start getting duplicate data like before @J7mbo
31. Problem 2: Authentication and Duplicate data The Problem • Pass user id from PHP to JS variable • Subscribe to torrents with topic containing user id • Client could have just changed user id JS variable before subscribing One Solution • Create unique token on each standard HTTP request in your application • Use PHP 7.1s CSPRNG rather than open_ssl_random_pseudo_bytes() • Pass token with user id in topic and check against database • Recreate token in database so that a second check will fail $topic = { ‘topic’ => ‘torrents’, ‘userId’ => 245, ‘token’ => ‘385e33f741’ // bin2hex(random_bytes(5)) in PHP 7.1 }; @J7mbo
32. Problem 3: Blocking!! The Problem • Whilst command line call is being made (5 seconds), nothing else can happen • No .subscribe ——> onSubscribe() : Nobody else can do anything • This is blocking! Same goes for authentication (checking database) A few solutions • Use another process to do the work - on the same machine • Use a job queue - requires a way to get data back into event loop once done • Requires good architecture decisions for low coupling + high cohesion for event naming, job naming, whether worker knows who’s job it is etc @J7mbo
33. Blocking - Option 1 - Child Process The Result • Event loop can continue, fine for small projects or just as a proof of concept • Process is only on one machine - not scalable @J7mbo
34. Blocking - Option 2 - Job Queue (and ZMQ) In onSubscribe() - add to queue ZMQ listening - When setting up event handler @J7mbo
35. Blocking - Option 2 - Job Queue (and ZMQ) Worker sends data over :5555 when done to onZmqResponse() @J7mbo
36. Event Loop .subscribe(func, topic) 1 High level architecture onSubscribe($conn, $topic) • store connection in array • add job to queue (fire and forget) 2 Queue Workers pick up job • Perform job • Send result to :5555 3 Workers Anything sent to port 5555 goes to onZmqRequest() Broadcasted to client 4 • Event loop is just an intermediary for data retrieval and sending • Nobody is blocked from registering their interest in receiving data • Workers can be in Java, C*, Erlang, Python, Ruby, JS etc @J7mbo
37. Demo ? @J7mbo
38. What about scaling? (One way without a load balancer) @J7mbo Client API DB / Redis Workers Queue Call API Event Loop Event Loop Event Loop Call chosen ws:// server Round Robin? ZMQ Scalable Scalable Broadcast data to client
39. • Using an event loop != non-blocking or asynchronous • Easiest way to not block, use threads, processes or an external library • All connections between client / ws server over SSH with TOR • Could accomplish with NodeJS (but didn’t want to sell my soul) • Could have just made RPC calls to the server anyway instead of cmd- line (I found this out right at the end of it all) • Didn’t have to broadcast on a $topic, as $connection had methods of sending data to specific connections What I learned @J7mbo
40. • Event loops and reactors in PHP: React, amphp • Websocket server-side libraries: ratchet, php-websockets • WAMP-specced client-side libraries: Autobahn.js, AngularWAMP, Minion • Event-loop underlying libs: LibEv, LibEvent, LibUV (windows) Your Choices @J7mbo
41. • @j7mbo (PHP6 jokes, php, oop, software engineering) • PHPNW15: Justin Carmony - Scaling & Managing Asynchronous Workers (and staying sane!) - Watch this for queues etc - it’s an awesome talk! Thanks @J7mbo https://joind.in/talk/63c1c
42. DEMO (if it works)