The Dark Art Of Bundlers

Chain React 2017

Transcript

1. The dark art of bundlers Hello everyone, I am Mike and I am the Co-Founder and CTO at Callstack. I am also one of the core contributors to React Native. Today I’ll be demystifying Javascript bundlers, specifically in React Native context.
2. What is a bundler? Let’s start with the most fundamental question - do you know what a bundler is? Please raise your hand! > (Quite good) Nice! I can see a lot of hands! Please bear with me as we get through those slides > (Not good) Great! This might be the most educational presentation I have ever made then haha
3. a.js b.js d.js c.js e.js f.js bundle.js eg. Webpack So bundler is a tool that simply takes all your Javascript source files and puts them into one file. That process is called bundling. There are many of them these days, with Webpack, Browserify or Metro being most popular.
4. To bundle or not to bundle? You are probably now wondering why do we even need that - good question. To better understand that, let’s step back to dark days of jQuery and early web.
5. Early web Historically, Javascript hasn’t had a standard for requiring dependencies from your code. There was no `import` or `require` statements. So how did we make functions of our code visible to the outer world and how did we import functions from other people’s code?
6. Globals, yea! The only way was through global variables…
7. Globals, yea! For example, if you wanted to use jQuery… you would go ahead and add a script tag that loads it from external resource. As a result, your global scope would get populated with its bindings, in this case a dollar sign.
8. Where’s the problem? Okay, one could say that this is actually neat. All set with just a few lines of code, right? Imagine this scenario: Your prototype works out, your application grows and you end up looking into splitting your file into smaller, reusable pieces.
9. Multiple files You would end up with something like this, on slide. Now, the code that will get executed in the script tag should have access to all globals defined in the preceding files. The problem here is the order. What if foo.js wanted to use bar.js code. Manipulating the order files are loaded and executed is a design flaw. I call it script hell. From callback hell we know it’s bad!
10. How do we solve it? So how do we solve this? Turns out we already did this in other environments. Take Node.js for example.
11. Node.js It implements its own module system using `require` function and `exports` object amongst other things based on the Common.js module spec.
12. Web What if we could do the same in our frontend code? That would great, huh? There’s one technical limitation though.
13. `require` is sync `fetch` is async `require` itself is synchronous. In Node.js environment, when you want to load a file, it’s physically accessible on the disc, in most cases next to existing file. In our context, however, we need to make an HTTP request to load a file that has never been loaded by the browser. And that request is asynchronous.
14. To bundle or not to bundle? The solution is simple. We can bundle all the files into one, so that we have all required dependencies in memory at the time of `require` function being called. And that’s the primary reason for bundling explained in 5 minutes.
15. Wait… is that all a bundler can do? Bundlers can be perceived as a natural evolution of task runners, like Gulp or Grunt that happened over the course of past few years. There are lots of responsibilities in scope for them rather than just getting your code into one chunk for sake of organising dependencies.
16. Transpilation One of them is transpilation. A typical bundler can, or has support, for transpiling your code.
17. Transpilation That means for reading input source in particular format (e.g. newest Javascript spec) and outputting it in an older standard (e.g. ES5). The most popular out there is Babel and most bundlers out here provide a first-class integration with it by default.
18. Code splitting We mentioned already that in order for `require` to work, entire application has to be loaded into memory. That causes your bundle size to increase and ship with modules that don’t necessarily need to be loaded on the initial page run.
19. Code splitting Code splitting is a feature that allows you to split your code into various bundles which can then be loaded on demand or in parallel. In ES2015 compatible environment for example, you can use `import` function to obtain an instance of a module. The `Promise` is resolved as soon as the loading process is finished.
20. Code minification Bundlers also enable code minification. For example, Webpack 2 ships with a built-in support for tree shaking. That means it can determine what parts of your modules are not used (e.g. module foo is never required) and effectively remove those parts. Combined with regular minifiers, like Uglify, it contributes to smaller bundle size that gets delivered to users.
21. How does that all relate to React Native? So that’s a bit of theory. Now, how does that all relate to React Native?
22. Quick look at infrastructure With React Native, we express our native mobile applications by writing Javascript and layouts - by writing React components. However, since both iOS and Android are distinctive platforms with their own SDKs, languages and tools, there is no way to run the Javascript in the simple way.
23. Javascript VM What those all platforms have in common is the ability to spin a Javascript VM that can process and execute an arbitrary application bundle. React Native then sets up a bi-directional data flow by using a “bridge” - layer that can accept calls from Javascript and pass them through to a given native module
24. Same constraints The use of Javascript VM is what makes constraints we just talked in context of web, applicable here as well. Just to give you a great example: On iOS, React Native uses built-in Safari to execute your application logic. Its features and performance changes on per system basis. You will find a different set of ES6 supported on your testing Mac than on iOS8, 9 or 10. So you definitely want to transpile your code to lowest common subset supported.
25. Is React Native web alike? Sounds like React Native is pretty similar to web, right? We have access to some of the same polyfills, globals, APIs. The language and spec is also the same. We bundle code the same way. There are even modules on npm that work in both environments. However, there’s one thing that is fundamentally different.
26. react-native start It is the react-native start command we run every day. At first glance, all it does is it serves your entire application as a single bundle that can be accessed on your localhost. Pretty simple, right? Then, why don’t we spawn Webpack and configure it to serve the bundle at same URL? I like experiments, so I thought - why not try doing it.
27. Webpack config! I started with creating an empty webpack config where I could just point to an entry point and define some other settings for React Native to load the bundle. 
 Then, I thought of all React Native specific features I knew I have to support for the first implementation to work.
28. One of them was of course dev server - React Native works by loading bundle from an arbitrary URL. In most cases, it’s the name of entry point with platform.bundle extension.
29. Then, I had to integrate Babel to transpile ES6 and other Javascript code to run on my iOS device. By default, React Native does that to your entire node_modules folder, including even modules that don’t need it. To make my experiment backwards compatible, I decided to conditionally whitelist all modules start with “react” keyword (like react-native itself or any react-native plugin).
30. Last but not least, to make my platform-specific code to run, I used `resolve.extensions` to define order files should be resolved. In this case, I wanted my implementation to load first, platform specific file, then native file and end up with loading just a js file.
31. Haste module system Finally, the most entertaining part was supporting Facebook proprietary `node-haste` system.
 
 It’s pretty simple - any file annotated with @providesModule, can be globally required within the project by its name. That way, you can have files spread across entire repository and require them w/o knowing their exact location. Of course, Webpack has no idea how to support this and what it will try is to look inside node_modules.
32. But it has a great `resolve.alias` feature. It is a map where keys are names of modules that can be required globally and values - absolute path to a file that should get loaded. My solution turned out to be quite dumb, but perfect in terms of performance. I decided to glob for `js` files in provided array of folders and just parse doc blocks of every file that had it.
33. Prototype After hacking with it for few nights, I finally got it running. In less than 7 seconds Webpack created and served a bundle on port and URL I wanted. 
 Great, I thought - nothing but declarative Webpack config and I rule the world. My excitement went away as soon as I opened a native client. I have noticed lots of features were missing. Example: “Live reload” wasn’t working at all.
34. Debug in Chrome (screen) Attempting to “Debug in Chrome” started producing red screens. By that time, I didn’t really know what “Runtime is not ready” meant.
35. No more 
 nice errors (screen) More importantly, red screen messages started looking weird - the stack trace was unreadable and it was almost impossible to locate the original issue.
36. There must be more… I started to realise that in context of React Native, there has to be more than just a bundler that is running. Turns out, running `react-native start` spins up entire set of tools focused on providing a better developer experience. At its core, there’s a bundler. All the rest are just complementary features built on top of it.
37. Introducing: Packager I call it a packager - tool used to develop, debug and package an application. It builds on top of an arbitrary bundler, like Webpack and ships with pre-configured set of transforms required for the code to run in particular environment.
38. Digging in deeper! So I decided to keep my experiment going and see if I can make Webpack work with React Native. The plan was to support all its features, including symlinks, that never worked with default packager yet. I was in particular interested where’s the boundary, that is - how many features I’ll have to write on top in order to make it a drop-in replacement.
39. Looking for connection The first thing I started looking into was the Chrome Debugger. On standard React Native app, turning that mode opens a new tab in the browser. However, there wasn’t any API on iOS or Android that I was aware of to open an arbitrary app on a Mac. I quickly realised that it has to be packager that opens a Chrome tab, not React Native. Since it is a process running on a host machine, it has access to all required APIs. Now how do those two pieces talk to each other?
40. Packager is http server Turns out packager is an http server.
41. Http server It is composed out of standalone middlewares, each, having different responsibility. Together, they form a tool that provides the developer experience we know from React Native. It is a really cool and powerful concept - being able to plug & play certain parts as a middleware makes it really hackable and customisable. As you can see, the last middleware here in the list is the `metro-bundler`, the one that serves your bundle by default.
42. “RPC over HTTP” React Native Packager GET /launch-devtools status The communication is established by React Native, calling Packager endpoint, e.g. launch-devtools. Then, the handler is executing procedure on a host machine and reports back to React Native client. It is very similar to a remote procedure call, in distributed computing, with a difference that everything happens on your localhost!
43. Bundler middleware So in order to get my project rolling, instead of spawning webpack from command line and relying purely on webpack config, I decided to use express server and webpack-dev-middleware.
 
 That way, I could still get an instance of express server and get the ability to extend it with the features standard packager has by default.
44. Live reload middleware Let’s take a look at another middleware - live reload middleware. As the name indicates, it’s responsibility is to provide a “Live reload” function in our dev environment.
45. Long-lasting request Once activated, React Native sends a long-lasting GET request to the Packager.
46. On each such request, the middleware stores a response object associated with a particular request.
 
 As soon as change to files happen (in case of webpack, it can be done by listening to `done` event), all connections are closed with `changed: true` indicating that live reload is required on the client.
47. Dev tools middleware Now that we have a better understanding of how the communication between Packager and React Native is done, it’s finally the time to look at the dev tools middleware. 
 I am quite excited about that one - the ability to use an arbitrary browser to run and debug our Javascript was pure magic to me the first time I saw it.

48. Launch Dev Tools As soon as you press Debug with Chrome, React Native executes a remote procedure to launch dev tools inside Chrome tab when necessary. Right after, both device and debugger connect with each other by using Websocket connection. Since there’s no way for them to connect directly, they do it through Packager, which proxies all messages. If for some reason the connection cannot be established, we end up with “Runtime not ready for debugging” message we already saw on past slides.
49. Once all’s well and good, WebSocket executor on the device sends the JSON message to the Chrome tab. It has to contain at least a `method` property - it tells debugger which handler to execute. The rest of the keys varies on handler-basis.
50. The `executeApplicationScript` is a so-called special handler. It is hardcoded inside the debugger. First, it attaches the `inject` JSON onto the global object. One of them is `__fbBatchedBridge`. Later, it uses WebWorker `importScripts` to synchronously load Javascript at given URL into the worker scope.
51. If given method is not a special case handler…
52. … its handler will be looked up at __fbBatchedBridge. Oh, and by the way - this communication pattern is pretty much the same when you are using device Javascript Core instead of Chrome Debugger. The difference is that instead of Websocket connection, iOS specific API is used to evaluate scripts.
53. Symbolicate middleware Once I got debugger up and running, I started looking into understanding why error messages didn’t have proper stack traces. 
 Now I didn’t include this in my initial prototype - Steve Kellock from Infinite Red contributed it later - thank you! However, I find it very interesting so decided to describe it as if it was there from the very beginning.
54. Symbolicate frames What happens is that as soon as there’s an error - React Native asks packager to symbolicate the error stack trace. 
 It does that by passing an entire callstack and expects the same structure, but with appropriate lines, columns and files to be returned as a response.
55. As soon as we get such request, we create an instance of `SourceMapConsumer`. We pass it a raw source map (as for example read from file system).
 
 SourceMapConsumer represents a parsed source map which we can query for information about the original file positions by giving it a file position in the generated source.
56. We then map request body (which is an array of frames received from the device) to the original source, line and column information we got by querying SourceMapConsumer. Such array is then sent back to React Native which quickly replaces unsymbolicated stack trace with a better one.
57. Bundler should be pluggable The reason I am telling you this is I believe the bundler implementation should be pluggable. The packager in React Native does way more things than just compiling our code. And those things are not trivial - some of them were carefully designed over the course of months. From my experience, it took me two weeks to implement Webpack to work with React Native. If the entire platform was pluggable, it would’ve been a breeze.
58. Do we really need that? But… do we really need an alternative bundler? Let me answer this question with another one.
59. Do you want symlinks? Lack of support for symlinks was for long one of the most common issues with React Native development, especially for library developers. React Native promised the workflow we know from Web and Node.js in the world of mobile development. Breaking that expectation was painful for many. The default Metro Bundler uses Watchman to listen to file changes and reacts accordingly. Now, Watchman is a Facebook open source library constrained to provide maximum performance at their scale. One of its limitations is that it doesn’t listen for changes in symlinked locations and it’s unclear whether this feature is going to land anytime soon - not a priority.
60. Facebook scale At Facebook scale, lots of milliseconds that seem irrelevant to us matter a lot. It’s great that we get performance improvements for free. However, at the same time, I believe one should be able to opt-in for a slightly slower but more feature-rich solution.
61. Meet Haul That’s why, at Callstack, we made Haul. Haul is a drop-in replacement for existing packager. It builds on top of open tools, like Webpack. At its core, there’s Express server with pluggable middlewares. It’s a hackable dev server with predictable workflow. Most of the code snippets you have seen so far in this talk can be found in the repo.
62. Select platform Here’s how it feels.
 
 You first select a platform you want to prepare bundle for. If you don’t pass it as an option, Haul will ask you for it instead of exiting.
63. Then, wait for it At the same time, it will print how to call it next time to avoid interactive prompt from appearing. Then, you just wait for it to finish bundling…
64. Ready to go And you are ready to go! We designed it carefully around best developer experience possible taking as much inspiration as possible from community leaders such as create-react-app.
65. Reasons? There were a couple of reasons we made Haul. Two of them were already mentioned today, so to recap: (first) First one was to have a support for symlinks, something we could die for (second) Second was to embrace potential of plug-ability and make something, that can be extended in an easy way
66. Webpack is awesome! The last one was Webpack - it’s just awesome! What I like about it is its extensive, well documented interface with a broad ecosystem. It opens lots of exciting possibilities, like using new extensions for your files, using CSS, new loaders or even Typescript. Also, if you are already using Webpack on web, you could share the config and maximise the percentage of common business logic in your codebase.
67. What’s the future? There are lots of missing features we are working on. Currently, our top priority is to increase its adoption so that we get more bug reports and can move forward! If the idea of Haul sounds interesting to you and you are likely to use it in the future, please give it a go and let us know your feedback!
68. Make Haul a platform As I mentioned, I am really looking into making Haul a platform for other bundlers to run. There are faster alternatives to Webpack, like Fusebox. I believe that one should be able to use whatever works for him to bundle the app and still, get first class developer experience that Haul gives him.
69. Thank you Thank you!
70. Q & A Do we still have time for some questions?