Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JavaScript @ Uber

JavaScript @ Uber

How Uber leverages JavaScript across its engineering orgs. From building frontend properties with a full JavaScript stack using express and react to the tools and infrastructure we use to automate best practices.

Dustin Whittle

May 11, 2017
Tweet

More Decks by Dustin Whittle

Other Decks in Technology

Transcript

  1. 73
 Countries 470+ Cities 75%
 of the U.S. population lives

    in a county with access to Uber 1.5M+
 Active Driver Partners 5+ Million
 Trips per day 2+ Billion
 Trips completed
  2. Jun ‘10 First ride in San Francisco 2 Billion rides

    worldwide Jun ‘16 Launched self-driving car project Feb ‘15 May ‘12 1 Million rides worldwide 1 Billion rides worldwide Sep ‘16 First self-driving ride May ‘17 73 countries launched TODAY
  3. Organized as Programs and Platforms Program Platform Mission Business Focused

    Technology focused Consumers Non-technical + Internal/ External Technical + Internal Products Feature-based products Technology consumed by programs Team Cross-functional Specialized Direction Product drivers Engineering Drivers
  4. Languages in use at Uber We are a polyglot engineering

    organization, and we speak many languages. In order of preference - many services are moving to Go (or Java)
  5. Why JavaScript? • A single stack for frontend and backend

    • Simple interface with thorough documentation • Lends itself to fast prototyping and quick iteration • Asynchronous, nimble, flexible • Avoid concurrency challenges • Increasingly mature module ecosystem • Today we only use Node.js for web applications
  6. How Uber uses JavaScript Marketplace Platform (core services) Web Platform

    (web presence) Visualizing Data (everywhere) Developer Platform (external API)
  7. Let Builders Build 12000+ full-time employees globally • Send proposal

    and request for comments to entire eng org • Abstract, Architecture, UX, Ops, Security, Timeline • Start with scaffolding that is purpose built for our stack • Skip the boilerplate while following best practices • Deploy with standardized global infrastructure • Automate everything in a consistent way • Playbooks • Operationalize teams at scale
  8. Uber Service Foundation • Instrumentation (Jaegar + OpenTracing) • Source

    Control (Phabricator) • Continuous Integration (Jenkins) • Infrastructure + Deployments + ChatOps (u*) • Monitoring + Alerting • Metrics + Analytics • Security + Compliance • Experiments • Asset Pipeline + CDN • RPC (HTTP, TChannel, etc) • Translations + Internationalization • Data (Schemaless, Cassandra, SQL) • Network • Compute • Storage • Logging • Performance + Failure Testing
  9. How Uber uses JavaScript Marketplace Platform (core services) Web Platform

    (web presence) Visualizing Data (everywhere) Developer Platform (external API)
  10. Real-time API is the frontline web service of Uber serving

    99.99% of all the mobile traffic with >1000 HTTP endpoints. • Acts as a router to 3400+ microservices • A single place for monitoring and documentation Marketplace Gateway Real-time API is the only interface
  11. Pick up
 Passenger 1 Pick up
 Passenger 2 1st Drop-

    off 2nd Drop- off Logistics Challenges ... • How to optimize for overall marketplace efficiency? • How to dynamically price trips and balance efficiency? • How to maximize pick-up efficiency (pick up zones, pick up suggestions)? • How to minimize wait times for drivers and riders? • How to deal with different street configurations? • Combine different logistics products (Pool, Eats, Rush) UberPool is a traveling salesman problem on steroids
  12. Uber Engineering Stack Evolution 2009-2010 - Outsourced PHP + MySQL

    Jan 2011 - "Dispatch" - Node.JS/MongoDB Jan 2011 - "API" - Python/SQLAlchemy/MySQL Feb 2012 - Dispatch swaps MongoDB for Redis May 2012 - Dispatch adds on fallback Jan 2013 - First non-API Python services Feb 2013 - API switched to Postgres Mar 2014 - New Python services use MySQL Mar 2014 - Schemaless begins, must finish before Postgres collapse Sep 2014 - First Schemaless - trips out of Postgres Aug 2015 - Dispatch X.0 / Ringpop / Riak Jan 2016 - Go, Java, More Abstractions May 2017 - UberFx for Go, Cloud
  13. Monolith ≠ bad • Well, not bad at first, but

    can turn into a ball of mud quickly • Optimizes development velocity of early stage projects • Consolidates operational overhead in one place • Monolith isn’t as bad when there is a single team that owns it
  14. A microservice is a service that is focused on only

    one thing. One Thing != One API
  15. • Clear ownership boundaries • Most of the time •

    Product-specific velocity • Move the business faster • Able to solve scaling issues • Systems as well as company • Training, documentation, tooling • Language/platform independence • Best tool for the job • Reliability of the system • Independent • Scalable • Testable Pros What’s good • Increased complexity • Naming + Discovery • Eventual consistency • Performance Impact • Distributed tracing • Testing • Operational overhead • Monitoring • Continuous Integration • Deployment • Documentation • Instrumentation • Security • Repeated code/effort Cons What’s not so good
  16. Ringpop Ringpop is a library that brings cooperation and coordination

    to distributed applications. It maintains a consistent hash ring on top of a membership protocol and provides request forwarding as a routing convenience. TChannel TChannel is a networking framing protocol used for general RPC, supporting out- of-order responses at extremely high performance where intermediaries can make a forwarding decision quickly. Client libraries available in Go, Java, Node and Python. Jaeger Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It can be used for monitoring microservice-based architectures. Client libraries available in Go, Java, Node and Python using the OpenTracing standard.
  17. Lessons learned scaling Node.js • Latency is too high for

    ultra performant backend systems (p99 for max latency) • Early on it made it quick to iterate, but as the size of the team scaled the developer velocity started to slow down • Microservices enforce a tight interface so having static typing enables large teams to catches issues earlier. It really has an impact with 100+ devs. • Quick to learn, but easy to write poor quality code • Enables you to move fast, but allows for sloppy code • Great ecosystem of small libraries, but many were immature compared to Java/Go • Don't be afraid of writing c++ Growing pains learned the hard way
  18. Lessons learned scaling Uber • HTTP and JSON was designed

    for browsers; using RPC is better for computer-to-computer requests • After a certain age, microservices should become immutable • Having multiple languages allows for team preferences, but segregates developers based on language and prevents easy re-use of code across services • Company > Team > Self • Monorepos allow for changes to be made across multiple services atomically, but prevent future open-sourcing and subset checkouts • Performance problems are difficult to debug cross-language without standardised service dashboards and observability tools • Logging should never slow production down; in a failure storm, the logging system should drop rather than delay • Everything is a tradeoff: Be intentional Growing pains learned the hard way
  19. How Uber uses JavaScript Marketplace Platform (core services) Web Platform

    (web presence) Visualizing Data (everywhere) Developer Platform (external API)
  20. The Web Platform Serving 250+ web properties Without a consistent

    foundation, all of these applications could have been built off of entirely different tools and architectures, creating a higher likelihood of security vulnerabilities, duplicated effort across teams, technical debt, minimal UI consistency, etc.
  21. Uber Web Platform built from Open Source • Nginx +

    HAProxy: Web frontends + load balancers • NPM: NPM registry • Babel: For ES6 and ES7 transpilation • Express: Fast, unopinionated, minimalist web framework for Node.js • React: UI rendering library • Redux: Predictable state application framework • Styletron: Universal, high-performance JavaScript styles • ESLint: Maintain a consistent code style across many teams • Browserify: For bundling client-side code • Tape + Enzyme + Sinon: Testing frameworks • Unitest + Istanbul: Seamless Node.js and Browser testing with coverage • Gulp + Dev Tools: Standardized build tooling
  22. Uber Web Platform added special sauce • Internal npm registry

    with caching • Yeoman like scaffolding for bootstrap new apps with best practices • Customized middlewares for our Express based app framework • Authentication, Security (XSS, CSRF, CSP), Metrics, Logging, I18n, Errors, Instrumentation, Analytics • Bootstrap like UI component library build on top of React.js • RPC library for intelligently using http/tchannel • Shared Build + Test Gulp tasks with hot reloading, ES6/JSX, asset versioning and deployment pipelines, testing + linting • Universal rendering for server/client
  23. Frontend vs Backend • We separate our frontend services (those

    that serve web pages) from our backend services (those that get data from some database) • Services can take advantage of universal rendering + tooling • You can utilize languages better suited for your backend services (Java, Go) • You can deploy and scale your services separately • You can reuse the backend service API for other clients
  24. React components for flexibility Uber’s take on bootstrap • Beautiful

    sites with flexible components you don’t have to maintain yourself • Standard UI components • Optimized for performance • Encapsulated style • Consistent look and feel throughout all components that fit brand • Analytics and instrumentation baked-in • Seamless upgrades to new versions
  25. CSS in JS with Styletron import Styletron from 'styletron';
 import

    { injectStyle } from 'styletron-utils';
 
 // Create a Styletron instance
 const styletron = new Styletron();
 
 const className = injectStyle(styletron, {
 color: 'red',
 display: 'inline-block',
 fontSize: '1.6em'
 });
 // Css is injected into the page
 // and a class name is returned
  26. How Uber uses JavaScript Marketplace Platform (core services) Web Platform

    (web presence) Visualizing Data (everywhere) Developer Platform (external API)
  27. Visualization Frameworks • react-vis: Charts and Networks using D3 +

    React • react-map-gl: A React interface to MapboxGL-js • deck.gl: Layered WebGL approach system for visualization • luma.gl: A JavaScript WebGL Framework for Data Visualization
  28. • ES6, WebGL 2.0, component-based platform • Interoperable with other

    popular libraries like stack.gl • Shader library with a 64-bit floating point emulation package • Advanced debugging, tracing, error checking for WebGL luma.gl A JavaScript WebGL Framework for Data Visualization
  29. How Uber uses JavaScript Marketplace Platform (core services) Web Platform

    (web presence) Visualizing Data (everywhere) Developer Platform (external API)
  30. Credits All of the content from this presentation comes from

    other talks by engineers far smarter than myself • Lessons learned from scaling to 2000 engineers and 1000+ services - Matt Ranney • Uber Architecture: Moving Bits and Atoms at Scale - Andrii Iasynetskyi • Thanks to the many amazing people of the Marketplace Platform, Web Platform, Developer Platform, and Visualization Teams at Uber! • Come join us and work with these amazing people. Hiring globally (San Francisco, Amsterdam, Sofia, ...)