Hacking CloudFoundry

This talk was given at CloudFoundry Open Tour San Francisco. An overview of CloudFoundry architecture is given, follow by detailed examples of extensions to CloudFoundry built by AppFog and hints for building your own.

Transcript

1. Jeremy Voorhis Senior Engineer, AppFog Inc jeremy@appfog.com @jvoorhis http://appfog.com/ Hacking CloudFoundry Sunday, April 1, 12
2. Overview • Survey VCAP architecture • Example: Creating a new VCAP component • Example: Integrating OSS monitoring tools • Example: Adding runtime support Sunday, April 1, 12
3. CloudFoundry Architecture Sunday, April 1, 12
4. • http://github.com/cloudfoundry • NATS – messaging • VCAP – applications and services • VMC – API client • 10 more at the time of this talk Open Source Projects Sunday, April 1, 12
5. VCAP Design Principles • Loosely coupled components • Fail fast • Minimize single points of failure • Infrastructure unaware Sunday, April 1, 12
6. Loosely Coupled Components • Collection of single purpose daemons • Connected by pub/sub and HTTP • Distributed state • System state controlled by REST API Sunday, April 1, 12
7. Fail Fast • Optimized for mean-time-to-recovery • Detect unrecoverable errors (i.e. dropped connections), log and exit(1) • Process supervisor (i.e. Monit) strongly advised • Components are easily replaceable Sunday, April 1, 12
8. Minimize Single Points of Failure • Most components scale horizontally • Singletons • CCDB – use replication! • HealthManager – intermittent failures don’t disrupt service • NATS – cluster work in progress • Service daemons Sunday, April 1, 12
9. Infrastructure Unaware • Dev/Test: OS X, Micro CloudFoundry • Public Cloud: AWS, Rackspace, HP, Joyent, more... • Data Centers: vSphere, Bare metal • Ubuntu 10.04 LTS recommended Sunday, April 1, 12
10. Implementation Notes • UNIX daemons • Written in Ruby • EventMachine library for asynchronous I/O • Discoverable via NATS • HTTP APIs for rolling metrics and health checks Sunday, April 1, 12
11. CloudFoundry Components • NATS • Router • CloudController • DEA • HealthManager • Service Daemons Sunday, April 1, 12
12. VCAP Topology Sunday, April 1, 12
13. NATS • The “Nervous System” • Pub/Sub message bus • Messages have a topic + payload • Pattern matching for topics • Many communication patterns • Supports all CloudFoundry components Sunday, April 1, 12
14. Router • Proxies HTTP to hosted apps and CloudFoundry components • Discovers routes to backends via NATS • Random load balancing Sunday, April 1, 12
15. CloudController • Control systems via REST + JSON API • Single point of truth for accounts (CCDB) • Users • Apps • Services Sunday, April 1, 12
16. DEA • “Droplet Execution Agent” • Starts and stops apps • Allocates resources and enforces resource consumption • Announces app state transitions via NATS • Supports rolling restarts Sunday, April 1, 12
17. HealthManager • Monitors system for drift • i.e. # running instances / app • Signals to CloudController when something isn’t right • Only non-CC component connecting to CCDB Sunday, April 1, 12
18. Service Daemons • Services external resources that you can bind to your app • e.g. RDBMS, Key/Value Store, Filesystem, Message Queues, SMTP • Nodes control individual service installations • e.g. Create database and grant permissions for MySQL • Gateways broker nodes to system • Selects a node for provisioning Sunday, April 1, 12
19. Putting it together: pushing an app • Client POSTs app metadata, creates app in system • Client PUTs needed files • CloudController stages app, discovers DEA to run it • DEA pulls app package, announces route to app on NATS • Router discovers route, forwards traffic Sunday, April 1, 12
20. Example: VCAP Component for Cache Control • HTTP caching reverse proxy • Load balancer • Stores cache in virtual memory Sunday, April 1, 12
21. HTTP Caching: Motivation • Influenced by PHP Fog • Many read-heavy sites (e.g. Drupal, Wordpress) • Cache control headers often incorrect • Purge cached content on deploy • Doesn’t replace a CDN, but really helps! Sunday, April 1, 12
22. Deploying Varnish ELB Router DEA ELB Router DEA Varnish Sunday, April 1, 12
23. Varnishing • New VCAP component • Deployed on Varnish nodes • App deployments trigger purge command • Reports activity and latency via /varz API • < 300 LoC Sunday, April 1, 12
24. Varnishing Boilerplate • Read config • Register as VCAP component • Setup NATS subscriptions • Write PID file, trap signals, enter run loop Sunday, April 1, 12
25. Varnishing Logic • Subscribe to ‘dea.*.start’ • Scripting varnishadm • Authenticate with secret • Issues purge command for each route assoc’d with app Sunday, April 1, 12
26. Example: Monitoring • Monitoring infrastructure is a solved problem • What about monitoring CloudFoundry? • Problems: • System health • Capacity forecasting • QoS Sunday, April 1, 12
27. Monitoring VCAP: Goals • Leverage existing systems • IaaS agnostic • Alerts for on-call rotation • Correlate reservation metrics with usage • Aggregate service data (i.e. % mem reserved across DEA pool) • Detect changes in clusters automatically Sunday, April 1, 12
28. Collecting VCAP Data • Every component has a JSON + REST API • Components are discoverable via NATS • Capability security model • Credentials are generated at boot (UUID / UUID) • http://user:pass@component.int/varz Sunday, April 1, 12
29. Monitoring Prototype • Ad hoc solution: dashboard built with NodeJS • Beautiful graphs • Gained experience plumbing VCAP • Unsolved problems: • Inventory • Correlation with lower layer • Alerting Sunday, April 1, 12
30. Our Monitoring Stack Sunday, April 1, 12
31. varz-query • NodeJS CLI • Discovers components and presents metrics • < 100 LoC • Added support for timed request/response to node_nats Sunday, April 1, 12
32. Check_MK Configuration • Inventories VCAP components • VCAP-specific checks implemented in Python • Submits results to Nagios • WARNING and CRITICAL results reported to PagerDuty Sunday, April 1, 12
33. Chef’s Role • Generates main.mk • Nodes added automatically when provisioned • Deploys Check_MK agent, varz-query Sunday, April 1, 12
34. Putting It All Together varz-query NATS VCAP Sunday, April 1, 12
35. Example: Adding PHP Support • AppFog is the CloudFoundry Community Lead for PHP • We created and actively develop PHP Fog • We contributed our PHP experience back to CloudFoundry Sunday, April 1, 12
36. Our PHP Stack • Apache2 • PHP community widely depends on mod_rewrite, .htaccess • PHP 5.3 • Leverages UNIX for isolation • Every app runs under a unique UID / GID • Same effect as VCAP’s secure mode Sunday, April 1, 12
37. Our PHP Stack (Cont’d) • Uses Apache2 vhosts in multi-tenant environment • mpm-itk module allows workers to drop privileges Sunday, April 1, 12
38. How VCAP Represents Apps • Framework • App deployment know-how • e.g. Sinatra, Rails • Runtime • VMs, interpreters, common libraries • e.g. ruby-1.8.7-p, ruby-1.9.3-p125 • Constrained by framework Sunday, April 1, 12
39. App Packages • Contain source code and dependency metadata • Two flavors: • Unstaged – just the app bits • Staged – includes start/stop scripts, dependencies Sunday, April 1, 12
40. Staging Apps • Stager • Builds staged app packages • Plugin architecture Sunday, April 1, 12
41. Implementing a Staging Plugin • Inherit from abstract StagingPlugin class • MUST implement #framework, #stage_application • MAY specialize further Sunday, April 1, 12
42. Implementing a Staging Plugin (Cont’d) • Staging manifest • Runtimes • App servers • Detection rules Sunday, April 1, 12
43. Staging Plugin for PHP • Runs Apache2 in foreground • Generates and bundles Apache2 and PHP configs • Full support for environment vars, service bindings, resource limits • Available now in open source project Sunday, April 1, 12
44. Runtimes • Deploy multiple runtime versions on any node • DEA pool can be heterogeneous • Versions can be upgraded without migrating existing apps Sunday, April 1, 12
45. Runtime Rollout • CloudController: • config file • App::Runtimes # FIXME • Staging manifests • DEA: • dea.yml • Executables, stdlibs, cached dependencies Sunday, April 1, 12
46. Going Further: A Custom DEA • Motivation: multi-tenant PHP for free tier • Married PHPFog’s stack with VCAP’s secure user pool • mpm-itk • Subclassed DEA::Agent Sunday, April 1, 12
47. Go Forth And Hack! • http://github.com/cloudfoundry • http://cloudfoundry.org/ • http://appfog.com/ Sunday, April 1, 12
48. Jeremy Voorhis Senior Engineer, AppFog Inc jeremy@appfog.com @jvoorhis http://appfog.com/ Thank You! Sunday, April 1, 12
49. Questions? Sunday, April 1, 12