Calling CloudStack: Building a Phone Company One Zone at a Time

This talk will dive into using CloudStack as the IaaS foundation for building a world-class wireless telecommunications architecture. We'll cover the unique requirements in telephony and how CS lets us handle these in new and wonderful ways.

Transcript

1. BUILDING A PHONE COMPANY One Zone at a Time Evan McGee 1
2. A LITTLE BACKGROUND... • In operation since 2004 • Began offering wireless service in late 2012 • Running production on CloudStack 3.0.2 since Day 1 • Marrying the webscale with the telescale • Machine time vs Human time. 2
3. WHAT IS AN MVNO? Mobile Virtual Network Operator • A wireless company that doesn’t own it’s own spectrum or wireless infrastructure • Multiple flavors of MVNO depending on technical desire • Not necessarily tied to a particular wholesale system - can usually work with multiple partners. • Requires higher startup capital and greater amounts of technical knowledge MNO MVNO Billing CRM Accounts Marketing Service Inventory Sales Pricing 3
4. WHAT IS AN MVNE? Mobile Virtual Network Enabler • Works with major network operators to enable wireless services for 3rd parties. • Takes care of user account management, provisioning, billing, taxes, etc... • Can provide service to an aggregated pool of MNOs or partner more tightly with one • Isolates MVNO to focus on core marketing/branding while maintaining focus on backend logists. MNO MVNO Billing CRM Accounts Marketing Service Inventory Sales Pricing MVNE 4
5. THE • Equipment available from few vendors at very high $$$ • MVNOs limited to simply reselling service • Unique branding/customer service was the only differentiator. Historical Perspective WAY 5
6. HISTORICAL PERSPECTIVE 6
7. THE • Wireless is moving to all packet-switched networks • Resurgence in MVNOs due to local/niche markets • Opportunities to bring Internet services to the carrier-level • Commodity hardware + CloudStack = huge savings Today WAY 7
8. SO WHY? 8
9. WHY CLOUDSTACK • High code maturity and stability in 2012 • Simple initial install and use - Ease of Convincing • Open-Source, Top-Level Apache Project • “No one ever got fired for using Apache.” - Internet • Active development / support network • Good documentation 9
10. OUR 10
11. • Custom-built MVNE & MVNO System • Four Main Components (for simplicity) • Each component isolated on a guest network • Identical stacks deployed to Production, Staging, and Dev Core API Web SQL OUR 10.1.1/24 10.1.2/24 10.1.3/24 10.1.4/24 108.50.50.5 108.50.51.5 108.50.52.5 108.50.53.5 Internet MNO 11
12. OUR • Datacenters are located in Manhattan near Tier-1 providers • Latency, jitter, round-trip times are important • Hosts on Centos 6.4 using KVM and Virtual Routers • Looking seriously at vSwitch • VMs are primarily Debian (with a few others for spice) 12
13. Freeswitch, Kamailio, Adhearsion PostgresQL Rails, Node, Redis Rails, Node, Nginx, Spree OUR Core API Web SQL 13
14. • Very heavy network IO - each call is 87Kbps symmetric • Saturate a gigabit NIC at ~8-10,000 simultaneous calls • Transcoding audio/video requires high CPU • Why do we proxy the media? • Constantly monitoring the state of the calls for in- call app functionality • XMPP gateways for future WebRTC 3PCC OUR Core 14
15. • Very heavy disk IO • Lots of writes - ACID compliance • Call data, billing records, account management, webside services, etc... • Lots of reads • Inbound call account lookups, least-cost routing, etc... • Experimenting with bare metal OUR SQL 15
16. • Stateless and Lightweight, Multi-Tenanted • Two Versions - Core API and 3rd Party API • Each VM carries a copy of the full API • Utilizes VR load balancing to scale across instances • Monitor CPU and Memory, spin up more if necessary • Works fantastically well! OUR API 16
17. • Typical Rails-Nginx-Unicorn Webstack • Running Spree e-commerce to accomidate phone sales • Ties directly into API for provisioning and account management • Allows multi-tenanted management of all MVNO functions OUR Web 17
18. WHAT’S MOST IMPORTANT • High Availability - All of the 9’s • Fast performance - but on a human scale • A different scale than most web services; process than take up to multiple seconds are acceptable (if undesired) • Possibly only 100,000x-1MM transactions an hour, or 8MM at one time. • Viral growth isn’t likely - although I’d be happy to be wrong! 18
19. US 19
20. DEPLOYMENT STRATEGY 20
21. CONTINUOUS INTEGRATION TELCO STYLE • All code stored in Git - 95% unit testing, CI with Travis • Deploy code to web-based services ASAP -- usually as soon as comitted and passed tests. • Some of our core softswitch source can take a long time to build, so ignore that and only update repo code • VMs are updated by Chef for the selected roles 21
22. CLOUDSTACK + CHEF • Chef - Used to bootstrap / update master images • Roles defined in the Chef cookbooks • Some of our software can take a long time to build, so ignore that and only update repo code • VMs are updated by Chef for the selected roles 22
23. CLOUDSTACK + CHEF • Do we currently use Chef to update existing images? • Depends on component: API, Web -- Yes! • Core... sometimes.... • When updating softswitch components, update the Master image then recreate the stack deployment. • Destroy the old VMs as the new ones are provisioned 23
24. CLOUDMONKEY • As of our transition to 4.1, we’ve started heavily using Cloudmonkey • Currently used to auto-snapshot and templatize the images created by Chef • Either Cloudmonkey or Knife-CloudStack then used by internal tools to deploy template VMs to appropriate clusters • Building Cloudmonkey into more and more internal deploy tools - keep an eye out! 24
25. MONITORING • We need to keep a close eye on CPU / Memory / Storage / Network IO depending on the section • Zabbix - agentless monitoring • We usually put an agent on anyway. • Internal tools scan logs for irregularities and start the VM setup/teardown process • Scale to public cloud in an emergency 25
26. DISASTER RECOVERY 26
27. MULTIPLE ZONES • Currently building out multiple geographic zones • Useful for keeping short average path length in call • Slightly limited by MNO ingress/egress points • MNO is the ultimate backup: will process all calls. • Billing is batch processed after connectivity is restored 27
28. DATA / SERVICE BACKUP • Critical billing data currently backup up off-site • Possibilities of using S3 for non-critical/anonymous data • Slightly limited by MNO ingress/egress points • MNO is the ultimate backup: will process all calls. • Billing is batch processed after connectivity is restored 28
29. WHAT’S NEXT? 29
30. UPCOMING GOALS • Hadoop Deployment • Telecom has a classic “big data” problem • Currently passing off data to Elastic MapReduce • Bring that in house, expose results to users/clients via API 30
31. O T T SERVICES OVER THE TOP SERVICES In-Call Translation Post Call Transcription Gameification Blacklisting 31
32. M2M MACHINE-2-MACHINE 32
33. • Browser-based communications • Encrypted by default • Peer-to-Peer (or via proxy if desired) for both media and data • Exciting possibilities abound 33
34. MAKE THE PEOPLE HAPPY http://www.churchesgoingmobile.com/wp-content/themes/canvas/functions/thumb.php?src=wp-content/uploads/2012/07/girls_mobile_phone_laughing-300x285.jpg&w=940&h=350&zc=1&q=90 34
35. LIVE THE TELECOM FUTURE Allow people to wander and pluck their own data OAuth2 API for all accounts, tenant-to-user level The Data Silo becomes the Data Prairie 35
36. Evan McGee @startledmarmot / Twitter @emcgee / Github evan@ringplus.net 36