Building an API with Go at Microco.sm

18k lines of Go code later, I decided to condense some of what I'd learned in the last six months into a half-hour talk. Given at Go London User Group, July 2013

1. APIs at Matt Cottingham http://microco.sm matt@microcosm.cc @mattrco Image credit: Renée French
2. Platform overview • Hosted platform for community forums • API is 18k lines of Go code • API documentation: http: //microcosm-cc.github.io • Alpha site: http://meta.microco.sm Microco.sm
3. Engineering overview Requirements • API does everything: all clients (including ours) built on it • Performance constraints (sub 50ms response times) • Runtime configurable • Scheduled tasks • Up-front schema design • Pluggable authentication • Websocket support API DATA CLIENT AUTH
4. Why Go? Plusses Potential problems? • Third-party packages (e.g. oAuth, database support) • How much will change around 1.1? • Who's using it? Will we be able to hire Go programmers? • Type system • Powerful standard library • Small core language, easy to learn • Concurrency primitives (this is how things looked before I knew much about Go, i. e. six months ago)
5. Implementation Stack • PostgreSQL 9.3 • Go 1.1 (we were following tip up to 1.1. We'll upgrade to 1.1.1 shortly) • Memcached • Nginx (TLS, SPDY, gzip, etc.)
6. Implementation • Request processor inspired by goweb • gorilla/mux: request routing • lib/pq: pure Go PostgreSQL driver Go packages we use • Memcached client: bradfitz/gomemcache • Cron: robfig/cron • Configuration: laurent22/toml-go
7. Example route declarations Implementation
8. Implementation Simple request Handler (switch on HTTP method)
9. Implementation Basic controller
10. Implementation Persistence • What about struct-to-database-table mappings? • Several ORM options, e.g. • https://github.com/coopernurse/gorp/ • https://github.com/eaigner/hood • but with lib/pq we just talk to PostgreSQL through prepared statements (more maintenance but finer query control)
11. Implementation Storing data (receiver enables `m.Insert()` syntax)
12. Implementation Caveats/compromises • Documented examples are quite sparse for database access • database/sql could not (and should not) model all column datatypes, so some are provided by lib/pq • Dealing with SQL NULL is quite simple, `thing.Valid` idiom:
13. S3 file storage • We use the package from Juju (by Canonical) • There are lots of packages, well worth reading the source: • https://launchpad.net/juju-core • Go and Juju at Canonical Implementation
14. Deployment • `scp binary server` (yep...) • upstart to manage processes (may move to supervisord) • No graceful restarts. There's some work going on for this, but we'll likely just remove nodes from the load balancer, restart, add them back in. Systems
15. Systems Logging and monitoring • stdlib logging package does syslog or user-defined formats • fine for capturing panics etc. • riemann or heka are possibilities for application monitoring • must accept a payload over UDP, not Go specific
16. Flotsame & Jetsam Performance • Web framework benchmarks (uses Go 1.1) • Benchmark source • Go compares favourably in tests that look realistic • We haven't (yet) tested memory usage at e.g. high concurrency levels
17. Reading • the official Go blog, Dave Cheney's blog, and go-nuts • Go at Apcera • Russ Cox on regex implementation is an interesting read • Go Newsletter (disclaimer: by me!) Close