Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Writing a High Performance Database in Go

Writing a High Performance Database in Go

My talk from GopherCon 2014.

benbjohnson

April 24, 2014
Tweet

More Decks by benbjohnson

Other Decks in Technology

Transcript

  1. Things that need to be really f*cking fast Things that

    need to be pretty fast User Management Schema Management Query Parsing Backup / Recovery Bulk Data Insertion etc...
  2. Things that need to be really f*cking fast Things that

    need to be pretty fast Query Execution User Management Schema Management Query Parsing Backup / Recovery Bulk Data Insertion etc...
  3. Basics of Bolt Pure Go port of LMDB Memory-mapped B+tree

    MVCC, ACID transactions Zero copy reads
  4. Batch Size 1 Bolt Batch Benchmarks Performance 10 100 1000

    Baseline 9x Baseline 45x Baseline 90x Baseline Disclaimer: YMMV
  5. Use a channel to stream changes Transaction Coalescing Group changes

    into single transaction Either all changes commit or rollback
  6. See also: Albert Strasheim’s “Serialization in Go” Talk Encoding Performance

    http://www.slideshare.net/albertstrasheim/serialization-in-go https://github.com/cloudflare/goser
  7. // Create a byte slice with the same size as

    type T. var value = make([]byte, unsafe.Sizeof(T{}) // Map a typed pointer from the byte slice and update it. var t = (*T)unsafe.Pointer(&value[0]) t.ID = 123 t.MyIntValue = 20 // Insert value into database. db.Update(func(tx *bolt.Tx) error { return tx.Bucket(“T”).Put([]byte(“123”), value) }) Map a struct to a []byte
  8. // Start a read transaction. db.View(func(tx *bolt.Tx) error { c

    := tx.Bucket(“T”).Cursor() // Iterate over each value in the bucket. for k, v := c.First(); k != nil; k, v = c.Next() { var t = (*T)unsafe.Pointer(&value[0]) // ... do something with “t” ... } return nil }) Map a []byte to a struct