95% Symbology requests should fit in 50ms. 95% Suggest requests should fit in 25ms. • Use normalized data • Use less memory as much as possible • Fast data loading into DB • Windows environment and .Net plaZorm
• Oracle TimesTen • RelaAonal • Completely in-‐memory: guaranteed latency but slow startup • Expensive • McObject’s ExtremeDb • Object DB • NaAve C interface: designed for performance • Ultra reliability • SAll expensive What we considered from commercial databases
for .NET available) • Simple maintenance (monitoring, replicaAon, sharding) • Data is stored in-‐memory once used. • 1ms average response Ame • Cross-‐plaZorm (naAve Windows support)
space taken by the data, since we need it to fit into RAM • Data is Text only (no pictures etc) • Full document required always • Only some fields are used to query data, and these fields are short (3..10 symbols) • New fields should be easily added to the “queryable” list • Composite queries are needed someAmes • AB and CD and not EF or GH • Fast data loading Symbology DB: challenge
syntax is the same • Single index – less space occupied • Easy to add new searchable data "queryablefields":[{ "k": 1, "v": "MSFT.O" },{ "k": 2, "v": "Microsoft Inc." } ] Symbology DB: soluAon
field (binary data) • Encode with Protocol Buffers or MsgPack – In our case, MsgPack 2x faster than Protobuf • Zip with Snappy – Fastest algorithm in the world. { "b" : BinData(0,"CgcxMDkwMzcwEgZ1cztJQk0xAAAAAAAA8D86A 05ZU0IXTmV3IFl vcmsgU3RvY2sgRXhjaGFuZ2VZAAAAAAAA8D9gAXABeAGJAQA AAAAAAPA/ogEFNDc0MU6qAQU0NzQxTrI…“) } Symbology DB: soluAon
About 750 mln temporary documents • MongoDB Map Reduce is too slow • All MongoDB based algorithms takes a lot of Ame Use Amazon ElasAc MapReduce! Suggest DB: soluAon