Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyCon2014China-Zhuhai-luna kv db

Zoom.Quiet
November 17, 2014

PyCon2014China-Zhuhai-luna kv db

Zoom.Quiet

November 17, 2014
Tweet

More Decks by Zoom.Quiet

Other Decks in Technology

Transcript

  1. Nginx日 志 Nginx日 志 体系结构 解析、压缩 解析、压缩 分片发送 分片发送 RabbitMq(消息系统)

    RabbitMq(消息系统) 业务计算 业务计算 KV库 KV库 任务调度 任务调度 报表展示 报表展示 这是KV库的位置
  2. 实时计算 离线计算 消息发送 Nginx服务端 Nginx服务端 Infoc队列 Infoc队列 实时队列 实时队列 Infoc流程

    Infoc流程 二进制解析成字 符,并做gzip压 缩 二进制解析成字 符,并做gzip压 缩 1)压缩后的文本不超 过128KB,日志延时 不超过20秒。 2)发送失败重试IP, 3)网络失败本地缓存 1)压缩后的文本不超 过128KB,日志延时 不超过20秒。 2)发送失败重试IP, 3)网络失败本地缓存 消息队列 消息队列
  3. 解决IO和稳定性 work_dir = /dev/shm/ssdb/master1 pidfile = /dev/shm/ssdb/master1/ssdb.pid server: ip: 172.28.26.178

    port: 8888 replication: binlog: yes # Limit sync speed to *MB/s, -1: no limit sync_speed: -1 slaveof: type: sync ip: 172.28.26.175 port: 8889 leveldb: cache_size: 500 block_size: 32 write_buffer_size: 64 compaction_speed: 1000 compression: yes ssdb.conf work_dir = /data/ssdb/slave1 pidfile = /data/ssdb/slave1/ssdb.pid server: ip: 172.28.26.175 port: 8889 replication: binlog: yes sync_speed: 100 slaveof: id: svc_1 type: sync ip: 172.28.26.178 port: 8888 leveldb: cache_size: 500 block_size: 32 write_buffer_size: 64 compaction_speed: 200 compression: yes ssdb_slave.conf
  4. LunaProxy核心代码 HOST_PORT_CONF = {"ball_click":{"status":"split", "0":("192.28.26.100", "10000"), "1":("192.28.26.100", "10001"), "2":("192.28.26.100", "10002"),

    "f":("192.28.26.100", "10016")}, “uninstall”:{“status”:“nosplit”, “db”:(“192.28.26.200”, “10000”)}} #真实的运行环境中用zookeeper实现配置管理 def is_split(name): is_split(name): is_split(name): is_split(name): return True if HOST_PORT_CONF[name]["status"] == "split" else False def get_host_port(name, split_key=None): get_host_port(name, split_key=None): get_host_port(name, split_key=None): get_host_port(name, split_key=None): if split_key: return HOST_PORT_CONF[name][split_key] return HOST_PORT_CONF[name]["db"] �SSDB可配置
  5. LunaProxy核心代码 def __multi_run(name, kset, func): __multi_run(name, kset, func): __multi_run(name, kset,

    func): __multi_run(name, kset, func): results = [] kmap = {} for key in kset: if key[0] not in kmap: kmap[key[0]] = set([]) kmap[key[0]].append(key) i , threadpool = 0, [] for split_key, hset in kmap.iteritems(): results.insert(i, None) tt = threading.Thread(target=func, args=(name, split_key, hset,results[i])) tt.start() threadpool.append(tt) i = i+ 1 for i in xrange(0, len(kmap), 1): threadpool[i].join() return results �线性扩展
  6. Q&A