Couchbase Architecture-‐ ▪ Single Node Architecture and operations ▪ Clustered architecture and operations ▪ Cross Data Center Replication (XDCR) ▪ Indexing/Querying/Full-‐Text ▪ Couchbase Mobile Architecture ▪ Backup and Restore ▪ Security with Couchbase 4
7 High availability cache Key-‐value store Document database Embedded database Sync management Multi-‐purpose capabilities support a broad range of apps and use cases Enterprises often start with cache, then broaden usage to other apps and use cases
access, and update profile data to support user interactions, such as authentication, customer lookup, and preference tracking Internet of Things Ingest data from endpoint devices, such as sensors, smart meters, and other network devices, and scale to support massive datasets HA Caching Consistently low latency and high throughput access to any type of data, alleviating load on backend systems and maintaining snappy user experience
leader General purpose Simplified administration Cache Key Value Document Always-‐on availability 24 x 365 Enterprises choose Couchbase for several key reasons Local / Mobile
XDCR No manual sharding Database manages data movement to scale out -‐ Not the user Market’s only memory to memory database replication across clusters and geos Provides disaster recover / data locality Single Node Type Hugely simplifies management of clusters Easy to scale clusters by adding any # of nodes
cache Consolidated cache and database Tune memory required based on application requirements Flexible schemas with JSON Represent data with varying schemas using JSON on the server or on the device Index and query data with javascript views Cache Key Value Document Local / Mobile Couchbase Lite Light weight embedded DB for always available apps Sync gateway syncs data seamlessly with Couchbase server
Built-‐in enterprise class admin console Online software, hardware and DB upgrades Indexing, Compaction, Rebalance, backup & Restore Perform all administrative tasks with the click of a button Monitor status of the system visual at cluster level, database level, server level Restful APIs All admin operations available via UI, REST APIs or CLI commands Integrate third party monitoring tools easily using REST
x 365 Backup & Restore Full backup or Incremental backup with online restore Delta node catch-‐ups for faster recovery after failures In-‐memory replication with manual or automatic fail over Rack-‐zone awareness to minimize data unavailability Disaster Recovery Memory-‐to-‐memory cross cluster replication across data centers or geos Active-‐active topology with bi-‐directional setup
managed Cache Storage Engine DATA MANAGER 11210 / 11211 Data access ports 8092 Query API HTTP REST management API/Web UI Replication, Rebalance, Shard State Manager Erlang / OTP CLUSTE R MANAGE R 8091 Admin Console • Single-‐node type means easier administration and scaling • Single installation • Two major components/processes: Data manager cluster manager • Data manager: C/C++ Layer consolidation of caching and persistence • Cluster manager: Erlang/OTP Administration UI’s Out-‐of-‐band for data requests
Managed Cache Disk Queue Disk Replication Queue App Server Couchbase Server Node Doc 1 Doc 1 Doc 1 To other node • Single-‐node type means easier administration and scaling • Writes are async by default • Application gets acknowledgement when successfully in RAM and can trade-‐off waiting for replication or persistence per-‐ write • Replication to 1, 2 or 3 other nodes • Replication is RAM-‐based so extremely fast • Off-‐node replication is primary level of HA • Disk written to as fast as possible – no waiting
3 2 2 Disk Queue Replicatio n Queue App Server Couchbase Server Node Doc 1 Doc 1 Doc 1 Managed Cache Disk To other node • Single-‐node type means easier administration and scaling • Reads coming out of cache are extremely fast • No other process or system to communicate with • Data connection is a TCP-‐binary protocol
and scaling. Other technologies have multiple types and processes that need monitoring and analysis for scaling ▪ Built-‐in, managed caching layer: Super high-‐performance for reads and writes. Other technologies have much more limited caching layer or use filesystem, block-‐level buffering. Also, key separation of RAM from disk allows for much more consistent performance as data sets grow beyond RAM available. ▪ Replication and disk queues: Everything done asynchronously from RAM by default for best performance, application has the ability to wait (per-‐write) for replication and/or persistence. 19
21 ▪ A bucket is a logical, unique key space ▪ Multiple buckets can exist within a single cluster of nodes ▪ Each bucket has active and replica data sets ▪ Each data set has 1024 Virtual Bucket (vBuckets) ▪ Documents get logically mapped to vBuckets ▪ Document IDs always get hashed to the same virtual bucket ▪ Virtual buckets to do not have a fixed physical server location ▪ Mapping between the virtual buckets and physical server is called the cluster map ▪ Each virtual bucket contains 1/1024th portion of the data set vB Data buckets vB 1 ….. 1024 Virtual buckets
21 ▪ A bucket is a logical, unique key space ▪ Multiple buckets can exist within a single cluster of nodes ▪ Each bucket has active and replica data sets ▪ Each data set has 1024 Virtual Bucket (vBuckets) ▪ Documents get logically mapped to vBuckets ▪ Document IDs always get hashed to the same virtual bucket ▪ Virtual buckets to do not have a fixed physical server location ▪ Mapping between the virtual buckets and physical server is called the cluster map ▪ Each virtual bucket contains 1/1024th portion of the data set vB Data buckets vB 1 ….. 1024 Virtual buckets
Server Key SFO MQ2 Key vBucket (hash function) vBucket Server (table lookup) CRC 32 vBucket1 vBucket2 vBucket3 vBucket4 … vBucket1024 Server 1 Server 2 Server 3 Couchbase Cluster Server 4 Server 5
Server Key SFO MQ2 Key vBucket (hash function) vBucket Server (table lookup) CRC 32 vBucket1 vBucket2 vBucket3 vBucket4 … vBucket1024 Server 1 Server 2 Server 3 Couchbase Cluster Server 4 Server 5
SERVER 2 Active SERVER 3 Shard 5 Shard 2 Shard 9 Shard Shard Shard Shard 4 Shard 7 Shard 8 Shard Shard Shard Shard 1 Shard 3 Shard 6 Shard Shard Shard Couchbase Clustering 25 • Application has single logical connection to cluster (client object) • Data is automatically sharded resulting in even document data distribution across cluster
SERVER 2 Active SERVER 3 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Shard 5 Shard 2 Shard 9 Shard Shard Shard Shard 4 Shard 7 Shard 8 Shard Shard Shard Shard 1 Shard 3 Shard 6 Shard Shard Shard Replica Replica Replica Shard 4 Shard 1 Shard 8 Shard Shard Shard Shard 6 Shard 3 Shard 2 Shard Shard Shard Shard 7 Shard 9 Shard 5 Shard Shard Shard Couchbase Clustering 25 • Application has single logical connection to cluster (client object) • Data is automatically sharded resulting in even document data distribution across cluster • Each vbucket replicated 1, 2 or 3 times (“peer-‐to-‐ peer” replication) • Docs are automatically hashed by the client to a shard’ • Cluster map provides location of which server a shard is on
Active SERVER 2 Active SERVER 3 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 Shard 5 Shard 2 Shard 9 Shard Shard Shard Shard 4 Shard 7 Shard 8 Shard Shard Shard Shard 1 Shard 3 Shard 6 Shard Shard Shard Replica Replica Replica Shard 4 Shard 1 Shard 8 Shard Shard Shard Shard 6 Shard 3 Shard 2 Shard Shard Shard Shard 7 Shard 9 Shard 5 Shard Shard Shard Couchbase Clustering 25 • Application has single logical connection to cluster (client object) • Data is automatically sharded resulting in even document data distribution across cluster • Each vbucket replicated 1, 2 or 3 times (“peer-‐to-‐ peer” replication) • Docs are automatically hashed by the client to a shard’ • Cluster map provides location of which server a shard is on • Every read/write/update/delete goes to same node for a given key • Strongly consistent data access (“read your own writes”) • A single Couchbase node can achieve 100k’s ops/sec so no need to scale reads
REPLICA ACTIVE Doc 5 Doc 2 Doc Doc Doc 4 Doc 1 Doc Doc SERVER 1 REPLICA ACTIVE Doc 4 Doc 7 Doc Doc Doc 6 Doc 3 Doc Doc SERVER 2 REPLICA ACTIVE Doc 1 Doc 2 Doc Doc Doc 7 Doc 9 Doc Doc SERVER 3 SERVER 4 SERVER 5 REPLICA ACTIVE REPLICA ACTIVE Doc Doc 8 Doc Doc 9 Doc Doc 2 Doc Doc 8 Doc Doc 5 Doc Doc 6 READ/WRITE/UPDATE READ/WRITE/UPDATE APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 COUCHBASE SERVER CLUSTER User Configured Replica Count = 1 • Multiple nodes added or removed at once • One-‐click operation • Incremental movement of active and replica vbuckets and data • Client library updated via cluster map • Fully online operation, no downtime or loss of performance
ACTIVE Doc 5 Doc 2 Doc Doc Doc 4 Doc 1 Doc Doc SERVER 1 REPLICA ACTIVE Doc 4 Doc 7 Doc Doc Doc 6 Doc 3 Doc Doc SERVER 2 REPLICA ACTIVE Doc 1 Doc 2 Doc Doc Doc 7 Doc 9 Doc Doc SERVER 3 SERVER 4 SERVER 5 REPLICA ACTIVE REPLICA ACTIVE Doc 9 Doc 8 Doc Doc 6 Doc Doc Doc 5 Doc Doc 2 Doc 8 Doc Doc APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 User Configured Replica Count = 1 COUCHBASE SERVER CLUSTER • When node goes down, some requests will fail • Failover is either automatic or manual • Client library is automatically updated via cluster map • Replicas not recreated to preserve stability • Best practice to replace node and rebalance
ACTIVE Doc 5 Doc 2 Doc Doc Doc 4 Doc Doc SERVER 1 REPLICA ACTIVE Doc 4 Doc 7 Doc Doc Doc 6 Doc Doc SERVER 2 REPLICA ACTIVE Doc 1 Doc 2 Doc Doc Doc 7 Doc 9 Doc Doc SERVER 3 SERVER 4 SERVER 5 REPLICA ACTIVE REPLICA ACTIVE Doc 9 Doc 8 Doc Doc 6 Doc Doc Doc 5 Doc 2 Doc 8 Doc Doc Doc Doc 1 Doc 3 APP SERVER 1 COUCHBASE Client Library CLUSTER MAP COUCHBASE Client Library CLUSTER MAP APP SERVER 2 User Configured Replica Count = 1 COUCHBASE SERVER CLUSTER • When node goes down, some requests will fail • Failover is either automatic or manual • Client library is automatically updated via cluster map • Replicas not recreated to preserve stability • Best practice to replace node and rebalance
for application to define a shard key. No hot spots. Strongly consistent, immediately “read your own writes” ▪ Clustering and topology awareness: Abstraction layer for application means no architecture or config changes when growing cluster. ▪ Truly shared nothing: No single-‐point of failure and linearly scalable. ▪ No load balancer: Couchbase client library handles load balancing and distribution. ▪ Failover: Manual or automatic, activates replica vbuckets, prevents split-‐brain 28
Managed Cache Disk Queue Disk Replication Queue App Server Couchbase Server Node Doc 1 Doc 1 To other node XDCR Queue To other cluster Doc 1 Doc 1 • Single-‐node type means easier administration and scaling • Writes are async by default • XDCR is also RAM-‐based • XDCR happens between individual nodes – no gateway • All queues are processed in parallel
DOC DOC DOC DOC DOC DOC DOC 2 DOC 9 DOC 4 DOC 1 DOC 8 DOC 6 DOC 3 DOC 2 DOC 7 DOC 9 DOC 5 ACTIVE (RAM) DISK ACTIVE(RAM) DISK ACTIVE(RAM) DISK SERVER 1 SERVER 3 SERVER 2 DOC DOC DOC DOC DOC DOC DOC DOC 2 DOC 9 DOC 4 DOC 1 DOC 8 DOC 6 DOC 3 DOC 2 DOC 7 DOC 9 DOC 5 ACTIVE (RAM) DISK ACTIVE(RAM) DISK ACTIVE(RAM) DISK SERVER 1 SERVER 3 SERVER 2 New York City Datacenter Couchbase Server Cluster San Francisco Datacenter Couchbase Server Cluster
aware: Setup replication once, no need to change afterwards. Supports differently sized clusters ▪ Linearly Scalable: Node-‐node communication, no central gateway ▪ High-‐performance: RAM-‐RAM replication, designed for WAN connections ▪ Multi-‐master: All clusters able to service reads and/or writes, application controls. ▪ Automatic conflict resolution: Eventually consistent writes 34
operations ▪ Views: Incremental map-‐reduce, great for complex computations and handling very large datasets. ▪ Elastic Search: Couchbase as primary data-‐store, feeds Elastic Search for full-‐text querying ▪ Hadoop: Couchbase as operational data-‐store, dumping data to Hadoop for longer-‐ running analytics, potentially storing results back in Couchbase for application access. ▪ N1QL: Next-‐generation query language coming in 2015. “SQL-‐like”, extended for JSON 36
lightweight, native embedded JSON database Sync Gateway Synchronize on-‐device Couchbase Lite with Couchbase Server in the cloud Couchbase Server High performance, scalable, always-‐on JSON database in the cloud
▪ Admin user (full access) vs. Read-‐only user (monitoring, developer access) ▪ SSL encryption to REST API and Web UI ▪ HTTP access log ▪ Data Security: ▪ Applications connect via SASL with single user/pass ▪ Data-‐at-‐Rest encryption via partnership with Vormetric ▪ SSL encryption for over-‐the-‐wire ▪ Coming soon: ▪ LDAP/Kerberos integration ▪ More extensive administrative action auditing
backup and restore ▪ Built-‐in utilities: cbbackup / cbrestore ▪ Full, differential and cumulative backup available per-‐bucket. ▪ Restore from any point, to any bucket or topology
latency with high throughput; memory-‐centric architecture General purpose Simplified administration Easy to depl0y & manage; integrated Admin Console, single-‐click cluster expansion & rebalance Cache Key Value Document Cache, key value store, document database, and local/mobile database in single platform Always-‐on availability Data replication across nodes, clusters, and data centers 24 x 365 Enterprises choose Couchbase for several key reasons Local / Mobile
Database using familiar patterns and language ▪ The SDK does the heavy lifting for you – Cluster topology – Routing operations to the nodes – Dealing with the protocol(s) ▪ Mutualize work on performance and offer good abstractions 48
is a higher level abstraction ▪ Exposes RxJava Observables in the Asynchronous API ▪ Adds a Synchronous API on top of it ▪ Java 6+ (especially great for clients in Java 8!) 52 Core Java
could be leveraged to build SDKs for other JVM languages ▪ Also adapters for popular frameworks – There's a Spring Data connector for the 1.4 branch, to be upgraded to 2.x – Hadoop connector planned 53 Core Java Scala JRuby, ... Hadoop Spring Play!2
Cluster and Bucket both have async versions, obtained by calling async() method. ▪ Asynchronous API exposes RxJava Observables. ▪ Very rich and expressive API in terms of combinations and transformations. 57
document and extracting data for output bucket.async() .get("key1") .map(doc -‐> doc.content().getString("name")) .subscribe(name -‐> System.out.println("Hello " + name)) 59 This is from the async API, exposing an Observable<JsonDocument>
document and extracting data for output bucket.async() .get("key1") .map(doc -‐> doc.content().getString("name")) .subscribe(name -‐> System.out.println("Hello " + name)) 60 The Observable is a stream, can be connected to an Observer. It is the dual to Iterable/Iterator, in a push model instead of a pull model.
document and extracting data for output bucket.async() .get("key1") .map(doc -‐> doc.content().getString("name")) .subscribe(name -‐> System.out.println("Hello " + name)) 61 This is from RxJava
data for output bucket.async() .get("key1") .map(doc -‐> doc.content().getString("name")) .subscribe(name -‐> System.out.println("Hello " + name)) 62 One of the simplest transformation operators, T -‐> R Here gets a JsonDocument and extract the name String value Resulting in an Observable<String>
data for output bucket.async() .get("key1") .map(doc -‐> doc.content().getString("name")) .subscribe(name -‐> System.out.println("Hello " + name)) 63 Then we subscribe, connecting an Observer to the stream "For each name in the stream, print out "Hello name" "
▪ onNext – each time a new item is emitted in the stream ▪ onError – terminating event in case of Exception ▪ onCompleted – terminating event when the stream is finished – an infinite stream could exist, not calling this 64
http://docs.couchbase.com/developer/java-‐2.0/overview.html – Section on RxJava ▪ ReactiveX http://reactivex.io – RxJava is now part of ReactiveX, which offers Rx bindings in other languages ▪ RxJava has a great documentation, check it out! https://github.com/ReactiveX/RxJava/wiki 67
simonbasle
happymana
tasshi
kakao
thehaigo
kyoheig3
otakakot
polpielladev
seike460
tsoshiro
fukubaka0825
yusukebe
kohecchi
tanoku
eileencodes
wjessup
chriscoyier
3n
moore
geoffreycrofte
geeforr
skipperchong
tenderlove
pedronauck
rmw
