Share
Embed
Building Applications with DynamoDB
Amazon DynamoDB is a managed NoSQL database. These slides introduce DynamoDB and discuss best practices for data modeling and primary key selection.
Matt Wood
May 16, 2012
Transcript
129 slides
-
DynamoDB Building Applications with An Online Seminar - 16th May
2012 Dr Matt Wood, Amazon Web Services -
DynamoDB is a managed NoSQL database service. Store and retrieve
any amount of data. Serve any level of request traffic. -
Flexible data model. Key/attribute pairs. No schema required. Easy to
create. Easy to adjust. -
Two decisions + three clicks = ready for use Primary
keys + level of throughput -
Provisioned throughput. Reserve IOPS for reads and writes. Scale up
(or down) at any time. -
Write throughput. $0.01 per hour for 10 write units Units
= size of item x writes/second -
Read throughput. $0.01 per hour for 50 read units Provisioned
units = size of item x reads/second strongly consistent eventually consistent -
Read throughput. $0.01 per hour for 100 read units Provisioned
units = size of item x reads/second 2 strongly consistent eventually consistent -
Read throughput. Mix and match at “read time”. Same latency
expectations. strongly consistent eventually consistent -
$create_response = $dynamodb->create_table(array( 'TableName' => 'ProductCatalog', 'KeySchema' => array( 'HashKeyElement'
=> array( 'AttributeName' => 'Id', 'AttributeType' => AmazonDynamoDB::TYPE_NUMBER ) ), 'ProvisionedThroughput' => array( 'ReadCapacityUnits' => 10, 'WriteCapacityUnits' => 5 ) )); -
Authentication. Session based to minimize latency. Uses Amazon Security Token
Service. Handled by AWS SDKs. Integrates with IAM. -
Libraries, mappers & mocks. http://j.mp/dynamodb-libs ColdFusion, Django, Erlang, Java, .Net,
Node.js, Perl, PHP, Python, Ruby -
Items are a collection of attributes. Each attribute has a
key and a value. An item can have any number of attributes, up to 64k total. -
Two scalar data types. String: Unicode, UTF8 binary encoding. Number:
38 digit precision. Multi-value strings and numbers. -
id = 100 date = 2012-05-16-09-00-10 total = 25.00 id
= 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 -
id = 100 date = 2012-05-16-09-00-10 total = 25.00 id
= 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Table -
id = 100 date = 2012-05-16-09-00-10 total = 25.00 id
= 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Item -
id = 100 date = 2012-05-16-09-00-10 total = 25.00 id
= 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Attribute -
Where is the schema? Tables do not require a formal
schema. Items are an arbitrary sized hash. Just need to specify the primary key. -
Items are indexed by primary key. Single hash keys and
composite keys. -
id = 100 date = 2012-05-16-09-00-10 total = 25.00 id
= 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Hash Key -
id = 100 date = 2012-05-16-09-00-10 total = 25.00 id
= 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Hash Key Range Key + -
Conditional updates. PutItem, UpdateItem, DeleteItem can take optional conditions for
operation. UpdateItem performs atomic increments. -
One API call, multiple items. BatchGet returns multiple items by
primary key. BatchWrite performs up to 25 put or delete operations. Throughput is measured by IO, not API calls. -
Query vs Scan Query for composite key queries. Scan for
full table scans, exports. Both support pages and limits. Maximum response is 1Mb in size. -
Query patterns. Retrieve all items by hash key. Range key
conditions: ==, <, >, >=, <=, begins with, between. Counts. Top and bottom n values. Paged responses. -
1. Mapping relationships with range keys. No cross-table joins in
DynamoDB. Use composite keys to model relationships. Patterns -
Data model example: online gaming. Storing scores and leader boards.
Players with high Scores. Leader board for each game. -
Data model example: online gaming. Storing scores and leader boards.
Players with high Scores. Leader board for each game. user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key -
Data model example: online gaming. Storing scores and leader boards.
Players with high Scores. Leader board for each game. user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key -
Data model example: online gaming. Storing scores and leader boards.
Players with high Scores. Leader board for each game. user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards: composite key -
Data model example: online gaming. Storing scores and leader boards.
user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards: composite key Scores by user (and by game) -
Data model example: online gaming. Storing scores and leader boards.
user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards: composite key High scores by game -
2. Handling large items. Unlimited attributes per item. Unlimited items
per table. Max 64k per item. Patterns -
Data model example: large items. Storing more than 64k across
items. message_id = 1 part = 1 message = <first 64k> message_id = 1 part = 2 message = <second 64k> message_id = 1 part = 3 joined = <third 64k> Large messages: composite keys Split attributes across items. Query by message_id and part to retrieve. -
Store a pointer to objects in Amazon S3. Large data
stored in S3. Location stored in DynamoDB. 99.999999999% data durability in S3. Patterns -
Data model example: secondary indices. Storing more than 64k across
items. user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Users: hash key -
Data model example: secondary indices. Storing more than 64k across
items. user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys -
Data model example: secondary indices. Storing more than 64k across
items. Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys Second name index: composite keys last_name = Wood user_id = mza last_name = Fox user_id = mattfox last_name = Vogels user_id = werner user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels -
last_name = Wood user_id = mza last_name = Fox user_id
= mattfox last_name = Vogels user_id = werner user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Data model example: secondary indices. Storing more than 64k across items. Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys Second name index: composite keys -
last_name = Wood user_id = mza last_name = Fox user_id
= mattfox last_name = Vogels user_id = werner user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Data model example: secondary indices. Storing more than 64k across items. Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys Second name index: composite keys -
4. Time series data. Logging, click through, ad views, game
play data, application usage. Non-uniform access patterns. Newer data is ‘live’. Older data is read only. Patterns -
Data model example: time series data. Rolling tables for hot
and cold data. event_id = 1000 timestamp = 2012-05-16-09-59-01 key = value event_id = 1001 timestamp = 2012-05-16-09-59-02 key = value event_id = 1002 timestamp = 2012-05-16-09-59-02 key = value Events table: composite keys -
Data model example: time series data. Rolling tables for hot
and cold data. event_id = 1000 timestamp = 2012-05-16-09-59-01 key = value event_id = 1001 timestamp = 2012-05-16-09-59-02 key = value event_id = 1002 timestamp = 2012-05-16-09-59-02 key = value Events table: composite keys Events table for April: composite keys Events table for January: composite keys event_id = 400 timestamp = 2012-04-01-00-00-01 event_id = 401 timestamp = 2012-04-01-00-00-02 event_id = 402 timestamp = 2012-04-01-00-00-03 event_id = 100 timestamp = 2012-01-01-00-00-01 event_id = 101 timestamp = 2012-01-01-00-00-02 event_id = 102 timestamp = 2012-01-01-00-00-03 -
Hot and cold tables. Jan April May Feb Mar higher
throughput Dec Patterns -
Hot and cold tables. Jan April May Feb Mar higher
throughput lower throughput Dec Patterns -
Hot and cold tables. Jan April May Feb Mar data
to S3, delete cold tables Dec Patterns -
Not out of mind. DynamoDB and S3 data can be
integrated for analytics. Run queries across hot and cold data with Elastic MapReduce. Patterns -
Uniform workloads. DynamoDB divides table data into multiple partitions. Data
is distributed primarily by hash key. Provisioned throughput is divided evenly across the partitions. -
Uniform workloads. To achieve and maintain full provisioned throughput for
a table, spread your workload evenly across the hash keys. -
Non-uniform workloads. Some requests might be throttled, even at high
levels of provisioned throughput. Some best practices... -
1. Distinct values for hash keys. Patterns Hash key elements
should have a high number of distinct values. -
Data model example: hash key selection. Well distributed work loads
user_id = mza first_name = Matt last_name = Wood user_id = jeffbarr first_name = Jeff last_name = Barr user_id = werner first_name = Werner last_name = Vogels user_id = mattfox first_name = Matt last_name = Fox ... ... ... Users -
Data model example: hash key selection. Well distributed work loads
user_id = mza first_name = Matt last_name = Wood user_id = jeffbarr first_name = Jeff last_name = Barr user_id = werner first_name = Werner last_name = Vogels user_id = mattfox first_name = Matt last_name = Fox ... ... ... Users Lots of users with unique user_id. Workload well distributed across user partitions. -
2. Avoid limited hash key values. Patterns Hash key elements
should have a high number of distinct values. -
Data model example: small hash value range. Non-uniform workload. status
= 200 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 status 404 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 Status responses -
Data model example: small hash value range. Non-uniform workload. status
= 200 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 status 404 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 Status responses Small number of status codes. Unevenly, non-uniform workload. -
3. Model for even distribution of access. Patterns Access by
hash key value should be evenly distributed across the dataset. -
Data model example: uneven access pattern by key. Non-uniform access
workload. mobile_id = 100 access_date = 2012-04-01-00-00-01 mobile_id = 100 access_date = 2012-04-01-00-00-02 mobile_id = 100 access_date = 2012-04-01-00-00-03 mobile_id = 100 access_date = 2012-04-01-00-00-04 ... ... Devices -
mobile_id = 100 access_date = 2012-04-01-00-00-01 mobile_id = 100 access_date
= 2012-04-01-00-00-02 mobile_id = 100 access_date = 2012-04-01-00-00-03 mobile_id = 100 access_date = 2012-04-01-00-00-04 ... ... Devices Large number of devices. Small number which are much more popular than others. Workload unevenly distributed. Data model example: uneven access pattern by key. Non-uniform access workload. -
mobile_id = 100.1 access_date = 2012-04-01-00-00-01 mobile_id = 100.2 access_date
= 2012-04-01-00-00-02 mobile_id = 100.3 access_date = 2012-04-01-00-00-03 mobile_id = 100.4 access_date = 2012-04-01-00-00-04 ... ... Devices Randomize access pattern. Workload randomised by hash key. Data model example: randomize access pattern by key. Towards a uniform workload. -
Hadoop under the hood. Take advantage of the Hadoop ecosystem:
streaming interfaces, Hive, Pig, Mahout. -
Query flexibility with Hive. create external table items_db (id string,
votes bigint, views bigint) stored by 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' tblproperties ("dynamodb.table.name" = "items", "dynamodb.column.mapping" = "id:id,votes:votes,views:views"); -
Query flexibility with Hive. select id, likes, views from items_db
order by views desc; -
Data export/import. CREATE EXTERNAL TABLE orders_s3_new_export ( order_id string, customer_id
string, order_date int, total double ) PARTITIONED BY (year string, month string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 's3://export_bucket'; INSERT OVERWRITE TABLE orders_s3_new_export PARTITION (year='2012', month='01') SELECT * from orders_ddb_2012_01; -
Integrate live and archive data Run queries across external Hive
tables on S3 and DynamoDB. Live & archive. Metadata & big objects. -
In summary... DynamoDB Data modeling Predictable performance Provisioned throughput Libraries
& mappers Tables & items Read & write patterns Time series data -
In summary... DynamoDB Data modeling Partitioning Predictable performance Provisioned throughput
Libraries & mappers Tables & items Read & write patterns Time series data Automatic partitioning Hot and cold data Size/throughput ratio -
In summary... DynamoDB Data modeling Partitioning Analytics Predictable performance Provisioned
throughput Libraries & mappers Tables & items Read & write patterns Time series data Automatic partitioning Hot and cold data Size/throughput ratio Elastic MapReduce Hive queries Backup & restore -
DynamoDB free tier 5 writes, 10 consistent reads per second
100Mb of storage