Building Applications with DynamoDB

Matt Wood

May 16, 2012

Tweet Share

Transcript

1. DynamoDB Building Applications with An Online Seminar - 16th May

   2012 Dr Matt Wood, Amazon Web Services

2. Thank you!

3. Building Applications with DynamoDB

4. Building Applications with DynamoDB Getting started

5. Building Applications with DynamoDB Getting started Data modeling

6. Building Applications with DynamoDB Getting started Data modeling Partitioning

7. Building Applications with DynamoDB Getting started Data modeling Partitioning Analytics

8. Getting started with DynamoDB quick review

9. DynamoDB is a managed NoSQL database service. Store and retrieve

   any amount of data. Serve any level of request traffic.

10. Without the operational burden.

11. Consistent, predictable performance. Single digit millisecond latencies. Backed on solid-state

    drives.

12. Flexible data model. Key/attribute pairs. No schema required. Easy to

    create. Easy to adjust.

13. Seamless scalability. No table size limits. Unlimited storage. No downtime.

14. Durable. Consistent, disk-only writes. Replication across data centres and availability

    zones.

15. Without the operational burden.

16. Without the operational burden. FOCUS ON YOUR APP

17. Two decisions + three clicks = ready for use

18. Two decisions + three clicks = ready for use Primary

    keys + level of throughput

19. Provisioned throughput. Reserve IOPS for reads and writes. Scale up

    (or down) at any time.

20. Pay per capacity unit. Priced per hour of provisioned throughput.

21. Write throughput. $0.01 per hour for 10 write units Units

    = size of item x writes/second

22. Consistent writes. Atomic increment/decrement. Optimistic concurrency control. aka: “conditional writes”.

23. Transactions. Item level transactions only. Puts, updates and deletes are

    ACID.

24. Read throughput. strongly consistent eventually consistent

25. Read throughput. $0.01 per hour for 50 read units Provisioned

    units = size of item x reads/second strongly consistent eventually consistent

26. Read throughput. $0.01 per hour for 100 read units Provisioned

    units = size of item x reads/second 2 strongly consistent eventually consistent

27. Read throughput. Mix and match at “read time”. Same latency

    expectations. strongly consistent eventually consistent

28. Two decisions + three clicks = ready for use

29. None
30. None
31. None

32. Two decisions + three clicks = ready for use

33. Two decisions + one API call = ready for use

34. $create_response = $dynamodb->create_table(array( 'TableName' => 'ProductCatalog', 'KeySchema' => array( 'HashKeyElement'

    => array( 'AttributeName' => 'Id', 'AttributeType' => AmazonDynamoDB::TYPE_NUMBER ) ), 'ProvisionedThroughput' => array( 'ReadCapacityUnits' => 10, 'WriteCapacityUnits' => 5 ) ));

35. Two decisions + one API call = ready for use

36. Two decisions + one API call = ready for development

37. Two decisions + one API call = ready for production

38. Two decisions + one API call = ready for scale

39. None

40. Authentication. Session based to minimize latency. Uses Amazon Security Token

    Service. Handled by AWS SDKs. Integrates with IAM.

41. Monitoring. CloudWatch metrics: latency, consumed read and write throughput, errors

    and throttling.

42. Libraries, mappers & mocks. http://j.mp/dynamodb-libs ColdFusion, Django, Erlang, Java, .Net,

    Node.js, Perl, PHP, Python, Ruby

43. DynamoDB data models

44. DynamoDB semantics. Tables, items and attributes.

45. Tables contain items. Unlimited items per table.

46. Items are a collection of attributes. Each attribute has a

    key and a value. An item can have any number of attributes, up to 64k total.

47. Two scalar data types. String: Unicode, UTF8 binary encoding. Number:

    38 digit precision. Multi-value strings and numbers.

48. id = 100 date = 2012-05-16-09-00-10 total = 25.00 id

    = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00

49. id = 100 date = 2012-05-16-09-00-10 total = 25.00 id

    = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Table

50. id = 100 date = 2012-05-16-09-00-10 total = 25.00 id

    = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Item

51. id = 100 date = 2012-05-16-09-00-10 total = 25.00 id

    = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Attribute

52. Where is the schema? Tables do not require a formal

    schema. Items are an arbitrary sized hash. Just need to specify the primary key.

53. Items are indexed by primary key. Single hash keys and

    composite keys.

54. id = 100 date = 2012-05-16-09-00-10 total = 25.00 id

    = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Hash Key

55. Range key for queries. Querying items by composite key.

56. id = 100 date = 2012-05-16-09-00-10 total = 25.00 id

    = 101 date = 2012-05-15-15-00-11 total = 35.00 id = 101 date = 2012-05-16-12-00-10 total = 100.00 id = 102 date = 2012-03-20-18-23-10 total = 20.00 id = 102 date = 2012-03-20-18-23-10 total = 120.00 Hash Key Range Key +

57. Programming DynamoDB. Small but perfectly formed. Whole programming interface fits

    on one slide.

58. CreateTable UpdateTable DeleteTable DescribeTable ListTables PutItem GetItem UpdateItem DeleteItem BatchGetItem

    BatchWriteItem Query Scan

59. CreateTable UpdateTable DeleteTable DescribeTable ListTables PutItem GetItem UpdateItem DeleteItem BatchGetItem

    BatchWriteItem Query Scan

60. CreateTable UpdateTable DeleteTable DescribeTable ListTables PutItem GetItem UpdateItem DeleteItem BatchGetItem

    BatchWriteItem Query Scan

61. Conditional updates. PutItem, UpdateItem, DeleteItem can take optional conditions for

    operation. UpdateItem performs atomic increments.

62. CreateTable UpdateTable DeleteTable DescribeTable ListTables PutItem GetItem UpdateItem DeleteItem BatchGetItem

    BatchWriteItem Query Scan

63. One API call, multiple items. BatchGet returns multiple items by

    primary key. BatchWrite performs up to 25 put or delete operations. Throughput is measured by IO, not API calls.

64. CreateTable UpdateTable DeleteTable DescribeTable ListTables PutItem GetItem UpdateItem DeleteItem BatchGetItem

    BatchWriteItem Query Scan

65. Query vs Scan Query for composite key queries. Scan for

    full table scans, exports. Both support pages and limits. Maximum response is 1Mb in size.

66. Query patterns. Retrieve all items by hash key. Range key

    conditions: ==, <, >, >=, <=, begins with, between. Counts. Top and bottom n values. Paged responses.

67. Modeling patterns

68. 1. Mapping relationships with range keys. No cross-table joins in

    DynamoDB. Use composite keys to model relationships. Patterns

69. Data model example: online gaming. Storing scores and leader boards.

    Players with high Scores. Leader board for each game.

70. Data model example: online gaming. Storing scores and leader boards.

    Players with high Scores. Leader board for each game. user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key

71. Data model example: online gaming. Storing scores and leader boards.

    Players with high Scores. Leader board for each game. user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key

72. Data model example: online gaming. Storing scores and leader boards.

    Players with high Scores. Leader board for each game. user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards: composite key

73. Data model example: online gaming. Storing scores and leader boards.

    user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards: composite key Scores by user (and by game)

74. Data model example: online gaming. Storing scores and leader boards.

    user_id = mza location = Cambridge joined = 2011-07-04 user_id = jeffbarr location = Seattle joined = 2012-01-20 user_id = werner location = Worldwide joined = 2011-05-15 Players: hash key user_id = mza game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = werner location = bejewelled score = 55,000 Scores: composite key game = angry-birds score = 11,000 user_id = mza game = tetris score = 1,223,000 user_id = mza game = tetris score = 9,000,000 user_id = jeffbarr Leader boards: composite key High scores by game

75. 2. Handling large items. Unlimited attributes per item. Unlimited items

    per table. Max 64k per item. Patterns

76. Data model example: large items. Storing more than 64k across

    items. message_id = 1 part = 1 message = <first 64k> message_id = 1 part = 2 message = <second 64k> message_id = 1 part = 3 joined = <third 64k> Large messages: composite keys Split attributes across items. Query by message_id and part to retrieve.

77. Store a pointer to objects in Amazon S3. Large data

    stored in S3. Location stored in DynamoDB. 99.999999999% data durability in S3. Patterns

78. 3. Managing secondary indices. Not supported by DynamoDB. Create your

    own. Patterns

79. Data model example: secondary indices. Storing more than 64k across

    items. user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Users: hash key

80. Data model example: secondary indices. Storing more than 64k across

    items. user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys

81. Data model example: secondary indices. Storing more than 64k across

    items. Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys Second name index: composite keys last_name = Wood user_id = mza last_name = Fox user_id = mattfox last_name = Vogels user_id = werner user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels

82. last_name = Wood user_id = mza last_name = Fox user_id

    = mattfox last_name = Vogels user_id = werner user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Data model example: secondary indices. Storing more than 64k across items. Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys Second name index: composite keys

83. last_name = Wood user_id = mza last_name = Fox user_id

    = mattfox last_name = Vogels user_id = werner user_id = mza first_name = Matt last_name = Wood user_id = mattfox first_name = Matt last_name = Fox user_id = werner first_name = Werner last_name = Vogels Data model example: secondary indices. Storing more than 64k across items. Users: hash key first_name = Matt user_id = mza first_name = Matt user_id = mattfox first_name = Werner user_id = werner First name index: composite keys Second name index: composite keys

84. 4. Time series data. Logging, click through, ad views, game

    play data, application usage. Non-uniform access patterns. Newer data is ‘live’. Older data is read only. Patterns

85. Data model example: time series data. Rolling tables for hot

    and cold data. event_id = 1000 timestamp = 2012-05-16-09-59-01 key = value event_id = 1001 timestamp = 2012-05-16-09-59-02 key = value event_id = 1002 timestamp = 2012-05-16-09-59-02 key = value Events table: composite keys

86. Data model example: time series data. Rolling tables for hot

    and cold data. event_id = 1000 timestamp = 2012-05-16-09-59-01 key = value event_id = 1001 timestamp = 2012-05-16-09-59-02 key = value event_id = 1002 timestamp = 2012-05-16-09-59-02 key = value Events table: composite keys Events table for April: composite keys Events table for January: composite keys event_id = 400 timestamp = 2012-04-01-00-00-01 event_id = 401 timestamp = 2012-04-01-00-00-02 event_id = 402 timestamp = 2012-04-01-00-00-03 event_id = 100 timestamp = 2012-01-01-00-00-01 event_id = 101 timestamp = 2012-01-01-00-00-02 event_id = 102 timestamp = 2012-01-01-00-00-03

87. Hot and cold tables. Jan April May Feb Mar Dec

    Patterns

88. Hot and cold tables. Jan April May Feb Mar higher

    throughput Dec Patterns

89. Hot and cold tables. Jan April May Feb Mar higher

    throughput lower throughput Dec Patterns

90. Hot and cold tables. Jan April May Feb Mar data

    to S3, delete cold tables Dec Patterns

91. Hot and cold tables. Feb May June Mar Apr Jan

    Patterns

92. Hot and cold tables. Mar June July Apr May Feb

    Patterns

93. Hot and cold tables. Apr July Aug May June Mar

    Patterns

94. Hot and cold tables. May Aug Sept June July Apr

    Patterns

95. Hot and cold tables. June Sept Oct July Aug May

    Patterns

96. Not out of mind. DynamoDB and S3 data can be

    integrated for analytics. Run queries across hot and cold data with Elastic MapReduce. Patterns

97. Partitioning best practices

98. Uniform workloads. DynamoDB divides table data into multiple partitions. Data

    is distributed primarily by hash key. Provisioned throughput is divided evenly across the partitions.

99. Uniform workloads. To achieve and maintain full provisioned throughput for

    a table, spread your workload evenly across the hash keys.

100. Non-uniform workloads. Some requests might be throttled, even at high

     levels of provisioned throughput. Some best practices...

101. 1. Distinct values for hash keys. Patterns Hash key elements

     should have a high number of distinct values.

102. Data model example: hash key selection. Well distributed work loads

     user_id = mza first_name = Matt last_name = Wood user_id = jeffbarr first_name = Jeff last_name = Barr user_id = werner first_name = Werner last_name = Vogels user_id = mattfox first_name = Matt last_name = Fox ... ... ... Users

103. Data model example: hash key selection. Well distributed work loads

     user_id = mza first_name = Matt last_name = Wood user_id = jeffbarr first_name = Jeff last_name = Barr user_id = werner first_name = Werner last_name = Vogels user_id = mattfox first_name = Matt last_name = Fox ... ... ... Users Lots of users with unique user_id. Workload well distributed across user partitions.

104. 2. Avoid limited hash key values. Patterns Hash key elements

     should have a high number of distinct values.

105. Data model example: small hash value range. Non-uniform workload. status

     = 200 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 status 404 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 Status responses

106. Data model example: small hash value range. Non-uniform workload. status

     = 200 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 status 404 date = 2012-04-01-00-00-01 status = 404 date = 2012-04-01-00-00-01 Status responses Small number of status codes. Unevenly, non-uniform workload.

107. 3. Model for even distribution of access. Patterns Access by

     hash key value should be evenly distributed across the dataset.

108. Data model example: uneven access pattern by key. Non-uniform access

     workload. mobile_id = 100 access_date = 2012-04-01-00-00-01 mobile_id = 100 access_date = 2012-04-01-00-00-02 mobile_id = 100 access_date = 2012-04-01-00-00-03 mobile_id = 100 access_date = 2012-04-01-00-00-04 ... ... Devices

109. mobile_id = 100 access_date = 2012-04-01-00-00-01 mobile_id = 100 access_date

     = 2012-04-01-00-00-02 mobile_id = 100 access_date = 2012-04-01-00-00-03 mobile_id = 100 access_date = 2012-04-01-00-00-04 ... ... Devices Large number of devices. Small number which are much more popular than others. Workload unevenly distributed. Data model example: uneven access pattern by key. Non-uniform access workload.

110. mobile_id = 100.1 access_date = 2012-04-01-00-00-01 mobile_id = 100.2 access_date

     = 2012-04-01-00-00-02 mobile_id = 100.3 access_date = 2012-04-01-00-00-03 mobile_id = 100.4 access_date = 2012-04-01-00-00-04 ... ... Devices Randomize access pattern. Workload randomised by hash key. Data model example: randomize access pattern by key. Towards a uniform workload.

111. Design for a uniform workload.

112. Analytics with DynamoDB

113. Seamless scale. Scalable methods for data processing. Scalable methods for

     backup/restore.

114. Amazon Elastic MapReduce. http://aws.amazon.com/emr Managed Hadoop service for data-intensive workflows.

115. Hadoop under the hood. Take advantage of the Hadoop ecosystem:

     streaming interfaces, Hive, Pig, Mahout.

116. Distributed data processing. API driven. Analytics at any scale.

117. Query flexibility with Hive. create external table items_db (id string,

     votes bigint, views bigint) stored by 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' tblproperties ("dynamodb.table.name" = "items", "dynamodb.column.mapping" = "id:id,votes:votes,views:views");

118. Query flexibility with Hive. select id, likes, views from items_db

     order by views desc;

119. Data export/import. Use EMR for backup and restore to Amazon

     S3.

120. Data export/import. CREATE EXTERNAL TABLE orders_s3_new_export ( order_id string, customer_id

     string, order_date int, total double ) PARTITIONED BY (year string, month string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LOCATION 's3://export_bucket'; INSERT OVERWRITE TABLE orders_s3_new_export PARTITION (year='2012', month='01') SELECT * from orders_ddb_2012_01;

121. Integrate live and archive data Run queries across external Hive

     tables on S3 and DynamoDB. Live & archive. Metadata & big objects.

122. In summary... DynamoDB Predictable performance Provisioned throughput Libraries & mappers

123. In summary... DynamoDB Data modeling Predictable performance Provisioned throughput Libraries

     & mappers Tables & items Read & write patterns Time series data

124. In summary... DynamoDB Data modeling Partitioning Predictable performance Provisioned throughput

     Libraries & mappers Tables & items Read & write patterns Time series data Automatic partitioning Hot and cold data Size/throughput ratio

125. In summary... DynamoDB Data modeling Partitioning Analytics Predictable performance Provisioned

     throughput Libraries & mappers Tables & items Read & write patterns Time series data Automatic partitioning Hot and cold data Size/throughput ratio Elastic MapReduce Hive queries Backup & restore

126. DynamoDB free tier 5 writes, 10 consistent reads per second

     100Mb of storage

127. aws.amazon.com/dynamodb aws.amazon.com/documentation/dynamodb best practice + sample code

128. Thank you!

129. Q & A[email protected] @mza