Paginating table query results in DynamoDB
DynamoDBpaginates the results fromQuery operations. With pagination, theQuery results are divided into "pages" of data that are 1 MB in size (or less). An application can process the first page of results, then the second page, and so on.
A singleQuery only returns a result set that fits within the 1 MB size limit. To determine whether there are more results, and to retrieve them one page at a time, applications should do the following:
Examine the low-level
Queryresult:If the result contains a
LastEvaluatedKeyelement and it's non-null, proceed to step 2.If there isnot a
LastEvaluatedKeyin the result, there are no more items to be retrieved.
Construct a
Querywith the sameKeyConditionExpression. However, this time, take theLastEvaluatedKeyvalue from step 1 and use it as theExclusiveStartKeyparameter in the newQueryrequest.Run the new
Queryrequest.Go to step 1.
In other words, theLastEvaluatedKey from aQuery response should be used as theExclusiveStartKey for the nextQuery request. If there is not aLastEvaluatedKey element in aQuery response, then you have retrieved the final page of results. IfLastEvaluatedKey is not empty, it does not necessarily mean that there is more data in the result set. The only way to know when you have reached the end of the result set is whenLastEvaluatedKey is empty.
You can use the AWS CLI to view this behavior. The AWS CLI sends low-levelQuery requests to DynamoDB repeatedly, untilLastEvaluatedKey is no longer present in the results. Consider the following AWS CLI example that retrieves movie titles from a particular year.
aws dynamodb query --table-name Movies \ --projection-expression "title" \ --key-condition-expression "#y = :yyyy" \ --expression-attribute-names '{"#y":"year"}' \ --expression-attribute-values '{":yyyy":{"N":"1993"}}' \ --page-size 5 \ --debugOrdinarily, the AWS CLI handles pagination automatically. However, in this example, the AWS CLI--page-size parameter limits the number of items per page. The--debug parameter prints low-level information about requests and responses.
If you run the example, the first response from DynamoDB looks similar to the following.
2017-07-07 11:13:15,603 - MainThread - botocore.parsers - DEBUG - Response body:b'{"Count":5,"Items":[{"title":{"S":"A Bronx Tale"}},{"title":{"S":"A Perfect World"}},{"title":{"S":"Addams Family Values"}},{"title":{"S":"Alive"}},{"title":{"S":"Benny & Joon"}}],"LastEvaluatedKey":{"year":{"N":"1993"},"title":{"S":"Benny & Joon"}},"ScannedCount":5}'TheLastEvaluatedKey in the response indicates that not all of the items have been retrieved. The AWS CLI then issues anotherQuery request to DynamoDB. This request and response pattern continues, until the final response.
2017-07-07 11:13:16,291 - MainThread - botocore.parsers - DEBUG - Response body:b'{"Count":1,"Items":[{"title":{"S":"What\'s Eating Gilbert Grape"}}],"ScannedCount":1}'The absence ofLastEvaluatedKey indicates that there are no more items to retrieve.
The AWS SDKs handle the low-level DynamoDB responses (including the presence or absence ofLastEvaluatedKey) and provide various abstractions for paginatingQuery results. For example, the SDK for Java document interface providesjava.util.Iterator support so that you can walk through the results one at a time.
For code examples in various programming languages, see theAmazon DynamoDB Getting Started Guide and the AWS SDK documentation for your language.
You can also reduce page size by limiting the number of items in the result set, with theLimit parameter of theQuery operation.
For more information about querying with DynamoDB, seeQuerying tables in DynamoDB.
