Paging based on numeric limits and offsets in DynamoDB (Java).
I want to implement paging based on numeric limits and offsets in DynamoDB similar to Postgres
My API looks like this: http://foo.bar/user?offset=50&limit=20
.
Consider that DynamoDB uses ExclusiveStartKey
and LastEvaluatedKey
to paginate, what is the best way to do this in Java without the risk of OutOfMemoryError
?
Edit:
Let’s assume that offset-based paging is a hard requirement, and I don’t know about “previous” pages. My API contract has offset
and limit
query parameters as described above. I’m not looking for an answer to “don’t do offset-based pagination”.
Solution
Looked at how PaginatedList work in the SDK, it seems that the most efficient way to do this is to use the ITERATION_ONLY
paging policy in your configuration and do the following:
DynamoDBMapperConfig config = new DynamoDBMapperConfig.Builder()
.withPaginationLoadingStrategy(ITERATION_ONLY)
.build();
PaginatedQueryList<MyUser> users = dynamoDBMapper.query(MyUser.class, myQueryExpression, config);
This iterator will fetch 1MB worth of data 'on-demand'
Iterator<MyUser> userIterator = users.iterator();
skip(iterator, offset);
List<MyUser> aPageOfUser = collect(iterator, limit);
Of course, I’ll bear the cost of implementing my own skip
and collect
methods there. Is there a better/more concise way to do this?
Edit:
It seems that I can take advantage of IteratorUtils from commons-collections free skips
and collections
:
Iterator<MyUser> userIterator = IteratorUtils.boundedIterator(users.iterator(), offset, limit);
List<MyUser> aPageOfUser = IteratorUtils.toList(userIterator);