Java – Paging based on numeric limits and offsets in DynamoDB (Java).

Paging based on numeric limits and offsets in DynamoDB (Java)…. here is a solution to the problem.

Paging based on numeric limits and offsets in DynamoDB (Java).

I want to implement paging based on numeric limits and offsets in DynamoDB similar to Postgres

My API looks like this: http://foo.bar/user?offset=50&limit=20.

Consider that DynamoDB uses ExclusiveStartKey and LastEvaluatedKey to paginate, what is the best way to do this in Java without the risk of OutOfMemoryError?

Edit:

Let’s assume that offset-based paging is a hard requirement, and I don’t know about “previous” pages. My API contract has offset and limit query parameters as described above. I’m not looking for an answer to “don’t do offset-based pagination”.

Solution

Looked at how PaginatedList work in the SDK, it seems that the most efficient way to do this is to use the ITERATION_ONLY paging policy in your configuration and do the following:

DynamoDBMapperConfig config = new DynamoDBMapperConfig.Builder()
                       .withPaginationLoadingStrategy(ITERATION_ONLY)
                       .build();
PaginatedQueryList<MyUser> users = dynamoDBMapper.query(MyUser.class, myQueryExpression, config);
 This iterator will fetch 1MB worth of data 'on-demand'
Iterator<MyUser> userIterator = users.iterator();
skip(iterator, offset);
List<MyUser> aPageOfUser = collect(iterator, limit);

Of course, I’ll bear the cost of implementing my own skip and collect methods there. Is there a better/more concise way to do this?

Edit:

It seems that I can take advantage of IteratorUtils from commons-collections free skips and collections:

Iterator<MyUser> userIterator = IteratorUtils.boundedIterator(users.iterator(), offset, limit);
List<MyUser> aPageOfUser = IteratorUtils.toList(userIterator);

Related Problems and Solutions