Assumptions

• Firestore is used on your API server
• We want to use cursor-based pagination instead of Offset-based pagination in the API server
  • The client sends the cursor string to the API server, which then retrieves the continuation and returns it

What I want to do

• I want to modularize the pagination process using Firestore as a data source in the API server.
• The module requires cursor, limit, and Firestore of Query likefirestore().collection('posts').where(...).orderBy(...)
• At a minimum, the followings will be returned as a result
  • An array of the documents you retrieved
  • Whether there is next data (hasNextPage)
  • The cursor of the last document

Difficulties

ThestartAfter/startAt used for pagination in Firestore can be specified by "the value of the field specified byorderBy" or "a snapshot of the document".

In the former case, the type of the cursor to be passed depends on what orderBy is specified. In TypeScript, it can bestring, number, firestore.Timestamp, etc., but it is more practical to usestring for returning the cursor to the API client (the cursor for pagination in Relay in GraphQL is a string). However, in order to make pagination processing common, it is complicated to say "the type of the cursor in this query is XX, so it must be converted like this".

In the latter case (document snapshot), the snapshot is an object and is not suitable to be converted directly into a cursor (string). So, I came up with the idea to convert the path of the document into a cursor.

How to do it

The following is an example of TypeScript code (roughly equivalent to GraphQL's Relay style cursor pagination).

import{firestore}from'firebase-admin'// base64 encode the snapshot's pathconstencodeCursor=(snapshot:firestore.DocumentSnapshot|firestore.QueryDocumentSnapshot)=>{returnBuffer.from(snapshot.ref.path).toString('base64')}constdecodeCursor=(cursor:string)=>{returnBuffer.from(cursor,'base64').toString('utf8')}typeConnection={nodes:{id:string}[].pageInfo:{hasNextPage:booleanendCursor?:string|null}}exportconstpaginateFirestore=async(query:firestore.Query,limit:number,cursor?:string|null):Promise<Connection>=>{// get one more item for hasNextPageletq=query.limit(limit+1)if(cursor){// If a cursor is passed, convert it to a path and get a snapshot of the documentconstpath=decodeCursor(cursor)constsnap=awaitadmin.firestore().doc(path).get()if(!snap.exists){return{nodes:[],pageInfo:{hasNextPage:false}}// pass to startAfterq=q.startAfter(snap)}constsnapshot=awaitq.get()consthasNextPage=snapshot.size>limitconstdocs=snapshot.docs.slice(0,limit)// make the path of the last document a cursorconstendCursor=hasNextPage?encodeCursor(docs[docs.length-1]):nullreturn{nodes:docs.map(doc=>({id:doc.id,..doc.data()})),pageInfo:{hasNextPage,endCursor,},}}

The usage is as follows.

constquery=firestore().collection('posts').orderBy('createdAt','desc')constconnection=awaitpaginateFirestore(query,100,args.cursor)

`

Advantages and disadvantages.

Advantages

• Simplifies the process because the cursor will always be path.
• Clients only needs to pass the generated cursor.
• Pagination by snapshot is more accurate than pagination by orderBy field.

Disadvantages

• Overhead from getting one extra item.

What do you think?

I thought that the overhead of retrieving one extra item would not have a big impact on gRPC, so I implemented it this way, prioritizing the simplicity of the code. If you have any thoughts on this, I'd love to hear your feedback!