ListItemChangesSinceToken doesn't page correctly?

Question

I'm trying to page through getListItemChangesSinceToken to get an entire list and then any changes to it but I'm getting very inconsistent results.

This is what I do to get the initial list of changes:

1. Call getListItemChangesSinceToken (using the PNP JS library) with a rowlimit and no ChangeToken or Paging ListItemCollectionPositionNext.
2. Get some results, the next ChangeToken and ListItemCollectionPositionNext
3. repeat from 1 with the received Paging ListItemCollectionPositionNext until there's no ListItemCollectionPositionNext in the results
4. Store the ChangeToken from the very first results for calling next time.

Doing it this way on a test collection with 42 folders and 10 files gives me 14 folders+0 files when I use row limit 3 but 17 folders when I use row limit 8, and 16 folders with low limit 5. The paging goes through several iterations but skips a whole bunch of data it seems.

Furthermore, the paged results have their timestamps changed from UTC ISO format to local time space-separated format.

Here are two examples of the same folder result (after XML to JSON conversion), from the initial result and when it's part of a paged result. Timestamps and user references seem to change.

initial:

"ows_Created": "2019-12-07T13:07:28Z",
"ows_Author": "35;#Wout Mertens,#i:0#.f|membership|[email protected],#[email protected],#[email protected],#Wout Mertens",

paged:

"ows_Created": "2019-12-07 05:07:28",
"ows_Author": "35;#Wout Mertens",

Is this API just broken? Am I using it wrongly?

Answer 1

Answer: getlistItemChangesSinceToken is just a terrible API. It misses some types of changes and is hard to parse.

Furthermore, it looks like the changelog can't be queried forever, it only seems to keep "recent" changes.

So the best way to get changes is to use ... getChanges. To use a token, pass ChangeTokenStart: {StringValue: token} to the SP.ChangeQuery. You can get the initial token from the list, it's on CurrentChangeToken.StringValue.

You can also pass {FetchLimit: "100"} (yes, as a string) to get changes 100 at a time;

Read the next token from the last item. If there are no results, you're done for now.

So with getChanges you get a sane way to monitor changes and you can page through the results.