How to actually use webhooks, GetChanges and GetListItemChangesSinceToken?

Question

My main goal is a very simple duplicate of alerts, to play around with webhooks, just a simple

"User X changed field Title from 'Hello' to 'World'".

So far so good

After some struggle I finally managed to set up webhhoks (SharePoint List Subscription) to my Azure Functions using App Only Token (Azure AD) with adal-node and a function called acquireTokenWithClientCertificate.

Even though it was hard for me to figure out, the app only authentication seems very handy to have, and it seems the token will never expire.
The Subscription on the other hand will expire after a set date, but this date can't be older than 6 months, so I actually need a scheduled Azure Function just to re-new that subscription.
I run this on a weekly basis, I'm not going to expect some code in 6 months to "just work".

But then...

What I haven't been able to solve is actually fetching the item and knowing what has changed.
Surprisingly the webhook doesn't contain any interesting information. I can set some static querystring, but beside that I won't even get the item ID, all I get is "something changed". The webhook MSDN says "Your application is expected to use the GetChanges API...". So I have been playing around with the GetChanges API. To me it looks like a wrapper for the good ol' /_vti_bin/Lists.asmx, so much of a wrapper it actually returns XML (which I just parse with xml2js).

What I have so far

Anyway, I initially use GetChanges to get the newest ChangeToken, then on the next change I use that to get changes for GetListItemChangesSinceToken and this will actually give me the item with ows_FieldValue layout, which is the item in total plain text. But what should I compare this against? If I fetch the item using REST it will be the same updated item, I can't see the changes. It's also difficult to compare ows_SomeLookup against normal /item(${id}) without some logic.

I've been playing around some with GetListItemSinceLastToken as I can pass in a query, such as item ID. This works, but again I only get last change. I also pass in ViewFields since I have more than 8 lookup columns, so not setting it would fail.

So: How on an item change am I suppose to know what item fields have actually changed, and to what?

Update 2017-07-13

I have been looking at the new Graph API for SharePoint, and in theory I should be able to get items from there, and then run a deltaLink-query.
Except there are two problems:

1. The deltaLink doesn't actually show in results, only @odata.nextLink, so I suspect this api just a wrapper around the SharePoint REST API, so no deltaLink-query will be available.
2. I'm not able to get any values when calling Graph API with Azure AD token (had to create new token, added all Graph API permissions), only via the Graph explorer. I get the list, but not /items.

Note that this API just still just beta.

Update 2018-01-31

A discussion on Gitter, plus this commit draw the attention of an (at writing moment) undocumented feature; /versions.
It is now, amazingly, possible to get item versions using the REST API, solving this question, albeit not using GetChanges.

Note that this is only available on SharePoint Online, not on-premise.

Answer 1

I have created a solution. I'm not 100% sure if it is a good one, but it works. It only works on SharePoint Online. The demo code below is run using tick directly on a SharePoint site in Chrome. I also don't know what to pass as ChangeToken, but I figured I want Item, Update and Add. Code below requires sp-pnp-js version 3.0.4.

Getting the newest change token

I want to start looking for changes from "now", so I need the last change token. I didn't find any good way to do this, but a good ol' loop does the job:

import {sp} from 'sp-pnp-js';

const listUrl = '/Lists/Test';

(async () => {
  let changeToken;
  let run = true;
  while (run) {
    const items = await sp.web.getList(listUrl).getChanges({
      Item: true,
      Add: true,
      Update: true,
      ... changeToken ? {ChangeTokenStart: changeToken} : {}
    });

    run = items.length > 0;
    if (run) changeToken = items[items.length - 1].ChangeToken;
  }

  console.log(changeToken);
})();

Get changes using versions

Then I use versions on each changed (and added) item to get the difference. I only need the two latest versions (top(2)). In the example below I loop one and one item, and print the Title. What to actually do here is up to you.

import {sp} from 'sp-pnp-js';

const changeToken = {StringValue: "1;3;0b8afcc0-2184-4821-9e1c-e7d68f85c25e;636535159219700000;119831891"};
const listUrl = '/Lists/Test';

(async () => {
  console.clear();

  const list = sp.web.getList(listUrl);

  const items = await list.getChanges({
    Item: true,
    Add: true,
    Update: true,
    ChangeTokenStart: changeToken
  });

  for (let item of items) {
    const versions = await list.items.getById(item.ItemId).versions.top(2).get();
    console.log(versions[0].Title, versions[1].Title);
  }
})();

It's also important to store the change token from the code above (last item, as in first example) for the next run.

Answer 2

Here's a reference implementation I stumbled across in C# - similar patterns.

https://github.com/SharePoint/sp-dev-samples/blob/master/Samples/WebHooks.List/SharePoint.WebHooks.Common/ChangeManager.cs