Read large lists with Office365-REST-Python-Client

Question

I am currently trying to read only the last week's entries of a few lists, and I accomplish this by using filter("Modified gt datetime{date_in_isoformat}') or order_by("Modified desc").top(50) along with get(); but now one of my lists has crossed the 5000 entries threshold and I get a 500 Error because of it, if I use paged(5000) or get_all(5000) it doesn't seem to work with filter() or order_by() and top().

Is this a limitation? Do I need to chain these in a specific order?

Thank you for any insight

Answer 1

When working with large lists that exceed the 5000-item threshold in SharePoint, you may encounter limitations in retrieving items using the filter, orderby, and top methods. This is because these methods operate on the client side and rely on retrieving all items before applying the filtering, ordering, and limiting.

To overcome this limitation, I would consider following approaches to try:

• Paging and Sorting: If you need to retrieve items using the REST API only, you can implement paging and sorting techniques. You can use the startRow parameter to specify the starting row index, and orderby to sort the items before applying the filter. By fetching items in smaller batches and combining the results, you can overcome the 5000 item limitation.
• Indexed Columns: You can create an index on the column(s) you frequently use for filtering, such as the Modified column in your case. Indexing allows SharePoint to efficiently retrieve items based on indexed columns, even when the list exceeds the 5000 item threshold.