You're hitting the message size limit that SharePoint Online imposes on requests, which is why you're seeing the error The request message is too big. The server does not allow messages larger than 2097152 bytes. In simple terms, when you have a large list with many fields (like your 77-field list), the data being transferred is too big for SharePoint to handle in one go.
Why is this happening?
SharePoint Online has a limit of 2 MB (or 2,097,152 bytes) for the size of requests made to the server. When you're working with a list that has a large number of fields, even if there are no attachments or large data items, the sheer amount of metadata and field values being transferred can exceed this limit.
How to fix it:
Batch the Requests: Instead of copying all list items in one go, break the process down into smaller chunks. Copy fewer fields or items at a time. You can loop through the list and copy items in smaller batches to avoid hitting the size limit.
Example:
Process the list in sets of 10–20 items at a time.
Reduce Fields in One Request: If your script is pulling all 77 fields at once, try reducing the number of fields you're querying for in each request. Only query for the fields that you actually need, which can help reduce the size of the request.
Use Paging: SharePoint supports paging, which allows you to retrieve items in batches rather than pulling the entire list at once. By limiting the number of items retrieved per query, you can prevent exceeding the size limit.
PowerShell Example Using Batching:
Here’s a simple example of how you can batch your list item copying:
# Assuming you've already connected to SharePoint and set up your context
$batchSize = 20
$listItems = $sourceList.GetItems([Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery())
$ctx.Load($listItems)
$ctx.ExecuteQuery()
# Process items in batches
for ($i = 0; $i -lt $listItems.Count; $i += $batchSize) {
$itemsBatch = $listItems | Select-Object -Skip $i -First $batchSize
foreach ($item in $itemsBatch) {
# Your copy logic here for each item
# For example:
# Copy item fields to new list
}
$ctx.ExecuteQuery() # Make sure to execute the query for each batch
}
This approach prevents you from exceeding the request size by copying a limited number of items at a time.
Consider Using SharePoint REST API: If the PowerShell approach with the ExecuteQuery method continues to give you issues, you might consider using SharePoint’s REST API, which can offer more flexibility and control for large data sets. However, you'd still need to handle pagination and chunking.
References
To get more details about SharePoint Online request limits and how to work with large lists, you can refer to Microsoft's official documentation:
Give this a try by batching or reducing the number of fields, and let me know if you need more help!
