Slow CSV writing for huge document libraries in powershell

Question

by using below PowerShell script, i get all the items list from the document library including sub folders hierarchy in a CSV file. This script works very fine for small data but for huge data like 5GB , takes lot of time..it took me around 4 hours and still not completed. why it is very slow for huge data with big folder hierarchy? are there any other way to achieve the task?

$exportlist = @()
    $list.Items | foreach {
    $obj = New-Object PSObject -Property @{
    "Title" = $_["Title"]
    "Name" = $_["Name"]
    "Modified Date" = $_["Modified"]
    "Modified By" =$_["Modified By"]
    "Size"= $_["File Size"]
    "Path" = $web.Url + "/" + $_.File.Url

    }
    $exportlist += $obj
    $exportlist | Export-Csv -path 'C:\Test\MyList.csv' -noType
    }

Answer 1

Perhaps consider processing it in batches, instead of trying to process all items:

http://msdn.microsoft.com/en-us/library/office/ee557257%28v=office.14%29.aspx

Answer 2

You can also setup a basic SSRS report to list all the columns you want, then set a subscription on it to export it as a CSV file (to a doc library or network share) on a scheduled basis.

Iterating by loop of any object as per your script will always get slower with the increasing volume of objects.

Answer 3

I achieved it very easily . I did a view and exported it as excel. that gives required metadata as well. only path to the file was missing. so I used VBA sub to separate Hyperlink and used it in a separate column