Storing large files in SharePoint with users all over the world

Question

Any recommendations for storing large files in SharePoint 2010 for the following requirements:

1. Mostly smaller files 10KB-20MB
2. Many larger files for example 20-500MB
3. Users in all parts of world need fast download
4. Intranet Security

Especially interested in recommendations regarding the global aspect.

Any third party components or Microsoft Products?

Answer 1

We have a web application dedicated for serving up files and this is what I can tell you from my experience. The design of the web app is to house executables, license codes, and installation guides, so pretty similar to your requirements. The executables can range from 4MB to 1.5 GB with guides in the 200k range.

Under normal load, there aren't too many problems. Wait time for fetching the larger files out of the database can take upto 3 minutes for local users. Recently, we've been moving files out of SharePoint and putting them on a web server restricted by AD permissions and use the Link To Document content types to maask the file location.

This allows us to use the interface and security of SharePoint to give users access to the appropriate files and gives the users instant access to the file instead of waiting for it to be sucked out of the database and presented to the user.

This is definately not what Microsoft intended for the use of SharePoint. It has worked well to distribute some of the smaller executables and you might be safe if the max size you're anticipating is 500 MB. How it will scale out to a global deployment, I cannot tell you.

I should also note that we tested BLOB offloading of these files with no noticible impact on the speed of delivery.

Answer 2

We have few offices around the World and SP is not the only system here, because sometimes it is not good enought for our needs.

We don't want to store large files on SharePoint at all. Users can upload 50 MB per one file, but it depends. HR and TT departments working with video files, presentations or trainings and we created streaming server for them. Big marketing files are stored at shared network place with automatic cleaning and report files or data cubes are stored in external systems. SharePoint is central place for all links to big files (from pages or libraries, it doesen' matter) and honestly, most users don't recognize the difference.

Maybe it is not the best practice, but we don't want to use SharePoint as file share only.

Answer 3

There really is very little you can do to overcome the challenges of a global SharePoint deployment when it comes to the handling of large files. Normally we'd talk about a CDN or some other mechanism of using ones geographic context to establish where content should come from (i.e. it would be beneficial to have a user in the U.K. fetch a document from a server in Germany than it would be to grab it from Dallas, Texas), but that type of capability is not easily acheived with SharePoint given the centralized nature of any deployment.

It's not to say that you can't de-centralize web front-ends (...you can) but then you have to ensure that the traffic between the WFEs and a centralized database server does not create a performance bottle-neck of it's own. Simply remoting the BLOBs does nothing to overcome this.

You could also look at creating separate farms in different geo locations and then replicate between them, but that typically introduces a level fo complexity and cost that is challenging for most organizations.

It's a very tough, arguably impossible, problem to solve...and I say impossible because the vast majority of companies wouldn't spend the kind of money you would need to spend to overcome it.

Answer 4

You wont be able to guarantee or even control performance unless you own all the connections. It is possibly to increase your chances of success by adding WAN optimizer at your locations, such as Riverbed devices. You may also be able to achieve similar benefit at less cost if you utilize proxies at remote locations that can pre-fetch your large object and store them locally (WAN optimizer usually employ some type of similar caching functionality). This is a network scoped solution, deployed outside of a SharePoint context.

Answer 5

It is possible to build a control that uses a BLOB CDN in the backend and serve up within SharePoint. My company has built a 3rd party solution to this issue. Feel free to reach out to me at brandong@attunix.com if you have not found a solution yet.