It's not just a pointer to the file, the crawled content are stored as records within dedicated database(s) and contains key parts of the item (all the crawled properties) allowing it to be discovered through a search query. These crawled properties could lead to security breach if the permissions on the site / documents where not set properly (or if the security on the database itself is too weak, allowing someone to have a sneak peek inside).
Office:nn crawled properties are holding precious information that could be inadapted for someone to have access (even if it's just within a search result details and no real access to the secured item itself)
So if you have critical documents, the biggest concerns are how do you handle security on these libraries and their site ? What are you farm accounts ? Under which identity SharePoint is crawling your content ? Are you sure only relevant people have access to the list ? SharePoint will keep track of the security on each documents but that doesn't fix human misconfiguration (I've seen crawled account that are similar to app pool account, run a search query with elevated privilege by code for anyone and one can reach all documents...)
Oh, and don't forget that you can prevent crawling per library in extreme case :) I recently discovered military graded documents while searching for public SharePoint 2010/2013 sites using mobile view (view form page lockdown not activated and definitively huge issues regarding permissions on the libraries / network topoly / setup). I would not be the architect behind that site...
Edit : if you're the administrator / architect / lead developer / ... responsible for this, I would heavily recommend to carefully read this book from APress : http://www.apress.com/9781430234074. There are a bunch of book covering the subject but this is by far the best I purchased. Make sure that you read official ressources / guideline from a trusted authority (= not me!) to avoid any unforeseen results.