firstly you need office web apps installed on the server, once you have that up and running follow this msdn guid! you have several option!
http://technet.microsoft.com/en-us/library/ee837425(v=office.14).aspx
How documents open in SharePoint varies depending on whether the
OpenInClient feature is present, and either enabled or disabled:
•If the OpenInClient feature is not present and Office Web Apps is not
installed, documents will open in the client application (SharePoint
default).
•If the OpenInClient feature is not present, Office Web Apps is
installed and Office Web Apps service applications are activated,
documents will open in the browser (Office Web Apps default).
•If the OpenInClient Feature is present and enabled, and Office Web
Apps service applications are activated, documents will open in the
client application.
•If the OpenInClient Feature is present and disabled, and Office Web
Apps service applications are activated, documents in will open in the
browser.
now for what you want, you need to disable this function:
To set the default open behavior for site collections
1.In the SharePoint site collection, click Site Actions, and then click Site Settings.
2.On the Site Settings page, under Site Collection Administration, click Site Collection Features.
3.On the Features page, for the Open Documents in Client Applications by Default feature, click Activate (OpenInClient Feature is enabled)
to open documents in the client application. Click Deactivate
(OpenInClient Feature is disabled) to open documents in the browser.
or using powershell script
$defaultOpenBehaviorFeatureId = $(Get-SPFeature -limit all | where {$_.displayname -eq "OpenInClient"}).Id
Get-SPSite -limit ALL |foreach{ Disable-SPFeature $defaultOpenBehaviorFeatureId -url $_.URL }
for full guid follow the link at the top!
or just for a specific document library:
1.On the : All Documents page, under Library Tools, click Library.
2.In the library toolbar, click Library Settings.
3.On the Document Library Settings page, click Advanced Settings.
4.On the Advanced Settings page, in Opening Document in the Browser, select one of the following options:
Open in the client application When a user clicks on a document in
this library, the document will open in the corresponding client
application (if available).
Open in the browser When a user clicks on a document in this
library, the document will open in the Web browser Web app for that
document type. When the document is opened in the Web app, the user
can then decide to open the document in the client application
Use the server default When a user clicks on a document in this
library, the document will open by using the default open behavior
specified for the server on which SharePoint 2010 Products is
installed.