How do I control the type of document (only .docx allowed) is being uploaded in one of the SharePoint 2010 document libraries?
Through central administration you can block upload file types.
central admin -> Operations -> Blocked FileTypes -> now select the site and add the extension you want to block.
you can read on this:
or as hugh stated have an eventReciver to handle it for selected document libraryies only