There's a lot of behind-the-scenes work InfoPath Forms Services on your SharePoint server has to do to render your form in a browser. You can tweak some some performance-based settings in Central Administration, especially postback, the number of times a browser has to communicate with the server.
Rajan Kapoor offers some tweaks on how to limit postback:
Some InfoPath form controls, actions, and features require the browser to communicate with the server during a form session. This exchange of data during the session is called a postback, and usually occurs when a form feature has to send data to the server for processing. Unnecessary postbacks impose an additional load on both the browser and the server. To protect the server, a threshold is set for the maximum number of postbacks per session. This limits the number of postbacks that can be executed during a single session when a user is filling out a form, and prevents malicious users from trying to bring down the server.
He also posts some links that allow you at the very least to understand what's going on with your form as it communicates with the server.
You can also use some web debugging tools like Fiddler to zero in on (and limit) the calls made between the browser and the server.