SharePoint internally uses the function ExpGroupRenderData
to expand or collapse a group of a grouped view. You need to overwrite this function to fire your custom JavaScript code.
UPDATE: Here's the original script code taken from init.js:
function ExpGroupRenderData(htmlToRender, groupName, isLoaded)
{ULSxSy:;
var tbody=document.getElementById("tbod"+groupName+"_");
var wrapDiv=document.createElement("DIV");
var rg=groupName.split("-");
wrapDiv.innerHTML="<TABLE><TBODY id=\"tbod"+ groupName+"_\" isLoaded=\""+isLoaded+ "\">"+htmlToRender+"</TBODY></TABLE>";
tbody.parentNode.replaceChild(wrapDiv.firstChild.firstChild,tbody);
// Add your custom script code here
}
UPDATE 2: I'm sorry if my answer implied to modify internal SharePoint files. It wasn't meant like that. In fact I strongly recommend against modifying such files for reasons @Colin laid out.
As @webdes03 mentioned, you can "overwrite" a built-in function by simply putting the above code into your own file e.g. "myscript.js" and include it into your environment via a feature (my preferred method) or using SharePoint Designer (shudder) to add it to your masterpage.
That's how I usually handle such overwrites without requiring jQuery with a plugin which is not always available in projects I'm involved into.