You actually setup the alias's on the SharePoint servers, not the SQL servers. Run
c:\Windows\System32\cliconfg.exe
on each of your SharePoint servers and then create a new alias with the "Server alias" being the same sql name and instance of sql 2005 and the servername being the name and instance of your sql 2008.
After doing that, make sure all of the DBs are on the SQL 2008 instance and restart all of your SharePoint servers. Assuming everything is setup correctly, SharePoint will now be going to your SQL 2008 box for DBs instead of the SQL 2005 one. All of the DBs on SQL 2005 are unaffected by this change and any other applications still using the SQL 2005 DBs (as long as those apps don't live on the SharePoint boxes) should connect just fine to the SQL 2005 DBs.
The best practice I follow is when installing SharePoint, I got ahead and create sql aliases for my SQL instances I plan on using. The alias names I usually use are SPSQL01, SPSQL02, etc. This way if my sql names change or I need to upgrade SQL, all I need to do is change my aliases on the SharePoint servers to point to the new server.
I hope this makes sense.