What decides which machine a Timer Job is run on for jobs that have a lock of type JobLockType.Job?
Is it completely random?
I was researching the job lock types last week and found some good resources. Unfortunately none of them answer your question directly.
Scot Hillier: "Where is my timer job?". Scot tests SPJobLockType.Job and it should only run on a single server. He was testing with a console app that submitted the job and it ran on the same server. Scot doesn't mention running the job from central admin and where it gets run.
Robin Meure: About custom Timer Jobs and SPJobLockTypes. This is how I found Scot's article. It's specifically about SPJobLockType.ContentDatabase though.
Andrew Connell on MSDN: Creating Custom Timer Jobs in Windows SharePoint Services 3.0. This describes the various SPJobLockTypes and the whole timer job story. A good 'complete' resource. Again, doesn't answer your question.
The best thing to do is run some tests similar to Scot's, then come back here and leave your results in an answer! :-)
I think its random, perhaps with a tendency to pick the same server that it ran on before if you reinstall the job. On latest engagement we looked at trying to get it to run on the WFE dedicated for being hit by query server, but had no luck with this. I even once saw the job run on two WFE even though it had SPJobLockType.Job !! (Hint: so make sure your code is stable enough to run in two instances even though you think it will always run alone)
It is a mystery to me too. I have had the best success, setting the scope of a timer job to WEB APPLICATION, and not Web or Site, then running the job directly off of the central administration web site.
To do this you need to activate the timer job install feature at the web application scope, make sure your feature doesn't auto activate by default either, or it will install and activate on every application.
Once your timer job is activated from central admin site, the job is forced to run on the server that hosts the central admin site, in 99.9% of the situations, you can determine this up front.
Also worth mentioning that if you intend to interact with managed meta data, you can expect problems if you run the job on anything other than the server that hosts central admin.