I'm not 100% sure, but I think it's as follows:
- Your user opens up the site. It's authenticated via the AD group and gains access to the site.
- The user now creates an alert. SharePoint sends out a confirmation e-mail. This is being done by getting the current context and reading the e-mail address of the current user (which is stored in the users claim probably).
- The alerts are being sent by the timer job though. Since that service doesn't have any context, it has to look up the site to which the alert belongs to. On that site, it'll query the users collection to fetch the user which created the alert and read it's e-mail address. In your case; it won't find the user cause it's simply not there. You might find an error in ULS, you might not; not sure.
If this is so; it's pretty reproducable bug. And easily fixable; SharePoint should just call EnsureUser when it creates an alert, to make sure the current user is in the users collection of the site collection. But that's for Microsoft to fix (report it!)
As for a quick & dirty solution: you could write a little powershell script which loops through the users in the AD group and calls EnsureUser on the site. That way you'll won't have any manual work. Otherwise the only option you've got is to add your users to the SharePoint group and not reuse the AD group.