One way:
You can set up a timer in client code to make an ajax requerst to the server-side code to check your desirable conditions. One way to do it is a standard ASP.NET technique described here: http://ajaxzen.wordpress.com/tag/icallbackeventhandler/ You can implement it in the custom webpart or delegate control. Then all you need is to receive instructions periodically via ajax request and to show a notification with standard SharePoint notifications framework in the ajax request callback function. The whole work is about twenty lines of code.
Two way:
Another way is to employ the javascript version of the SharePoint client object model - then you can retrive a number of documents/items in the list, save it in the variable on client and compare with the new value next time - if the new value is greater than new documents was added.
Conclusion:
Anyway the key components is the javascript timer + request to server via callback or client object model + standard SharePoint notifications framework.