This feature is not yet supported in SharePoint Online.
Please see the SharePoint UserVoice post here: Custom restrictions of file types that user can upload to a document library.
--
As a workaround, you can create a content approval flow to approve only PDF files and reject all other file extensions uploaded to a document library.
UPDATE:
Here is a sample flow for your reference:
Turn on Content Approval for the document library. The flow will delete new uploaded documents that are not PDF files and approve PDF files automatically.
Here is a reference: Require approval of documents in SharePoint using Microsoft Flow.