Should we use custom master pages, custom page layout, custom theme in SP Online? I heard it is not recommended in SP online, is this true? If it is true then what is the main cause for this.
The general guidance from the PnP group is no, there are other supported approaches to do the desired customizations without modifying the master page.
That being said, they say if you are going to do it, you have to understand the risks of doing so. As soon as you break away, you aren't getting the updates and new features as they are implemented.