I set up a modern layout SharePoint web site. But when I edit the left hand navigation I get it only in classic layout. How can I switch it to the modern layout?
Left hand Navigation Bar is only available in the Team Site Template, not the Communication Site template.
If you set up a new Site Collection with the Team Site template, the navigation bar should be automatically in the Modern Experience.
If you created just a Site Page in the Modern Experience in an existing Site Collection (that was built with the Classic Exprerience) then you will have to:
navigate to the Site Pages document library, from the Site Contents page,
locate the new Page (built with the Modern Experience),
press the three doted options button and select "Make Homepage" option, in order to set up the Site Collection with the new Site Page as the Homepage (and subsequentally change to the Modern Experience).
Another bug that i had encountered in the past was that i could not use the Modern Features in a Hub Site when i had the SharePoint Server Publishing Infrastucture Feature enabled. It is similar to what you are facing.
If you are not utilising this Feature, go into site settings, then Site Collection Features and deactivate the below feature.