New answers tagged language
After a while figured this out. The language of the root site collection https://tenant.sharepoint.com was Finnish so after changing the language on that site collection the page was switched properly to English. All the production activity is running on other site collections so the root was left unnoticed initially.
Top 50 recent answers are included