Many folks in the community raised this question about this - they tried to update the group owner using the REST API but they have not been succeeded, ended up with error.
Some of the similar threads are still opened (un-answered):
As an alternate you could try with the PnP PowerShell:
SharePoint Online: Change Group Owner using PnP PowerShell
$SiteURL = "https://crescenttech.sharepoint.com/Sales"
$GroupName="Sales Portal Members"
#Group owner variable: Can be existing group or user account Email
$GroupOwner = "Sales Portal Owners" #or Salaudeen@Crescent.com
#Connect PNP Online
Connect-PnPOnline -Url $SiteURL -Credentials (Get-Credential)
#Set Group Owner
Set-PnPGroup -Identity $GroupName -Owner $GroupOwner
PowerShell to Change Group Owner in SharePoint Online:
#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
$GroupName="Team Site Members"
#$GroupOwnerName="Team Site Owners"
#Setup Credentials to connect
$Cred = Get-Credential
$Cred = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Cred.UserName,$Cred.Password)
#Setup the context
$Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$Ctx.Credentials = $Cred
#Get the Group owner - Can be an another Group or User Account
#$GroupOwner = $Ctx.Web.SiteGroups.GetByName($GroupOwnerName)
$GroupOwner = $Ctx.Web.EnsureUser($GroupOwnerName)
#Get the Group
$Group = $Ctx.Web.SiteGroups.GetByName($GroupName)
#Set the Group Owner
$Group.Owner = $GroupOwner
Write-host -f Green "Group Owner has been Updated!"
write-host -f Red "Error changing Group Owner!" $_.Exception.Message
SharePoint Online: Change Group Owner using PowerShell
Conclusion - update
Based on the research and comments from other experts in the community, it is not possible using the REST API, Microsoft has stopped looking into this.
"This will not work to set a group as the owner. This is a known
issue and the SharePoint team has declined to fix it" - Joshua