2

How can I get HTML content of a SharePoint online page from code, e.g. PowerShell?

I tried Invoke-WebRequest with all possible options and nothing worked for me.

It returns another "authentication" page, but I need real page content.

In other words, how to run correctly this PowerShell command?:

Invoke-WebRequest 'https://contoso.sharepoint.com/sites/Birding/_layouts/15/viewlsts.aspx'

Update: viewlsts.aspx is just an example (and there are better methods to get all site lists), but the actual page I need content of is '_layouts/15/appprincipals.aspx'.

Improve this question

1 Answer 1

Reset to default
1

You can try the approach below, but bear in mind that SharePoint Online:

  • Requires authentication. Simply calling Invoke-WebRequest will just return an access denied page.
  • Modern pages use lots of client scripts that run after the page is served to the client. These scripts first pull in data and then render the page contents. Since PowerShell will simply get the initial bear-to-bone page, you won't get what you probably expect to get.

Install prerequisite PowerShell module

Install-Module PnP.PowerShell -Scope CurrentUser

Authenticate and perform a query to a page of your choice

Connect-PnPOnline -url "https://contoso.sharepoint.com" -UseWebLogin 
$HTML_Page_Content = Invoke-PnPSPRestMethod -url /_layouts/15/viewlsts.aspx -Raw 
Write-Host $HTML_Page_Content

Invoke-PnPSPRestMethod can be used together with the -Raw switch to get contents of any SharePoint page. Invoke-PnPSPRestMethod is not restricted to calling only endpoints located under /_api/.

So, instead of doing what you are doing, I suggest simply getting what you want from SharePoint by using PowerShell PnP commandlets. For example, to get a list of lists and libraries, simply use:

Get-PnPList

No need to suffer through the unnecessary web scraping.

Improve this answer
4
  • Hi Denis. You probably meant something like Invoke-PnPSPRestMethod -Url '/_api/web/lists. Good point. But only in case if I needed only list of lists. The page I actually need is _layouts/15/appprincipals.aspx. Any ideas?
    – Vlad Karasov SharePoint MCSE
    Commented Oct 30, 2021 at 12:21
  • @Vladilen. Sorry, but I don't understand. do you mean something is wrong with the code I shared? I remember I tested it and it worked exactly as expected. In your case the last line will be Invoke-PnPSPRestMethod -url /_layouts/15/appprincipals.aspx -Raw.
    – Denis Molodtsov
    Commented Oct 30, 2021 at 13:12
  • 1
    It works! But could you please double-check your PnP versions. In my environment it worked only with PnP.PowerShell and with -UseWebLogin and with -raw option. Any other combination worked OK for /_api but gave 401 against system page. Module SharePointPnPPowerShellOnline did not work at all (EXCEPTION,PnP.PowerShell.Commands.Admin.InvokeSPRestMethod). I tested with SharePointPnPPowerShellOnline v 3.29.2101.0 and PnP.PowerShell 1.8.0.
    – Vlad Karasov SharePoint MCSE
    Commented Oct 31, 2021 at 15:45
  • Hi Vladlen. You are right: Turned out I had PnP.PowerShell installed instead. I didn't realize it since I always had issues with the new library. So, I tried PnP.PowerShell 1.7 and 1.8 versions and they both worked fine.
    – Denis Molodtsov
    Commented Oct 31, 2021 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.