PnP Powershell script to get hyperlinks from a page

Question

Before I begin, I need to tell you that I am an absolute beginner when it comes to Powershell, this is my first attempt at it and I've mostly been trying to use AI to help me get it running.

I am trying to create a script that will connect to an SPO site and then output a list of hyperlinks from a particular page to a txt file. This is what I have so far:

# Load the HtmlAgilityPack assembly
Add-Type -Path "C:\report\HtmlAgilityPack.dll"

# Function to decode HTML entities using System.Web.HttpUtility
function Decode-HtmlEntities {
    param (
        [string]$encodedString
    )

    Add-Type -AssemblyName System.Web
    $decodedString = [System.Web.HttpUtility]::HtmlDecode($encodedString)
    return $decodedString
}

# Function to perform multiple decoding steps
function MultiDecode-HtmlEntities {
    param (
        [string]$encodedString
    )

    $decodedString = $encodedString
    for ($i = 0; $i -lt 3; $i++) {
        $decodedString = Decode-HtmlEntities -encodedString $decodedString
    }
    return $decodedString
}

# Function to clean up escape characters
function Clean-UpEscapeCharacters {
    param (
        [string]$inputString
    )

    $cleanedString = $inputString -replace '\\\"', '"'
    return $cleanedString
}

# Connect to SharePoint Online using web login for MFA
Connect-PnPOnline -Url "https://xxx.sharepoint.com/sites/xxx" -UseWebLogin

# Get the specific wiki page
$page = Get-PnPListItem -List "Site Pages" -Query "<View><Query><Where><Eq><FieldRef Name='FileLeafRef'/><Value Type='File'>xxx.aspx</Value></Eq></Where></Query></View>"

# Initialize an array to store hyperlinks with page information
$hyperlinks = @()

if ($page) {
    # Get the page URL
    $pageUrl = "https://xxx.sharepoint.com" + $page.FieldValues["FileRef"]
    Write-Output "Fetching content for: ${pageUrl}"  # Debug line to check page URL

    # Get the rendered HTML of the page using Invoke-WebRequest
    $response = Invoke-WebRequest -Uri $pageUrl -UseBasicParsing
    $content = $response.Content
    Write-Output "Content fetched for ${pageUrl}"  # Debug line to check content

    # Perform multiple decoding steps
    $decodedContent = MultiDecode-HtmlEntities -encodedString $content
    Write-Output "Decoded content for ${pageUrl}"  # Debug line to check decoded content

    # Load the content into HtmlAgilityPack
    $htmlDoc = New-Object HtmlAgilityPack.HtmlDocument
    $htmlDoc.LoadHtml($decodedContent)

    # Extract the page title
    $pageTitle = $htmlDoc.DocumentNode.SelectSingleNode("//title").InnerText
    Write-Output "Page title: ${pageTitle}"  # Output the page title

    # Extract all hyperlinks and link elements
    $links = $htmlDoc.DocumentNode.SelectNodes("//a[@href] | //link[@href]")

    if ($links) {
        Write-Output "Links found on ${pageUrl}: $($links.Count)"
        foreach ($link in $links) {
            $hyperlink = $link.Attributes["href"].Value
            $cleanedHyperlink = Clean-UpEscapeCharacters -inputString $hyperlink

            # Add all hyperlinks to the array without any exclusions
            Write-Output "Link found on ${pageUrl}: ${cleanedHyperlink}"  # Debug line to check each link
            $hyperlinks += "Page: $($page.FieldValues['FileRef']), Link: ${cleanedHyperlink}"
        }
    } else {
        Write-Output "No links found on ${pageUrl}."
    }
} else {
    Write-Output "xxx.aspx page not found."
}

# Output the hyperlinks with page information
$hyperlinks | Out-File -FilePath "C:\report\hyperlinks.txt"

The script does find hyperlinks, but it doesn't retrieve page content - it always gets a Login page, even though it connects successfully to the site.

Is there something obvious I'm missing?

Answer 1

Connect-PnPOnline -Url "https://xxx.sharepoint.com/sites/xxx" -UseWebLogin

This line needs your interactive attention. That would be the login page. Do you login there manually?

Can you also stop the code at $content? And see what it says? What is inside the variable?

What do you get under $page? It looks like it could be a whole collection of pages instead of just one page. How many pages do you have under Site Pages? That could be the reason.

If you want to go through every page, you need a loop through pages

$pages = Get-PnPListItem -List "Site Pages" -Query "<View><Query><Where><Eq><FieldRef Name='FileLeafRef'/><Value Type='File'>xxx.aspx</Value></Eq></Where></Query></View>"


foreach ($page in $pages) {