Migrate SharePoint 2010 Document Libraries to SharePoint Online using PowerShell

Requirement: Migrate Document Libraries from SharePoint 2010 to SharePoint Online.

How to Migrate SharePoint 2010 Document Libraries to SharePoint Online?
The idea is: Export all document libraries from SharePoint On-premises to local drive, export metadata of each document to a CSV file and then Import them to SharePoint Online.

Before we begin, Please note, this is not a full fledged migration script which can migrate metadata, security, version history, etc. This script only copies document libraries with its file-folders and metadata "Created by", "Created On", "Modified By", "Modified On" to SharePoint Online. You need 3rd party migration tools if you want migrate all missing pieces!

Step 1: Export SharePoint On-Premises Document Library

As a first step, Login to any of your SharePoint On-premises web front end server and run the below PowerShell script to export either a single document library or all document libraries from SharePoint 2010 to local disk of the server. Change the Run-time variables at the bottom of the script according to your environment. Please note, this script exports libraries of a given site (not site collection), you can extend it if needed!

#region Runtime-Variables
$Global:SourceSiteURL = "https://SP2010.crescent.com/sites/marketing"
$Global:LocalFolderPath ="C:\Migration\Downloads"
#endregion Runtime-Variables


PowerShell to Export All Document Libraries from SharePoint 2010
This script Downloads On-premises Document Libraries from to SharePoint 2010 to local disk.
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

#Function to Extract Metadata of a File to CSV File
Function Extract-SPFileMetadata([Microsoft.SharePoint.SPFile]$SPFile)
{
    Try {
        #Calculate URLs
        $FileLibraryRelativeURL = $SPFile.ServerRelativeURL.Replace($SPFile.DocumentLibrary.RootFolder.ServerRelativeUrl,"")
        #Calculate Absolute URL
        If($global:Web.ServerRelativeUrl -eq "/")
        {
            $FileAbsoluteURL=  $("{0}{1}" -f $global:Web.Url, $SPFile.ServerRelativeUrl)
        }
        else
        {
            $FileAbsoluteURL=  $("{0}{1}" -f $global:Web.Url.Replace($global:Web.ServerRelativeUrl,''), $SPFile.ServerRelativeUrl)
        } 

        #Extract the Metadata of the file
        $Metadata = New-Object PSObject 
        $Metadata | Add-Member -MemberType NoteProperty -name "FileName" -value $SPFile.Name
        $Metadata | Add-Member -MemberType NoteProperty -name "ParentLibrary" -value $SPFile.DocumentLibrary.Title
        $Metadata | Add-Member -MemberType NoteProperty -name "FileAbsoluteURL" -value $FileAbsoluteURL
        $Metadata | Add-Member -MemberType NoteProperty -name "FileLibraryRelativeURL" -value $FileLibraryRelativeURL
        $Metadata | Add-Member -MemberType NoteProperty -Name "CreatedBy" -value $SPFile.Author.Email
        $Metadata | Add-Member -MemberType NoteProperty -name "ModifiedBy" -value $SPFile.ModifiedBy.Email
        $Metadata | Add-Member -MemberType NoteProperty -name "CreatedOn" -value $SPFile.TimeCreated 
        $Metadata | Add-Member -MemberType NoteProperty -name "ModifiedOn" -value $SPFile.TimeLastModified

        #Send the Metadata to CSV File
        If (-not(Test-Path $global:MetadataFile))
        {
            $Metadata | Export-Csv $global:MetadataFile -NoTypeInformation
        }
        else
        {
            #Append data to Existing CSV File - Remove Duplicates
            $CSVData = [Object[]] (Import-Csv $global:MetadataFile)
            $CSVData + $Metadata | Sort FileAbsoluteURL -unique | Export-Csv $global:MetadataFile -NoTypeInformation
        }
        Write-host -f Green "`tMetadata Extracted from the File:"$SPFile.ServerRelativeURL
     }
    Catch {
        write-host -f Red "Error Getting Metadatas:" $_.Exception.Message
    }
}

#Function to Download All Files from a SharePoint Folder
Function Download-SPFolder($SPFolderURL, $LocalFolderPath)
{
    Try {
        #Get the Source SharePoint Folder
        $SPFolder = $global:web.GetFolder($SPFolderURL)
 
        $LocalFolderPath = Join-Path $LocalFolderPath $SPFolder.Name 
        #Ensure the destination local folder exists! 
        If (!(Test-Path -path $LocalFolderPath))
        {    
            #If it doesn't exists, Create
            $LocalFolder = New-Item $LocalFolderPath -type directory 
        }
 
        #Loop through each file in the folder and download it to Destination
        ForEach ($File in $SPFolder.Files) 
        {
            #Download the file
            $Data = $File.OpenBinary()
            $FilePath= Join-Path $LocalFolderPath $File.Name
            [System.IO.File]::WriteAllBytes($FilePath, $data)
            Write-host -f Green "`tDownloaded the File:"$File.ServerRelativeURL
         
            #Get the Metadata of the File
            Extract-SPFileMetadata -SPFile $File
        }
 
        #Process the Sub Folders & Recursively call the function
        ForEach ($SubFolder in $SPFolder.SubFolders)
        {
            If($SubFolder.Name -ne "Forms") #Leave "Forms" Folder
            {
                #Call the function Recursively
                Download-SPFolder $SubFolder $LocalFolderPath
            }
        }
    }
    Catch {
        Write-host -f Red "Error Downloading Document Library:" $_.Exception.Message
    }  
}

#Function to export a Single Library
Function Export-SPLibrary()
{
    param
    (
        [Parameter(Mandatory=$true)] [string] $LibraryName
    )
    Try {
        #Get the Source Web
        $global:Web = Get-SPWeb $global:SourceSiteURL
        $global:MetadataFile = "$global:LocalFolderPath\Metadata.csv"

        #Get the library
        $Library = $Global:Web.Lists.TryGetList($LibraryName)

        #Get All Files from the Library and export
        If($Library -ne $Null)
        {
            Write-host -f magenta "Downloading Document Library:" $Library.Title

            #Call the function to download the document library
            Download-SPFolder -SPFolderURL $Library.RootFolder.Url -LocalFolderPath $global:LocalFolderPath
        }
        Else
        {
            Write-host -f Yellow "Could not Find Document Library:" $LibraryName
        }

        Write-host "*** Completed downloading Library '$LibraryName'  ***"
    }
    Catch {
        Write-host -f Red "Error Downloading Document Library:" $_.Exception.Message
    }  
}

#Function to export all libraries in a SharePoint Site
Function Export-SPAllLibraries()
{
    Try {
        #Get the Source Web
        $global:Web = Get-SPWeb $global:SourceSiteURL

        #Delete any existing files and folders in the download location
        If (Test-Path $global:LocalFolderPath) {Get-ChildItem -Path $global:LocalFolderPath -Recurse| ForEach-object {Remove-item -Recurse -path $_.FullName }}

        #Arry to Skip System Libraries
        $SystemLibraries =@("Pages", "Converted Forms", "Master Page Gallery", "Customized Reports", 
                 "Form Templates", "Images", "List Template Gallery", "Theme Gallery", "Reporting Templates", 
                          "Site Collection Documents", "Site Collection Images", "Site Pages", "Solution Gallery", 
                               "Style Library", "Web Part Gallery","Site Assets", "wfpub")

        #Get all document libraries - Exclude Hidden and System Libraries
        $LibraryCollection = $Web.lists | Where-Object  { ($_.BaseType -eq "DocumentLibrary") -and ($_.hidden -eq $false) -and ($SystemLibraries -notcontains $_.Title)}

        #Iterate through each list and export
        ForEach($Library in $LibraryCollection)
        {
            Write-host -f magenta "Downloading Document Library:" $Library.Title

            #Call the function to download the document library
            Download-SPFolder -SPFolderURL $Library.RootFolder.Url -LocalFolderPath $global:LocalFolderPath
        } 

        Write-host "*** Downloaded All Libraries From the Given Site! ***"
    }
    Catch {
        Write-host -f Red "Error Downloading All Document Libraries:" $_.Exception.Message
    }  
}

#region Runtime-Variables
$Global:SourceSiteURL = "https://SP2010.crescent.com/sites/marketing"
$Global:LocalFolderPath ="C:\Migration\Downloads"
#endregion Runtime-Variables

#Call the Function to export all document libraries from a site
Export-SPLibraries

#To Export a specific library, use: 
#Export-SPLibrary -LibraryName "Shared Documents"

Run the Script. Once executed, this script downloads all document libraries to the given LocalFolderPath. With this script you can either download all document libraries or a specific document library from a SharePoint On-premises site.

Also the script extracts document's metadata to a CSV file named "Metadata.csv" in the given $LocalFolderPath.

Update the Metadata If needed
The metadata.csv file has File's metadata for fields such as "Created By", "Modified By", etc. Make sure the Email Id's listed in the CSV file are valid in SharePoint Online environment. If needed, You can Update this CSV file and save.

Step 2. Import Document Libraries to SharePoint Online

Now, Copy the downloaded folder (Here, its "C:\Migration\Downloads") to your laptop or any machine where SharePoint Online Client SDK is installed and target SharePoint Online site is accessible. Copy the below script to PowerShell ISE, change the highlighted run time variables according to your environment. Make sure the site given at TargetSiteURL is already created and then execute the script!

#region Runtime-Variables
$Global:SourcePath = "C:\Migration\Downloads"
$Global:TargetSiteURL = "https://crescent.sharepoint.com/Sites/Marketing"
#endregion Runtime-Variables

Upload SharePoint 2010 Document Libraries to SharePoint Online and Set Metadata
#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

#Function to Ensure SharePoint Online User
Function Ensure-SPOUser([string]$UserID)
{
    Try {
        $User = $Global:Web.EnsureUser($UserID)
        $Global:Ctx.Load($User)
        $Global:Ctx.ExecuteQuery()
        Return $User
    }
    Catch {
        write-host -f Red "`t`t`tError Resolving User $UserID :" $_.Exception.Message
        Return $Null
    }
}

#Function to Set the Metadata of a Document
Function SetSPO-DocumentMetadata()
{
    param
    (
        [Parameter(Mandatory=$true)] [Microsoft.SharePoint.Client.File] $File,
        [Parameter(Mandatory=$true)] [Microsoft.SharePoint.Client.List] $TargetLibrary
    )    
    Try {
        #Calculate the Library Relative URL of the File
        $Global:Ctx.Load($TargetLibrary.RootFolder)
        $Global:Ctx.ExecuteQuery()
        $FileLibraryRelativeURL = $File.ServerRelativeUrl.Replace($TargetLibrary.RootFolder.ServerRelativeUrl,"")

        #Import Metadata CSV File
        $MetadataCSVPath = "$Global:SourcePath\Metadata.csv"
        $MetadataFile = Import-Csv -LiteralPath $MetadataCSVPath 
        #Get the Metadata of the File
        $Metadata = $MetadataFile | Where-Object {($_.ParentLibrary -eq ($TargetLibrary.Title)) -and $_.FileLibraryRelativeURL -eq $FileLibraryRelativeURL}
        If($Metadata)
        {
            Write-host -f Yellow "`t`tUpdating Metadata for File '$($File.ServerRelativeURL)'"

            #Get 'Created By' and 'Modified By' Users
            $Author = Ensure-SPOUser -UserID $Metadata.CreatedBy
            $Editor= Ensure-SPOUser -UserID $Metadata.ModifiedBy

            #Set Metadata of the File
            $ListItem = $File.ListItemAllFields
            If($Author -ne $Null)
            {
                $Listitem["Author"] = $Author
            }
            If($Editor -ne $Null)
            {
                $ListItem["Editor"] = $Editor
            }
            $ListItem["Created"] =  $Metadata.CreatedOn
            $ListItem["Modified"] = $Metadata.ModifiedOn
            $ListItem.Update()
            $Global:Ctx.ExecuteQuery()

            Write-host -f Green "`t`t`tMetadata for '$($File.ServerRelativeURL)' has been Updated Successfully!"
        }
    }
    Catch {
        write-host -f Red "`t`t`tError updating Metadata of the Document:" $_.Exception.Message
    }
}

#Function to migrate all Files and Folders from Local Path to SharePoint Online
Function MigrateSPO-LocalFolder()
{ 
    param
    (
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [Microsoft.SharePoint.Client.List] $TargetLibrary
    )
    Try {
        #Get the Target Folder to Upload
        $TargetFolder = $TargetLibrary.RootFolder
        $Global:Ctx.Load($TargetFolder)
        $Global:Ctx.ExecuteQuery()
 
        #Get All Files and Folders from the Source
        Get-ChildItem $SourceFolderPath -Recurse | ForEach-Object {
            If ($_.PSIsContainer -eq $True) #If its a Folder!
            {
                $TargetFolderRelativeURL = $TargetFolder.ServerRelativeURL+$_.FullName.Replace($SourceFolderPath,"").Replace("\","/")
                Write-host -f Yellow "`t`tEnsuring Folder '$TargetFolderRelativeURL'"
                     
                #Check If Folder Exists
                Try {
                    $Folder = $Global:Web.GetFolderByServerRelativeUrl($TargetFolderRelativeURL)
                    $Global:Ctx.Load($Folder)
                    $Global:Ctx.ExecuteQuery()
  
                    Write-host -f Magenta "`t`t`tFolder Already Exists: $TargetFolderRelativeURL"
                }
                Catch {
                    #Create New Sub-Folder
                    $Folder= $Global:Web.Folders.Add($TargetFolderRelativeURL)
                    $Global:Ctx.ExecuteQuery()
                    Write-host -f Green "`t`t`tCreated Folder at "$TargetFolderRelativeURL
                    $Global:Ctx.ExecuteQuery()
                }
            }
            Else #Its a File, Upload it!
            {
                #Calculate Source and Destination File Paths
                $TargetFileURL = $TargetFolder.ServerRelativeURL + $_.DirectoryName.Replace($SourceFolderPath,"").Replace("\","/") +"/"+$_.Name          
                $SourceFilePath = $_.FullName
 
                Write-host -f Yellow "`t`tUploading File '$_' to URL "$TargetFileURL
                #Get the file from disk
                $FileStream = ([System.IO.FileInfo] (Get-Item $SourceFilePath)).OpenRead()
    
                #Upload the File to SharePoint Library's Folder
                $FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
                $FileCreationInfo.Overwrite = $True
                $FileCreationInfo.ContentStream = $FileStream
                $FileCreationInfo.URL = $TargetFileURL
                $FileUploaded = $TargetFolder.Files.Add($FileCreationInfo)
                $Global:Ctx.ExecuteQuery()
                $FileStream.Close()                

                Write-host "`t`t`tFile '$TargetFileURL' Uploaded Successfully!" -ForegroundColor Green

                #get the uploaded file
                $File = $Global:Web.GetFileByServerRelativeUrl($TargetFileURL)
                $Global:Ctx.Load($File)
                $Global:Ctx.ExecuteQuery()

                #Update Metadata of the File
                SetSPO-DocumentMetadata -File $File -TargetLibrary $TargetLibrary
            }
        }
    }
    Catch {
        write-host -f Red "`t`t`tError Uploading File:" $_.Exception.Message
    }
}

#Function to Ensure a SharePoint Online document library
Function EnsureSPO-DocumentLibrary()
{
    param
    (
        [Parameter(Mandatory=$true)] [string] $DocumentLibraryName
    )    
    Try {
        Write-host -f Yellow "`nEnsuring Document Library '$DocumentLibraryName'"
        
        #Get All Existing Lists from the web
        $Lists = $Global:Web.Lists
        $Global:Ctx.Load($Lists)
        $Global:Ctx.ExecuteQuery()

        #Check if Library name doesn't exists already
        If(!($Lists.Title -contains $DocumentLibraryName))
        {
            #Create Document Library
            $ListInfo = New-Object Microsoft.SharePoint.Client.ListCreationInformation
            $ListInfo.Title = $DocumentLibraryName
            $ListInfo.TemplateType = 101 #Document Library
            $List = $Global:Web.Lists.Add($ListInfo)
            $List.Update()
            $Global:Ctx.ExecuteQuery()
 
            write-host  -f Green "`tNew Document Library '$DocumentLibraryName' has been created!"
            Return $List
        }
        Else
        {
            #Get the Library
            $List = $Global:Web.Lists.GetByTitle($DocumentLibraryName)
            $Global:Ctx.Load($List)
            $Global:Ctx.ExecuteQuery()
            Return $List

            Write-Host -f Magenta "`tA Document Library '$DocumentLibraryName' Already exist!"
        }
    }
    Catch {
        write-host -f Red "`tError Creating Document Library!" $_.Exception.Message
    }
}

#Main Function
Function Import-SPOLibraries()
{
    Try {
        #Setup the context
        $Global:Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($Global:TargetSiteURL)
        $Global:Ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Global:Cred.UserName,$Global:Cred.Password)
 
        #Get the Web
        $Global:Web = $Global:Ctx.Web
        $Global:Ctx.Load($Global:Web)
        $Global:Ctx.ExecuteQuery()

        #Get Top Level Folders from the Source as "Document Libraries"
        $SourceLibraries = Get-ChildItem -Directory -Path $Global:SourcePath

        #Create Document Libraries
        ForEach($SourceLibrary in $SourceLibraries)
        {
            #call the function to Ensure document library
            $TargetLibrary = EnsureSPO-DocumentLibrary -DocumentLibraryName $SourceLibrary.Name

            #Migrate Files and Folders from the Source to the Destination
            MigrateSPO-LocalFolder -SourceFolderPath $SourceLibrary.FullName -TargetLibrary $TargetLibrary
        }
        Write-host -f Cyan "`n*** Import Completed! ***`n"
    }
    Catch {
        write-host -f Red "Error:" $_.Exception.Message
    }
}

#region Runtime-Variables
$Global:SourcePath = "C:\Migration\Downloads"
$Global:TargetSiteURL = "https://crescent.sharepoint.com/Sites/Marketing"
#endregion Runtime-Variables

#Get Credentials to connect 
$Global:Cred = Get-Credential -Message "Enter the Admin Credentials:"

#Call the function to import document libraries and files to SharePoint Online
Import-SPOLibraries
This script is written to copy document libraries site-by-site and can be extended further!
Migrate SharePoint 2010 Document Libraries to SharePoint Online using PowerShell Migrate SharePoint 2010 Document Libraries to SharePoint Online using PowerShell Reviewed by Salaudeen Rajack on June 16, 2019 Rating: 5

No comments:

Please Login and comment to get your questions answered!

Powered by Blogger.