How to Migrate File Share to SharePoint Online using PowerShell?

April 27, 2023 Salaudeen Rajack 9 Comments how to migrate files from folder share to sharepoint online using powershell, migrate data from file server to sharepoint online, migrate data to sharepoint online powershell, migrate file share to sharepoint online powershell, migrate files to sharepoint online powershell

Requirement: Migrate All Files and Folders from a Network File Share to SharePoint Online.

Migrating your file shares to SharePoint Online is a great way to take advantage of the many features and benefits that come with using SharePoint. However, migrating files can be a complex process, especially if you have a lot of data to move. Luckily, tools and scripts are available that make the process much easier. This blog post will show you how to migrate your file shares to SharePoint Online using PowerShell.

Let’s migrate data from the file server to SharePoint Online using PowerShell. We can also copy or move local files to SharePoint Online using this PowerShell script. Here is my source:

Let’s copy file share with metadata to SharePoint Online using PowerShell:

#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

#Function to migrate all Files and Folders from FileShare to SharePoint Online
Function Migrate-FileShareToSPO()
{ 
    param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath
    )

    #Setup Credentials to connect
    $Cred= Get-Credential
    $Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Cred.Username, $Cred.Password)
 
    #Setup the context
    $Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
    $Ctx.Credentials = $Credentials
      
    #Get the Target Folder to Upload
    $Web = $Ctx.Web
    $Ctx.Load($Web)
    $List = $Web.Lists.GetByTitle($TargetLibraryName)
    $Ctx.Load($List)
    $TargetFolder = $List.RootFolder
    $Ctx.Load($TargetFolder)
    $Ctx.ExecuteQuery() 

    #Get All Files and Folders from the Source
    Get-ChildItem $SourceFolderPath -Recurse | ForEach-Object {
        If ($_.PSIsContainer -eq $True) #If its a Folder!
        {
            $TargetFolderRelativeURL = $TargetFolder.ServerRelativeURL+$_.FullName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/")
            Write-host -f Yellow "Ensuring Folder '$TargetFolderRelativeURL' Exists..."
                    
            #Check Folder Exists
            Try {
                #Get Source Folder's metadata
                $CreatedDate= [DateTime]$_.CreationTime
                $ModifiedDate =  [DateTime]$_.LastWriteTime

                $Folder = $Web.GetFolderByServerRelativeUrl($TargetFolderRelativeURL)
                $Ctx.Load($Folder)
                $Ctx.ExecuteQuery()
 
                #Write-host -f Green "Folder Already Exists!"
            }
            Catch {
                #Create New Sub-Folder
                $Folder=$Web.Folders.Add($TargetFolderRelativeURL)
                $Ctx.ExecuteQuery()
                Write-host -f Green "Created Folder at "$TargetFolderRelativeURL

                #Set Metadata of the Folder
                $FolderProperties = $Folder.ListItemAllFields
                $FolderProperties["Created"] = $CreatedDate
                $FolderProperties["Modified"] = $ModifiedDate
                $FolderProperties.Update()
                $Ctx.ExecuteQuery()
            }
        }
        Else #If its a File
        {
            $TargetFileURL = $TargetFolder.ServerRelativeURL + $_.DirectoryName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/") +"/"+$_.Name          
            $SourceFilePath = $_.FullName

            Write-host -f Yellow "Uploading File '$_' to URL "$TargetFileURL
            #Get the file from disk
            $FileStream = ([System.IO.FileInfo] (Get-Item $SourceFilePath)).OpenRead()
   
            #Upload the File to SharePoint Library's Folder
            $FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
            $FileCreationInfo.Overwrite = $true
            $FileCreationInfo.ContentStream = $FileStream
            $FileCreationInfo.URL = $TargetFileURL
            $FileUploaded = $TargetFolder.Files.Add($FileCreationInfo)
  
            $Ctx.ExecuteQuery()  
            #Close file stream
            $FileStream.Close()

            #Update Metadata of the File
            $FileProperties = $FileUploaded.ListItemAllFields
            $FileProperties["Created"] = [datetime]$_.CreationTime 
            $FileProperties["Modified"] =[datetime]$_.LastWriteTime
            $FileProperties.Update()
            $Ctx.ExecuteQuery()
            Write-host "File '$TargetFileURL' Uploaded Successfully!" -ForegroundColor Green
        }
    }
}

#Set parameter values
$SiteURL="https://Crescent.sharepoint.com/sites/marketing"
$TargetLibraryName="Documents"
$SourceFolderPath="\\Cre-LT575\Salaudeen\Project Documents"
 
#Call the function to Upload All files & folders from network Fileshare to SharePoint Online
Migrate-FileShareToSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName

This PowerShell script uploads all files and folders from given network file share to an existing SharePoint Online document library.

migrate file share to sharepoint online powershell

You can use this PowerShell script to Bulk upload files & folders from a local drive to SharePoint Online or OneDrive! Simply call the function “Migrate-FileShareToSPO” with relevant parameters

This time let’s upload files from a network file share to the SharePoint Online library with PnP PowerShell.

#Parameters
$SiteURL = "https://crescent.sharepoint.com/sites/london/"
$SourceFolderPath = "\\FileServer\Upload"
$ListName = "Documents"

#Connect to PnP Online
Connect-PnPOnline -Url $SiteURL -Interactive

#Get the Target Folder to Upload
$Web = Get-PnPWeb
$List = Get-PnPList $ListName -Includes RootFolder
$TargetFolder = $List.RootFolder

#Get All Files from the Source Folder
$Files = Get-ChildItem -Path $SourceFolderPath -Recurse | Where {!$_.PSIsContainer }

#Iterate through Each file and Upload to SharePoint Online Library
$Counter = 0
$Files | ForEach-Object {
        #Calculate Target Folder URL for the file
        $TargetFolderURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) + $_.DirectoryName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/") 
        #Remove the Leading '/' characters
        If($TargetFolderURL.StartsWith("/")) { $TargetFolderURL = $TargetFolderURL.Remove(0,1)}

        #Display Progress bar
        $Status  = "Uploading File '" + $_.Name + "' to " + $TargetFolderURL +" ($($Counter) of $($Files.Count))"
        Write-Progress -Activity "Uploading Files..." -Status $Status -PercentComplete (($Counter / $Files.Count)  * 100)

        #Check if File exists Already
        $FileURL = $TargetFolderURL+"/"+$_.Name
        $FileExists = Get-PnPFile -Url $FileURL -ErrorAction SilentlyContinue
        If(!$FileExists)
        {
            #Upload File to Folder
            $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
            Write-host "Uploaded File $($_.FullName) to Folder $TargetFolderURL"
        }
        $Counter++
}

Compare Source and Target and Sync New Files from the Source

The above script works fine, but when we have a large number of files, it takes a lot of time as it has to compare each file in the source with the target. So, let’s improve it a bit.

#Function to Ensure Files from Fileshare to SharePoint Online
Function Ensure-FileShareInSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $LibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile,
        [Parameter(Mandatory=$false)] [Int] $SyncInterval = -30 #Number of Days to check
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Import Files Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -ClientId "3c3ca329-f1b9-41xa-316-c3281aab82" -ClientSecret "vKLew0aJabhs/C3mmhSci61/ej2WE6G1eWasq3u4="

        #Get Number of Source Items in the source
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse | Where {$_.LastWriteTime -gt (Get-Date).AddDays($SyncInterval)}).count 

        #Get the Target Library to Upload
        $Web = Get-PnPWeb
        $Library = Get-PnPList $LibraryName -Includes RootFolder
        $TargetFolder = $Library.RootFolder
        $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) 
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse | Where {$_.LastWriteTime -gt (Get-Date).AddDays($SyncInterval)}
        $Source = @($SourceItems | Select @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}, @{Label='Timestamp';Expression={$_.LastWriteTime.ToString("MM/dd/yyyy hh:mm:ss tt")}}, FullName,  PSIsContainer)
    
        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $LibraryName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,[string]::Empty)}},
                            @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}}, @{Label='Timestamp';Expression={$_.FieldValues.Modified.ToString("MM/dd/yyyy hh:mm:ss tt")}})
 
        #Compare Source and Target and upload files which are not in the target
        $Counter = 1
        $FilesDifferent = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, TargetItemURL, PSIsContainer, Timestamp
        $FilesDiff = @($FilesDifferent | Where {$_.SideIndicator -eq "<="})
        $FilesDiffCount = $FilesDiff.Count

        #Check if Source Folder Items count and Target Folder Items count are different
        If($FilesDiffCount -gt 0)
        {
            Write-host "Found $FilesDiffCount new differences in the Source!" 
            Add-content $Logfile -value "Found $FilesDiffCount new differences in the Source!"   
    
            $FilesDiff | ForEach-Object {
                #Calculate Target Folder URL for the file
                $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                #Remove the Leading '/' characters
                If($TargetFolderURL.StartsWith("/")) { $TargetFolderURL = $TargetFolderURL.Remove(0,1)}
                $ItemName = Split-Path $_.FullName -leaf        
                #Replace Invalid Characters
                $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\"*:<>?/\|')), '_') 

                #Display Progress bar
                $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($FilesDiffCount))"
                Write-Progress -Activity "Uploading ..." -Status $Status -PercentComplete (($Counter / $FilesDiffCount) * 100)

                If($_.PSIsContainer)
                {
                    #Ensure Folder
                    $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName)
                    Get-PnPProperty -ClientObject $Folder -Property ListItemAllFields | Out-Null
                    Set-PnPListItem -List $LibraryName -Identity $Folder.ListItemAllFields.Id -Values @{"Modified"=$_.Timestamp;} | Out-Null
                    Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                }
                Else
                {
                    #Upload File
                    $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL -Values @{"Modified"=$_.Timestamp;}
                    Write-host "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no new Items in the Source! Total Items in Source: $SourceItemsCount , Number Items to Sync: $FilesDiffCount, Target: $($Library.Itemcount)" 
            Add-content $Logfile -value "Found no new Items in the Source! Items in Source: $SourceItemsCount ,  Number Items to Sync: $FilesDiffCount, Target: $($Library.Itemcount)" 
        }
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Import Files Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    } 
}
#Call the function
Import-FileShareToSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Reports" `
                            -ListName "Reports" -LogFile "C:\FileShareImportScript\Reports-LOG.log"

What if you want to migrate files from fileshare to SharePoint Online incrementally? This script imports all files from a network file share to the SharePoint Online document library.

#Function to Import FileShare to SharePoint Online Document Library
Function Import-FileSharetoSPO()
{
    param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $ListName,            
        [Parameter(Mandatory=$true)] [string] $LogFile,
        [Parameter(Mandatory=$true)] [string] $ConfigFilePath
    )

    Try {
        Add-content $Logfile -value "`n---------------------- File Import Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------" 
    
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -Interactive

        #Initialize Config File
        Try {
        $ConfigFile = [XML] (Get-Content $ConfigFilePath -ErrorAction Stop)
        }
        Catch [System.Management.Automation.ItemNotFoundException] {
            $Config = "<?xml version='1.0' encoding='UTF-8'?><Config><Timestamp LastRun='01/01/2000 12:00:00 AM'></Timestamp></Config>"
            New-Item -ItemType "File" -Path $ConfigFilePath | Out-Null
            Set-Content -Path $ConfigFilePath -Value $Config    
        }
    
        #Get the Last Run time of the script from Config file
        $ConfigFile = [XML] (Get-Content $ConfigFilePath)
        $ScriptLastRunTime = [DateTime]::ParseExact($ConfigFile.Config.Timestamp.GetAttribute("LastRun"), "dd/MM/yyyy hh:mm:ss tt", $null)

        #Update LastRun Timestamp in config file
        $ConfigFile.Config.Timestamp.SetAttribute("LastRun", (Get-date -format "dd/MM/yyy hh:mm:ss tt"))
        $ConfigFile.Save($ConfigFilePath)

        #Get the Most Recent Item Created Timestamp
        $LastItemTimestamp =  Get-ChildItem -Path $SourceFolderPath -Recurse | Measure-Object -Maximum -Property CreationTime | Select -ExpandProperty Maximum

        #Check if there are new items since last script executed
        If($LastItemTimestamp -ge $ScriptLastRunTime)
        {
            Write-host -f green "Found Changes to the Source since the last execution of script..." 
            Add-content $Logfile -value "Found Changes to the Source since the last execution of script!" 

            #Get the Target Folder to Upload
            $Web = Get-PnPWeb
            $List = Get-PnPList $ListName -Includes RootFolder
            $TargetFolder = $List.RootFolder
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty)

            #Get All New Items from the Source
            $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse | Where-Object {$_.CreationTime -gt $ScriptLastRunTime}
            $Source = $SourceItems | Select FullName, PSIsContainer, @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}
            Add-content $Logfile -value "Number of New Items Found in the Source: $($SourceItems.Count)"

            #Upload Source Items from Fileshare to Target SharePoint Online document library
            $Counter = 1
            $Source | ForEach-Object {
                    #Calculate Target Folder URL
                    $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                    #Remove the Leading '/' characters
                    If($TargetFolderURL.StartsWith("/")) { $TargetFolderURL = $TargetFolderURL.Remove(0,1)}

                    $ItemName = Split-Path $_.FullName -leaf
        
                    #Replace Invalid Characters
                    $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\*:<>?/\|')), '_')

                    #Display Progress bar
                    $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($SourceItems.Count))"
                    Write-Progress -Activity "Uploading ..." -Status $Status -PercentComplete (($Counter / $SourceItems.Count) * 100)

                    If($_.PSIsContainer)
                    {
                        #Ensure Folder
                        $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName)
                        Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                        Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    }
                    Else
                    {
                        #Upload File
                        $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
                        Write-host "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                        Add-content $Logfile -value "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                    }
                    $Counter++
            }
        }
        Else
        {
            Write-host -f Yellow "No Changes detected in the Source!"
            Add-content $Logfile -value "No Changes detected in the Source!" 
        } 
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- File Import Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

#Call the Function with different parameters
Import-FileSharetoSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Notices" `
                            -ListName "Notices" -LogFile "E:\FileShareImportScript\Notices-LOG.log" -ConfigFilePath "E:\FileShareImportScript\Notices-CONFIG.config"

Import-FileSharetoSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Reports" `
                            -ListName "Reports" -LogFile "E:\FileShareImportScript\Reports-LOG.log" -ConfigFilePath "E:\FileShareImportScript\Reports-CONFIG.config"

For the first time, when you execute the script, it creates a config file and updates the “LastRun” timestamp in it. Next time, it fetches files and folders created after the last run time of the script and uploads them.

9 thoughts on “How to Migrate File Share to SharePoint Online using PowerShell?”

Andrew
August 30, 2023 at 12:31 AM

Your scripts are amazing but I was curious if you have a script that adds in syncing the library back to the fileshare?Keeping the files in sync with one another. I have a situation where users cannot give outside access to SharePoint so we use SFTP as a landing zone. I was hoping to keep the files in sync as my team works on the files in SharePoint to allow it to sync back to the SFTP server.
Anon
January 27, 2023 at 6:04 PM

Hi When I run this each file seems to fail with this error:

Add-PnPFile : Server relative urls must start with SPWeb.ServerRelativeUrl

Any ideas please?
- GShark
  April 21, 2023 at 1:04 PM
  
  remove .Replace(“\”,”/”) at line 61
- Salaudeen RajackPost author
  April 27, 2023 at 7:04 PM
  
  This was due to the recent changes in PnP.PowerShell. The script has been updated now to accomodate it.
Elie Karkafy
October 11, 2021 at 8:34 PM

Hi ,

i have a folder that contains a pdf files and each pdf file have its own XML file that contains their metadata. files are identical in names how to upload each pdf file to sharepoint online along with his XML file with the medata based on the XML data of each pdf file
Anonymous
January 18, 2021 at 2:37 PM

Hello guys,

Is it possible to keep the file’s permission while migrating to Sharepoint with PowerShell?
DC
October 10, 2020 at 8:20 AM

I want to migrate from network drive to SharePoint On-Premise library. Can you please help?
- Salaudeen RajackPost author
  October 10, 2020 at 3:02 PM
  
  Here you go: How to Import Network File Share to SharePoint using PowerShell?
Satis..
May 15, 2020 at 8:22 AM

Wow.. Great script!! It worked like a charm.

Is there a way to incrementally copy files with meta data from Source SharePoint Online site library to Destination share point online site library instead of from file share and replace the author of the file if no longer exists with existing user?

Would you be able to modify this script please? Thank you!!

SharePoint Diary

Salaudeen Rajack

9 thoughts on “How to Migrate File Share to SharePoint Online using PowerShell?”

Leave a Reply Cancel reply

PowerShell to Bulk Upload All Files from Network File Share to SharePoint Online

Import File Share to SharePoint Online Document Library using PnP PowerShell

Compare Source and Target and Sync New Files from the Source

File Share to SharePoint Online Migration using PowerShell

Related Posts

Salaudeen Rajack

9 thoughts on “How to Migrate File Share to SharePoint Online using PowerShell?”

Leave a Reply Cancel reply