Migrate File Share to SharePoint Online using PowerShell

Requirement: Migrate All Files and Folders from a Network File Share to SharePoint Online.

PowerShell to Bulk Upload All Files from Network File Share to SharePoint Online

Let’s migrate data from the file server to SharePoint Online using PowerShell. We can also copy or move local files to SharePoint Online using this PowerShell script. Here is my source:

copy file share with metadata to sharepoint online using powershell

Let’s copy file share with metadata to SharePoint Online using PowerShell:

#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

#Function to migrate all Files and Folders from FileShare to SharePoint Online
Function Migrate-FileShareToSPO()
{ 
    param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath
    )

    #Setup Credentials to connect
    $Cred= Get-Credential
    $Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Cred.Username, $Cred.Password)
 
    #Setup the context
    $Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
    $Ctx.Credentials = $Credentials
      
    #Get the Target Folder to Upload
    $Web = $Ctx.Web
    $Ctx.Load($Web)
    $List = $Web.Lists.GetByTitle($TargetLibraryName)
    $Ctx.Load($List)
    $TargetFolder = $List.RootFolder
    $Ctx.Load($TargetFolder)
    $Ctx.ExecuteQuery() 

    #Get All Files and Folders from the Source
    Get-ChildItem $SourceFolderPath -Recurse | ForEach-Object {
        If ($_.PSIsContainer -eq $True) #If its a Folder!
        {
            $TargetFolderRelativeURL = $TargetFolder.ServerRelativeURL+$_.FullName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/")
            Write-host -f Yellow "Ensuring Folder '$TargetFolderRelativeURL' Exists..."
                    
            #Check Folder Exists
            Try {
                #Get Source Folder's metadata
                $CreatedDate= [DateTime]$_.CreationTime
                $ModifiedDate =  [DateTime]$_.LastWriteTime

                $Folder = $Web.GetFolderByServerRelativeUrl($TargetFolderRelativeURL)
                $Ctx.Load($Folder)
                $Ctx.ExecuteQuery()
 
                #Write-host -f Green "Folder Already Exists!"
            }
            Catch {
                #Create New Sub-Folder
                $Folder=$Web.Folders.Add($TargetFolderRelativeURL)
                $Ctx.ExecuteQuery()
                Write-host -f Green "Created Folder at "$TargetFolderRelativeURL

                #Set Metadata of the Folder
                $FolderProperties = $Folder.ListItemAllFields
                $FolderProperties["Created"] = $CreatedDate
                $FolderProperties["Modified"] = $ModifiedDate
                $FolderProperties.Update()
                $Ctx.ExecuteQuery()
            }
        }
        Else #If its a File
        {
            $TargetFileURL = $TargetFolder.ServerRelativeURL + $_.DirectoryName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/") +"/"+$_.Name          
            $SourceFilePath = $_.FullName

            Write-host -f Yellow "Uploading File '$_' to URL "$TargetFileURL
            #Get the file from disk
            $FileStream = ([System.IO.FileInfo] (Get-Item $SourceFilePath)).OpenRead()
   
            #Upload the File to SharePoint Library's Folder
            $FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
            $FileCreationInfo.Overwrite = $true
            $FileCreationInfo.ContentStream = $FileStream
            $FileCreationInfo.URL = $TargetFileURL
            $FileUploaded = $TargetFolder.Files.Add($FileCreationInfo)
  
            $Ctx.ExecuteQuery()  
            #Close file stream
            $FileStream.Close()

            #Update Metadata of the File
            $FileProperties = $FileUploaded.ListItemAllFields
            $FileProperties["Created"] = [datetime]$_.CreationTime 
            $FileProperties["Modified"] =[datetime]$_.LastWriteTime
            $FileProperties.Update()
            $Ctx.ExecuteQuery()
            Write-host "File '$TargetFileURL' Uploaded Successfully!" -ForegroundColor Green
        }
    }
}

#Set parameter values
$SiteURL="https://crescenttech.sharepoint.com/sites/marketing"
$TargetLibraryName="Documents"
$SourceFolderPath="\\Cre-LT575\Salaudeen\Project Documents"
 
#Call the function to Upload All files & folders from network Fileshare to SharePoint Online
Migrate-FileShareToSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName

This PowerShell script uploads all files and folders from given network file share to an existing SharePoint Online document library.

migrate file share to sharepoint online powershell
You can use this PowerShell script to Bulk upload files & folders from a local drive to SharePoint Online or OneDrive! Simply call the function “Migrate-FileShareToSPO” with relevant parameters

Import File Share to SharePoint Online Document Library using PnP PowerShell

This time, lets upload files from a network file share to SharePoint Online library with PnP PowerShell.

#Parameters
$SiteURL = "https://crescent.sharepoint.com/sites/london/"
$SourceFolderPath = "\\FileServer\Upload"
$ListName = "Documents"

#Connect to PnP Online
Connect-PnPOnline -Url $SiteURL -UseWebLogin

#Get the Target Folder to Upload
$Web = Get-PnPWeb
$List = Get-PnPList $ListName -Includes RootFolder
$TargetFolder = $List.RootFolder

#Get All Files from the Source Folder
$Files = Get-ChildItem -Path $SourceFolderPath -Recurse | Where {!$_.PSIsContainer }

#Iterate through Each file and Upload to SharePoint Online Library
$Counter = 0
$Files | ForEach-Object {
        #Calculate Target Folder URL for the file
        $TargetFolderURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) + $_.DirectoryName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/") 
        
        #Display Progress bar
        $Status  = "Uploading File '" + $_.Name + "' to " + $TargetFolderURL +" ($($Counter) of $($Files.Count))"
        Write-Progress -Activity "Uploading Files..." -Status $Status -PercentComplete (($Counter / $Files.Count)  * 100)

        #Check if File exists Already
        $FileURL = $TargetFolderURL+"/"+$_.Name
        $FileExists = Get-PnPFile -Url $FileURL -ErrorAction SilentlyContinue
        If(!$FileExists)
        {
            #Upload File to Folder
            $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
            Write-host "Uploaded File $($_.FullName) to Folder $TargetFolderURL"
        }
        $Counter++
}

Compare Source and Target and Sync New Files from the Source

The above script works fine, but when we have a large number of files, it takes a lot of time as it has to compare each file in the source with the target. So, let’s improve it a bit.

#Function to Import Files from Fileshare to SharePoint Online
Function Import-FileShareToSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $ListName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Import Files Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -AppId "3c3ca329-f1b9-41xa-316-c3281aab82" -AppSecret "vKLew0aJabhs/C3mmhSci61/ej2WE6G1eWasq3u4="

        #Get Number of Source Items in the source
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse -Force).count 

        #Get the Target List to Upload
        $Web = Get-PnPWeb
        $List = Get-PnPList $ListName -Includes RootFolder
        $TargetFolder = $List.RootFolder
        $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) 
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse -Force
        $Source = @($SourceItems | Select FullName, PSIsContainer, @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}})
    
        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $ListName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,[string]::Empty)}},
                            @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}})
 
        #Compare Source and Target and upload files which are not in the target
        $Counter = 1
        $FilesDiff = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, TargetItemURL, PSIsContainer
        $FilesDiff = @($FilesDiff | Where {$_.SideIndicator -eq "<="})
        $FilesDiffCount = $FilesDiff.Count

        #Check if Source Folder Items count and Target Folder Items count are different
        If($FilesDiffCount -gt 0)
        {
            Write-host "Found difference between Source and Target! Source: $SourceItemsCount Target: $($List.Itemcount)" 
            Add-content $Logfile -value "Found difference between Source and Target! Source: $SourceItemsCount Target: $($List.Itemcount)"  
    
            $FilesDiff | ForEach-Object {
                #Calculate Target Folder URL for the file
                $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                $ItemName = Split-Path $_.FullName -leaf        
                #Replace Invalid Characters
                $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\*:<>?/\|')), '_') 

                #Display Progress bar
                $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($FilesDiffCount))"
                Write-Progress -Activity "Uploading ..." -Status $Status -PercentComplete (($Counter / $FilesDiffCount) * 100)

                If($_.PSIsContainer)
                {
                    #Ensure Folder
                    $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName)
                    Write-host "Created Folder '$($ItemName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                }
                Else
                {
                    #Upload File
                    $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
                    Write-host "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no difference between Source and Target! Source: $SourceItemsCount Target: $($List.Itemcount)" 
            Add-content $Logfile -value "Found no difference between Source and Target! Source: $SourceItemsCount Target: $($List.Itemcount)" 
        }
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Import Files Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    } 
}
#Call the function
Import-FileShareToSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Reports" `
                            -ListName "Reports" -LogFile "C:\FileShareImportScript\Reports-LOG.log"

File Share to SharePoint Online Migration using PowerShell

What if you want to incrementally migrate files from fileshare to SharePoint Online? Well, this script imports all files from a network file share to the SharePoint Online document library.

#Function to Import FileShare to SharePoint Online Document Library
Function Import-FileSharetoSPO()
{
    param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $ListName,            
        [Parameter(Mandatory=$true)] [string] $LogFile,
        [Parameter(Mandatory=$true)] [string] $ConfigFilePath
    )

    Try {
        Add-content $Logfile -value "`n---------------------- File Import Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------" 
    
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -UseWebLogin

        #Initialize Config File
        Try {
        $ConfigFile = [XML] (Get-Content $ConfigFilePath -ErrorAction Stop)
        }
        Catch [System.Management.Automation.ItemNotFoundException] {
            $Config = "<?xml version='1.0' encoding='UTF-8'?><Config><Timestamp LastRun='01/01/2000 12:00:00 AM'></Timestamp></Config>"
            New-Item -ItemType "File" -Path $ConfigFilePath | Out-Null
            Set-Content -Path $ConfigFilePath -Value $Config    
        }
    
        #Get the Last Run time of the script from Config file
        $ConfigFile = [XML] (Get-Content $ConfigFilePath)
        $ScriptLastRunTime = [DateTime]::ParseExact($ConfigFile.Config.Timestamp.GetAttribute("LastRun"), "dd/MM/yyyy hh:mm:ss tt", $null)

        #Update LastRun Timestamp in config file
        $ConfigFile.Config.Timestamp.SetAttribute("LastRun", (Get-date -format "dd/MM/yyy hh:mm:ss tt"))
        $ConfigFile.Save($ConfigFilePath)

        #Get the Most Recent Item Created Timestamp
        $LastItemTimestamp =  Get-ChildItem -Path $SourceFolderPath -Recurse | Measure-Object -Maximum -Property CreationTime | Select -ExpandProperty Maximum

        #Check if there are new items since last script executed
        If($LastItemTimestamp -ge $ScriptLastRunTime)
        {
            Write-host -f green "Found Changes to the Source since the last execution of script..." 
            Add-content $Logfile -value "Found Changes to the Source since the last execution of script!" 

            #Get the Target Folder to Upload
            $Web = Get-PnPWeb
            $List = Get-PnPList $ListName -Includes RootFolder
            $TargetFolder = $List.RootFolder
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty)

            #Get All New Items from the Source
            $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse | Where-Object {$_.CreationTime -gt $ScriptLastRunTime}
            $Source = $SourceItems | Select FullName, PSIsContainer, @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}
            Add-content $Logfile -value "Number of New Items Found in the Source: $($SourceItems.Count)"

            #Upload Source Items from Fileshare to Target SharePoint Online document library
            $Counter = 1
            $Source | ForEach-Object {
                    #Calculate Target Folder URL
                    $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                    $ItemName = Split-Path $_.FullName -leaf
        
                    #Replace Invalid Characters
                    $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\*:<>?/\|')), '_')

                    #Display Progress bar
                    $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($SourceItems.Count))"
                    Write-Progress -Activity "Uploading ..." -Status $Status -PercentComplete (($Counter / $SourceItems.Count) * 100)

                    If($_.PSIsContainer)
                    {
                        #Ensure Folder
                        $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName)
                        Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                        Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    }
                    Else
                    {
                        #Upload File
                        $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
                        Write-host "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                        Add-content $Logfile -value "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                    }
                    $Counter++
            }
        }
        Else
        {
            Write-host -f Yellow "No Changes detected in the Source!"
            Add-content $Logfile -value "No Changes detected in the Source!" 
        } 
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- File Import Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

#Call the Function with different parameters
Import-FileSharetoSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Notices" `
                            -ListName "Notices" -LogFile "E:\FileShareImportScript\Notices-LOG.log" -ConfigFilePath "E:\FileShareImportScript\Notices-CONFIG.config"

Import-FileSharetoSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Reports" `
                            -ListName "Reports" -LogFile "E:\FileShareImportScript\Reports-LOG.log" -ConfigFilePath "E:\FileShareImportScript\Reports-CONFIG.config"

For the first time, when you execute the script, it creates a config file and updates “LastRun” timestamp in it. From the next time, it fetches files and folders created after the last run time of the script and uploads them.

Salaudeen Rajack

Salaudeen Rajack is a SharePoint Architect with Two decades of SharePoint Experience. He loves sharing his knowledge and experiences with the SharePoint community, through his real-world articles!

5 thoughts on “Migrate File Share to SharePoint Online using PowerShell

  • October 11, 2021 at 8:34 PM

    Hi ,

    i have a folder that contains a pdf files and each pdf file have its own XML file that contains their metadata. files are identical in names how to upload each pdf file to sharepoint online along with his XML file with the medata based on the XML data of each pdf file

    Reply
  • January 18, 2021 at 2:37 PM

    Hello guys,

    Is it possible to keep the file’s permission while migrating to Sharepoint with PowerShell?

    Reply
  • October 10, 2020 at 8:20 AM

    I want to migrate from network drive to SharePoint On-Premise library. Can you please help?

    Reply
  • May 15, 2020 at 8:22 AM

    Wow.. Great script!! It worked like a charm.

    Is there a way to incrementally copy files with meta data from Source SharePoint online site library to Destination share point online site library instead of from file share and replace the author of the file if no longer exists with existing user?

    Would you be able to modify this script please? Thank you!!

    Reply

Leave a Reply