How to Migrate File Share to SharePoint Online using PowerShell?

Requirement: Migrate All Files and Folders from a Network File Share to SharePoint Online.

PowerShell to Bulk Upload All Files from Network File Share to SharePoint Online

Migrating your file shares to SharePoint Online is a great way to take advantage of the many features and benefits that come with using SharePoint. However, migrating files can be a complex process, especially if you have a lot of data to move. Luckily, tools and scripts are available that make the process much easier. This blog post will show you how to migrate your file shares to SharePoint Online using PowerShell.

Let’s migrate data from the file server to SharePoint Online using PowerShell. We can also copy or move local files to SharePoint Online using this PowerShell script. Here is my source:

copy file share with metadata to sharepoint online using powershell

Let’s copy file share with metadata to SharePoint Online using PowerShell:

#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

#Function to migrate all Files and Folders from FileShare to SharePoint Online
Function Migrate-FileShareToSPO()
{ 
    param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath
    )

    #Setup Credentials to connect
    $Cred= Get-Credential
    $Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Cred.Username, $Cred.Password)
 
    #Setup the context
    $Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
    $Ctx.Credentials = $Credentials
      
    #Get the Target Folder to Upload
    $Web = $Ctx.Web
    $Ctx.Load($Web)
    $List = $Web.Lists.GetByTitle($TargetLibraryName)
    $Ctx.Load($List)
    $TargetFolder = $List.RootFolder
    $Ctx.Load($TargetFolder)
    $Ctx.ExecuteQuery() 

    #Get All Files and Folders from the Source
    Get-ChildItem $SourceFolderPath -Recurse | ForEach-Object {
        If ($_.PSIsContainer -eq $True) #If its a Folder!
        {
            $TargetFolderRelativeURL = $TargetFolder.ServerRelativeURL+$_.FullName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/")
            Write-host -f Yellow "Ensuring Folder '$TargetFolderRelativeURL' Exists..."
                    
            #Check Folder Exists
            Try {
                #Get Source Folder's metadata
                $CreatedDate= [DateTime]$_.CreationTime
                $ModifiedDate =  [DateTime]$_.LastWriteTime

                $Folder = $Web.GetFolderByServerRelativeUrl($TargetFolderRelativeURL)
                $Ctx.Load($Folder)
                $Ctx.ExecuteQuery()
 
                #Write-host -f Green "Folder Already Exists!"
            }
            Catch {
                #Create New Sub-Folder
                $Folder=$Web.Folders.Add($TargetFolderRelativeURL)
                $Ctx.ExecuteQuery()
                Write-host -f Green "Created Folder at "$TargetFolderRelativeURL

                #Set Metadata of the Folder
                $FolderProperties = $Folder.ListItemAllFields
                $FolderProperties["Created"] = $CreatedDate
                $FolderProperties["Modified"] = $ModifiedDate
                $FolderProperties.Update()
                $Ctx.ExecuteQuery()
            }
        }
        Else #If its a File
        {
            $TargetFileURL = $TargetFolder.ServerRelativeURL + $_.DirectoryName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/") +"/"+$_.Name          
            $SourceFilePath = $_.FullName

            Write-host -f Yellow "Uploading File '$_' to URL "$TargetFileURL
            #Get the file from disk
            $FileStream = ([System.IO.FileInfo] (Get-Item $SourceFilePath)).OpenRead()
   
            #Upload the File to SharePoint Library's Folder
            $FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
            $FileCreationInfo.Overwrite = $true
            $FileCreationInfo.ContentStream = $FileStream
            $FileCreationInfo.URL = $TargetFileURL
            $FileUploaded = $TargetFolder.Files.Add($FileCreationInfo)
  
            $Ctx.ExecuteQuery()  
            #Close file stream
            $FileStream.Close()

            #Update Metadata of the File
            $FileProperties = $FileUploaded.ListItemAllFields
            $FileProperties["Created"] = [datetime]$_.CreationTime 
            $FileProperties["Modified"] =[datetime]$_.LastWriteTime
            $FileProperties.Update()
            $Ctx.ExecuteQuery()
            Write-host "File '$TargetFileURL' Uploaded Successfully!" -ForegroundColor Green
        }
    }
}

#Set parameter values
$SiteURL="https://Crescent.sharepoint.com/sites/marketing"
$TargetLibraryName="Documents"
$SourceFolderPath="\\Cre-LT575\Salaudeen\Project Documents"
 
#Call the function to Upload All files & folders from network Fileshare to SharePoint Online
Migrate-FileShareToSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName

This PowerShell script uploads all files and folders from given network file share to an existing SharePoint Online document library.

migrate file share to sharepoint online powershell
You can use this PowerShell script to Bulk upload files & folders from a local drive to SharePoint Online or OneDrive! Simply call the function “Migrate-FileShareToSPO” with relevant parameters

Import File Share to SharePoint Online Document Library using PnP PowerShell

This time let’s upload files from a network file share to SharePoint Online library with PnP PowerShell.

#Parameters
$SiteURL = "https://crescent.sharepoint.com/sites/london/"
$SourceFolderPath = "\\FileServer\Upload"
$ListName = "Documents"

#Connect to PnP Online
Connect-PnPOnline -Url $SiteURL -Interactive

#Get the Target Folder to Upload
$Web = Get-PnPWeb
$List = Get-PnPList $ListName -Includes RootFolder
$TargetFolder = $List.RootFolder

#Get All Files from the Source Folder
$Files = Get-ChildItem -Path $SourceFolderPath -Recurse | Where {!$_.PSIsContainer }

#Iterate through Each file and Upload to SharePoint Online Library
$Counter = 0
$Files | ForEach-Object {
        #Calculate Target Folder URL for the file
        $TargetFolderURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) + $_.DirectoryName.Replace($SourceFolderPath,[string]::Empty).Replace("\","/") 
        
        #Display Progress bar
        $Status  = "Uploading File '" + $_.Name + "' to " + $TargetFolderURL +" ($($Counter) of $($Files.Count))"
        Write-Progress -Activity "Uploading Files..." -Status $Status -PercentComplete (($Counter / $Files.Count)  * 100)

        #Check if File exists Already
        $FileURL = $TargetFolderURL+"/"+$_.Name
        $FileExists = Get-PnPFile -Url $FileURL -ErrorAction SilentlyContinue
        If(!$FileExists)
        {
            #Upload File to Folder
            $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
            Write-host "Uploaded File $($_.FullName) to Folder $TargetFolderURL"
        }
        $Counter++
}

Compare Source and Target and Sync New Files from the Source

The above script works fine, but when we have a large number of files, it takes a lot of time as it has to compare each file in the source with the target. So, let’s improve it a bit.

#Function to Ensure Files from Fileshare to SharePoint Online
Function Ensure-FileShareInSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $LibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile,
        [Parameter(Mandatory=$false)] [Int] $SyncInterval = -30 #Number of Days to check
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Import Files Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -ClientId "3c3ca329-f1b9-41xa-316-c3281aab82" -ClientSecret "vKLew0aJabhs/C3mmhSci61/ej2WE6G1eWasq3u4="

        #Get Number of Source Items in the source
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse | Where {$_.LastWriteTime -gt (Get-Date).AddDays($SyncInterval)}).count 

        #Get the Target Library to Upload
        $Web = Get-PnPWeb
        $Library = Get-PnPList $LibraryName -Includes RootFolder
        $TargetFolder = $Library.RootFolder
        $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) 
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse | Where {$_.LastWriteTime -gt (Get-Date).AddDays($SyncInterval)}
        $Source = @($SourceItems | Select @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}, @{Label='Timestamp';Expression={$_.LastWriteTime.ToString("MM/dd/yyyy hh:mm:ss tt")}}, FullName,  PSIsContainer)
    
        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $LibraryName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,[string]::Empty)}},
                            @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}}, @{Label='Timestamp';Expression={$_.FieldValues.Modified.ToString("MM/dd/yyyy hh:mm:ss tt")}})
 
        #Compare Source and Target and upload files which are not in the target
        $Counter = 1
        $FilesDifferent = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, TargetItemURL, PSIsContainer, Timestamp
        $FilesDiff = @($FilesDifferent | Where {$_.SideIndicator -eq "<="})
        $FilesDiffCount = $FilesDiff.Count

        #Check if Source Folder Items count and Target Folder Items count are different
        If($FilesDiffCount -gt 0)
        {
            Write-host "Found $FilesDiffCount new differences in the Source!" 
            Add-content $Logfile -value "Found $FilesDiffCount new differences in the Source!"   
    
            $FilesDiff | ForEach-Object {
                #Calculate Target Folder URL for the file
                $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                $ItemName = Split-Path $_.FullName -leaf        
                #Replace Invalid Characters
                $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\"*:<>?/\|')), '_') 

                #Display Progress bar
                $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($FilesDiffCount))"
                Write-Progress -Activity "Uploading ..." -Status $Status -PercentComplete (($Counter / $FilesDiffCount) * 100)

                If($_.PSIsContainer)
                {
                    #Ensure Folder
                    $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName)
                    Get-PnPProperty -ClientObject $Folder -Property ListItemAllFields | Out-Null
                    Set-PnPListItem -List $LibraryName -Identity $Folder.ListItemAllFields.Id -Values @{"Modified"=$_.Timestamp;} | Out-Null
                    Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                }
                Else
                {
                    #Upload File
                    $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL -Values @{"Modified"=$_.Timestamp;}
                    Write-host "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no new Items in the Source! Total Items in Source: $SourceItemsCount , Number Items to Sync: $FilesDiffCount, Target: $($Library.Itemcount)" 
            Add-content $Logfile -value "Found no new Items in the Source! Items in Source: $SourceItemsCount ,  Number Items to Sync: $FilesDiffCount, Target: $($Library.Itemcount)" 
        }
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Import Files Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    } 
}
#Call the function
Import-FileShareToSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Reports" `
                            -ListName "Reports" -LogFile "C:\FileShareImportScript\Reports-LOG.log"

File Share to SharePoint Online Migration using PowerShell

What if you want to migrate files from fileshare to SharePoint Online incrementally? This script imports all files from a network file share to the SharePoint Online document library.

#Function to Import FileShare to SharePoint Online Document Library
Function Import-FileSharetoSPO()
{
    param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $ListName,            
        [Parameter(Mandatory=$true)] [string] $LogFile,
        [Parameter(Mandatory=$true)] [string] $ConfigFilePath
    )

    Try {
        Add-content $Logfile -value "`n---------------------- File Import Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------" 
    
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -Interactive

        #Initialize Config File
        Try {
        $ConfigFile = [XML] (Get-Content $ConfigFilePath -ErrorAction Stop)
        }
        Catch [System.Management.Automation.ItemNotFoundException] {
            $Config = "<?xml version='1.0' encoding='UTF-8'?><Config><Timestamp LastRun='01/01/2000 12:00:00 AM'></Timestamp></Config>"
            New-Item -ItemType "File" -Path $ConfigFilePath | Out-Null
            Set-Content -Path $ConfigFilePath -Value $Config    
        }
    
        #Get the Last Run time of the script from Config file
        $ConfigFile = [XML] (Get-Content $ConfigFilePath)
        $ScriptLastRunTime = [DateTime]::ParseExact($ConfigFile.Config.Timestamp.GetAttribute("LastRun"), "dd/MM/yyyy hh:mm:ss tt", $null)

        #Update LastRun Timestamp in config file
        $ConfigFile.Config.Timestamp.SetAttribute("LastRun", (Get-date -format "dd/MM/yyy hh:mm:ss tt"))
        $ConfigFile.Save($ConfigFilePath)

        #Get the Most Recent Item Created Timestamp
        $LastItemTimestamp =  Get-ChildItem -Path $SourceFolderPath -Recurse | Measure-Object -Maximum -Property CreationTime | Select -ExpandProperty Maximum

        #Check if there are new items since last script executed
        If($LastItemTimestamp -ge $ScriptLastRunTime)
        {
            Write-host -f green "Found Changes to the Source since the last execution of script..." 
            Add-content $Logfile -value "Found Changes to the Source since the last execution of script!" 

            #Get the Target Folder to Upload
            $Web = Get-PnPWeb
            $List = Get-PnPList $ListName -Includes RootFolder
            $TargetFolder = $List.RootFolder
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty)

            #Get All New Items from the Source
            $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse | Where-Object {$_.CreationTime -gt $ScriptLastRunTime}
            $Source = $SourceItems | Select FullName, PSIsContainer, @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}
            Add-content $Logfile -value "Number of New Items Found in the Source: $($SourceItems.Count)"

            #Upload Source Items from Fileshare to Target SharePoint Online document library
            $Counter = 1
            $Source | ForEach-Object {
                    #Calculate Target Folder URL
                    $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                    $ItemName = Split-Path $_.FullName -leaf
        
                    #Replace Invalid Characters
                    $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\*:<>?/\|')), '_')

                    #Display Progress bar
                    $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($SourceItems.Count))"
                    Write-Progress -Activity "Uploading ..." -Status $Status -PercentComplete (($Counter / $SourceItems.Count) * 100)

                    If($_.PSIsContainer)
                    {
                        #Ensure Folder
                        $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName)
                        Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                        Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    }
                    Else
                    {
                        #Upload File
                        $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL
                        Write-host "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                        Add-content $Logfile -value "Uploaded File '$($_.FullName)' to Folder $TargetFolderURL"
                    }
                    $Counter++
            }
        }
        Else
        {
            Write-host -f Yellow "No Changes detected in the Source!"
            Add-content $Logfile -value "No Changes detected in the Source!" 
        } 
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- File Import Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

#Call the Function with different parameters
Import-FileSharetoSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Notices" `
                            -ListName "Notices" -LogFile "E:\FileShareImportScript\Notices-LOG.log" -ConfigFilePath "E:\FileShareImportScript\Notices-CONFIG.config"

Import-FileSharetoSPO -SiteURL "https://crescent.sharepoint.com/sites/Funds" -SourceFolderPath "\\fileserver\Reports" `
                            -ListName "Reports" -LogFile "E:\FileShareImportScript\Reports-LOG.log" -ConfigFilePath "E:\FileShareImportScript\Reports-CONFIG.config"

For the first time, when you execute the script, it creates a config file and updates the “LastRun” timestamp in it. Next time, it fetches files and folders created after the last run time of the script and uploads them.

Salaudeen Rajack

Salaudeen Rajack - SharePoint Expert with Two decades of SharePoint Experience. Love to Share my knowledge and experience with the SharePoint community, through real-time articles!

5 thoughts on “How to Migrate File Share to SharePoint Online using PowerShell?

  • Hi ,

    i have a folder that contains a pdf files and each pdf file have its own XML file that contains their metadata. files are identical in names how to upload each pdf file to sharepoint online along with his XML file with the medata based on the XML data of each pdf file

    Reply
  • Hello guys,

    Is it possible to keep the file’s permission while migrating to Sharepoint with PowerShell?

    Reply
  • I want to migrate from network drive to SharePoint On-Premise library. Can you please help?

    Reply
  • Wow.. Great script!! It worked like a charm.

    Is there a way to incrementally copy files with meta data from Source SharePoint Online site library to Destination share point online site library instead of from file share and replace the author of the file if no longer exists with existing user?

    Would you be able to modify this script please? Thank you!!

    Reply

Leave a Reply

Your email address will not be published.