How to Sync File Share to SharePoint Online using PowerShell?

Requirement: Sync Fileshare to SharePoint Online using PowerShell.

sync file share to sharepoint online document library using powershell

PowerShell Script to Sync File share to SharePoint Online document Library:

How to sync a file server to SharePoint Online? While my other script Migrate File Share to SharePoint Online using PowerShell imports files and folders from network file share to SharePoint Online, this script syncs file share to SharePoint Online document library.

Import-Module SharePointPnPPowerShellOnline

#Function to Import Files from Fileshare to SharePoint Online
Function Import-FileShareToSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Import FileShare Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   
    
        #Get Number of Source Items from the Source Folder
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse).count

        #Get the Target Library to Upload
        $Web = Get-PnPWeb
        $Library = Get-PnPList $TargetLibraryName -Includes RootFolder
        $TargetFolder = $Library.RootFolder

        #Get the site relative path of the target folder
        If($web.ServerRelativeURL -eq "/")
        {
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeUrl
        }
        Else
        {        
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) 
        }  
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse
        $Source = @($SourceItems | Select FullName,  PSIsContainer,
                                     @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}, 
                                            @{Label='LastUpdated';Expression={$_.LastWriteTimeUtc.ToString()}})

        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $TargetLibraryName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}},
                                                    @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,[string]::Empty)}},
                                                        @{Label='LastUpdated';Expression={$_.FieldValues.Modified.ToUniversalTime().ToString()}})

        #Compare Source and Target and upload/update files that are not in the target
        $Counter = 1
        $FilesDiff = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, PSIsContainer, TargetItemURL, LastUpdated
        #$FilesDiff | Export-csv -path "C:\Temp\diff.csv" -NoTypeInformation
        $SourceDelta = @($FilesDiff | Where {$_.SideIndicator -eq "<="}) 
        $SourceDeltaCount = $SourceDelta.Count

        #Check if Source Files are changed
        If($SourceDeltaCount -gt 0)
        {
            Write-host "Found $SourceDeltaCount new differences in the Source!" 
            Add-content $Logfile -value "Found $SourceDeltaCount new differences in the Source!"   
    
            $SourceDelta | Sort-Object TargetItemURL | ForEach-Object {
                #Calculate Target Folder URL for the file
                $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                If($TargetFolderURL.StartsWith("/")) {$TargetFolderURL = $TargetFolderURL.Remove(0,1) }
                $ItemName = Split-Path $_.FullName -leaf
                #Replace Invalid Characters
                $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\*:<>?/\|')), '_')

                #Display Progress bar
                $Status  = "Importing $ItemName to $TargetFolderURL $($Counter) of $($SourceDeltaCount)"
                Write-Progress -Activity "Importing Files from the Source..." -Status $Status -PercentComplete (($Counter / $SourceDeltaCount) * 100)

                If($_.PSIsContainer)
                {
                    #Ensure Folder
                    $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName) -Includes ListItemAllFields
                    
                    Set-PnPListItem -List $TargetLibraryName -Identity $Folder.ListItemAllFields.Id -Values @{"Modified"=  ([DateTime]$_.LastUpdated).ToLocalTime()} | Out-null
                    Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                }
                Else
                {
                    #Upload File
                    $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL -Values @{"Modified"=  ([DateTime]$_.LastUpdated).ToLocalTime()}
                    Write-host "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no new Items in the Source! Total Items in Source: $SourceItemsCount , Number Items in Target: $($Library.Itemcount)" 
            Add-content $Logfile -value "Found no new Items in the Source! Items in Source: $SourceItemsCount ,  Number Items in Target: $($Library.Itemcount)" 
        }      
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Import File Share Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

#Function to Remove Files Delta in SharePoint Online (Files that are no longer exists in FileShare )
Function Remove-FileShareDeltaInSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Remove FileShare Delta Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   

        #Get Number of Source Items from the Source Folder
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse).count

        #Get the Target Library
        $Web = Get-PnPWeb
        $Library = Get-PnPList $TargetLibraryName -Includes RootFolder
        $TargetFolder = $Library.RootFolder

        #Get the site relative path of the target folder
        If($web.ServerRelativeURL -eq "/")
        {
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeUrl
        }
        Else
        {        
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,[string]::Empty) 
        }        
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse
        $Source = @($SourceItems | Select FullName,  PSIsContainer,
                                     @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}, 
                                            @{Label='LastUpdated';Expression={$_.LastWriteTimeUtc.ToString()}})

        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $TargetLibraryName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}},
                                                    @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,[string]::Empty)}},
                                                        @{Label='LastUpdated';Expression={$_.FieldValues.Modified.ToUniversalTime().ToString()}})

        #Compare Source and Target and remove files that are not in the Source
        $Counter = 1
        $FilesDiff = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, PSIsContainer, TargetItemURL, LastUpdated     
        $TargetDelta = @($FilesDiff | Where {$_.SideIndicator -eq "=>"})
        $TargetDeltaCount = $TargetDelta.Count

        #Check if Target Files Needs to be deleted
        If($TargetDeltaCount -gt 0)
        {
            Write-host "Found $TargetDeltaCount differences in the Target!" 
            Add-content $Logfile -value "Found $TargetDeltaCount differences in the Target!"   
    
            $TargetDelta | Sort-Object TargetItemURL -Descending | ForEach-Object {
                #Display Progress bar
                $Status  = "Removing Item " + $_.TargetItemURL +" ($($Counter) of $($TargetDeltaCount))"
                Write-Progress -Activity "Removing Items in the Target..." -Status $Status -PercentComplete (($Counter / $TargetDeltaCount) * 100)

                If($_.PSIsContainer)
                {
                    #Empty and Remove the Folder
                    $Folder  = Get-PnPFolder -Url $_.TargetItemURL -ErrorAction SilentlyContinue
                    If($Folder -ne $Null)
                    {
                        $Folder.Recycle() | Out-Null
                        Invoke-PnPQuery

                        Write-host "Removed Folder '$($_.TargetItemURL)'"
                        Add-content $Logfile -value "Removed Folder '$($_.TargetItemURL)'"
                    }
                }
                Else
                {
                    $File = Get-PnPFile -Url $_.TargetItemURL -ErrorAction SilentlyContinue
                    If($File -ne $Null)
                    {
                        #Remove the File
                        Remove-PnPFile -SiteRelativeUrl $_.TargetItemURL -Force
                        Write-host "Removed File '$($_.TargetItemURL)'"
                        Add-content $Logfile -value "Removed File '$($_.TargetItemURL)'"
                    }
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no differences in the Target! Total Items in Source: $SourceItemsCount , Number Items in Target: $($Library.Itemcount)" 
            Add-content $Logfile -value "Found no differences in the Target! Items in Source: $SourceItemsCount ,  Number Items in Target: $($Library.Itemcount)" 
        }
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Remove FileShare Delta Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

Function Sync-FileShareToSPO()
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -UseWebLogin

        #Call the function to Import New Files from Fileshare to SPO
        Import-FileShareToSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName -LogFile $LogFile

        #Call the function to Remove Files in SPO that are moved/deleted in Fileshare
        Remove-FileShareDeltaInSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName -LogFile $LogFile
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message
    }
    Finally {
       DisConnect-PnPOnline
    }
}

#Call the Function to Sync Files from Fileshare to SharePoint Online
Sync-FileShareToSPO -SiteURL "https://crescent.sharepoint.com/sites/Projects" -SourceFolderPath "\\FileServer\Reports" `
                            -TargetLibraryName "Documents" -LogFile "C:\Temp\SyncFileShare-LOG.log"

This script checks the timestamp of each file from the given fileshare and uploads it to SharePoint Online if they are updated in the File share. Similarly, any file deleted in Fileshare will be deleted in SharePoint Online as well. It generates a log file also every time it runs. Please note, this script does only one-way sync from Fileshare to SharePoint Online. You can Schedule PowerShell script in Windows Task scheduler so that it runs on schedule. E.g. for every 10 min.

You can also use this script to sync files from local drive to SharePoint by changing the SourceFolderPath parameter to any folder on your computer. E.g. “C:\Temp\Upload”

Salaudeen Rajack

Salaudeen Rajack is a SharePoint Architect with Two decades of SharePoint Experience. He loves sharing his knowledge and experiences with the SharePoint community, through his real-world articles!

6 thoughts on “How to Sync File Share to SharePoint Online using PowerShell?

  • March 2, 2021 at 8:22 PM

    Same Error as above. Server relative urls must start with SPWeb.ServerRelativeUrl

    Reply
  • March 2, 2021 at 1:23 PM

    Ensured File ‘millnlstoragecadlogfileswijziging.txt’ to Folder /Shared Documents
    Add-PnPFile : Server relative urls must start with SPWeb.ServerRelativeUrl
    At D:_MillscriptsSync-FileShareToSPO.ps1:84 char:30
    … $File = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL -V …
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    CategoryInfo : WriteError: (:) [Add-PnPFile], ServerException
    FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Files.AddFile

    Reply
    • October 13, 2021 at 1:22 PM

      Well, This is because the “Add-PnPFile” cmdlet in new PnP.PowerShell only accepts the folder without leading “/” character. E.g. Add-PnPFile -Path “C:\temp\sample.txt” -Folder “Branding/2020/New” will work. However,
      Add-PnPFile -Path “C:\temp\sample.txt” -Folder “/Branding/2020/New” will fail!

      So, I’ve updated the script now and verified! Line#64

      Reply
  • February 17, 2021 at 9:25 AM

    Hello
    first – thanks for the good work Salaudeen
    I also have a problem with the date-format as i’m based in germany:

    Error: Cannot convert value “17.02.2021 08:44:11” to type “System.DateTime”. Error: “String was not recognized as a valid DateTime.”

    did you already found an answer for this?

    thanks to all!

    Reply
  • January 11, 2021 at 12:20 PM

    Hi there, I appreciate your work and have used some of your scripts for my batch jobs.

    I am based at India and this script has two issues which I would like to bring to your attention.

    In getting list section: Comparing dates with UTC format does not work. I got a workaround though as @{Label=’LastUpdated’;Expression={$_.LastWriteTimeUTC.ToString(‘MM-dd-yyyy HH:mm’, [System.Globalization.CultureInfo]::InvariantCulture)}}) and @{Label=’LastUpdated’;Expression={$_.FieldValues.Modified.ToUniversalTime().ToString(‘MM-dd-yyyy HH:mm’, [System.Globalization.CultureInfo]::InvariantCulture)}}).

    The script uploads files after this modification, still there is second issue which I am trying to figure out.
    After uploading files and folder, the “Compare Source and Target and remove files which are not in the Source” section starts and removes all files and folders it just uploaded.

    Maybe it’s related to same date comparisons. I will let you know.

    Please keep up the good work.

    and

    Reply
  • November 5, 2020 at 2:08 PM

    Hi There, Thanks for putting effort into making these scripts available!
    I’m running into issues with larger files…. seems like there is a 250 MB limit?
    _Add-PnPFile : The underlying connection was closed: An unexpected error occurred on a send.
    https://github.com/pnp/PnP-PowerShell/issues/936

    Do you know of a way around this?

    Reply

Leave a Reply