How to Sync File Share to SharePoint Online using PowerShell?

Requirement: Sync Fileshare to SharePoint Online using PowerShell.
sync file share to sharepoint online document library using powershell

PowerShell Script to Sync File share to SharePoint Online document Library:
How to sync file server to SharePoint Online? While my other script Migrate File Share to SharePoint Online using PowerShell imports files and folders from network file share to SharePoint Online, this script syncs file share to SharePoint Online document library.
Import-Module SharePointPnPPowerShellOnline

#Function to Import Files from Fileshare to SharePoint Online
Function Import-FileShareToSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Import FileShare Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   
    
        #Get Number of Source Items from the Source Folder
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse).count

        #Get the Target Library to Upload
        $Web = Get-PnPWeb
        $Library = Get-PnPList $TargetLibraryName -Includes RootFolder
        $TargetFolder = $Library.RootFolder

        #Get the site relative path of the target Folder
        If($web.ServerRelativeURL -eq "/")
        {
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeUrl
        }
        Else
        {        
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,"") 
        }  
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse
        $Source = @($SourceItems | Select FullName,  PSIsContainer,
                                     @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}, 
                                            @{Label='LastUpdated';Expression={$_.LastWriteTimeUtc.ToString()}})

        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $TargetLibraryName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}},
                                                    @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,"")}},
                                                        @{Label='LastUpdated';Expression={$_.FieldValues.Modified.ToUniversalTime().ToString()}})

        #Compare Source and Target and upload/update files which are not in the target
        $Counter = 1
        $FilesDiff = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, PSIsContainer, TargetItemURL, LastUpdated
        #$FilesDiff | Export-csv -path "C:\Temp\diff.csv" -NoTypeInformation
        $SourceDelta = @($FilesDiff | Where {$_.SideIndicator -eq "<="}) 
        $SourceDeltaCount = $SourceDelta.Count

        #Check if Source Files are changed
        If($SourceDeltaCount -gt 0)
        {
            Write-host "Found $SourceDeltaCount new differences in the Source!" 
            Add-content $Logfile -value "Found $SourceDeltaCount new differences in the Source!"   
    
            $SourceDelta | Sort-Object TargetItemURL | ForEach-Object {
                #Calculate Target Folder URL for the file
                $TargetFolderURL = (Split-Path $_.TargetItemURL -Parent).Replace("\","/")
                $ItemName = Split-Path $_.FullName -leaf
                #Replace Invalid Characters
                $ItemName = [RegEx]::Replace($ItemName, "[{0}]" -f ([RegEx]::Escape([String]'\"*:<>?/\|')), '_')

                #Display Progress bar
                $Status  = "Importing '" + $ItemName + "' to " + $TargetFolderURL +" ($($Counter) of $($SourceDeltaCount))"
                Write-Progress -Activity "Importing Files from the Source..." -Status $Status -PercentComplete (($Counter / $SourceDeltaCount) * 100)

                If($_.PSIsContainer)
                {
                    #Ensure Folder
                    $Folder  = Resolve-PnPFolder -SiteRelativePath ($TargetFolderURL+"/"+$ItemName) -Includes ListItemAllFields
                    
                    Set-PnPListItem -List $TargetLibraryName -Identity $Folder.ListItemAllFields.Id -Values @{"Modified"=  ([DateTime]$_.LastUpdated).ToLocalTime()} | Out-null
                    Write-host "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured Folder '$($ItemName)' to Folder $TargetFolderURL"
                }
                Else
                {
                    #Upload File
                    $File  = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL -Values @{"Modified"=  ([DateTime]$_.LastUpdated).ToLocalTime()}
                    Write-host "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                    Add-content $Logfile -value "Ensured File '$($_.FullName)' to Folder $TargetFolderURL"
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no new Items in the Source! Total Items in Source: $SourceItemsCount , Number Items in Target: $($Library.Itemcount)" 
            Add-content $Logfile -value "Found no new Items in the Source! Items in Source: $SourceItemsCount ,  Number Items in Target: $($Library.Itemcount)" 
        }      
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Import File Share Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

#Function to Remove Files Delta in SharePoint Online (Files that are no longer exists in FileShare )
Function Remove-FileShareDeltaInSPO
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        Add-content $Logfile -value "`n---------------------- Remove FileShare Delta Script Started: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-------------------"   

        #Get Number of Source Items from the Source Folder
        $SourceItemsCount =  (Get-ChildItem -Path $SourceFolderPath -Recurse).count

        #Get the Target Library
        $Web = Get-PnPWeb
        $Library = Get-PnPList $TargetLibraryName -Includes RootFolder
        $TargetFolder = $Library.RootFolder

        #Get the site relative path of the target Folder
        If($web.ServerRelativeURL -eq "/")
        {
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeUrl
        }
        Else
        {        
            $TargetFolderSiteRelativeURL = $TargetFolder.ServerRelativeURL.Replace($Web.ServerRelativeUrl,"") 
        }        
 
        #Get All Items from the Source
        $SourceItems = Get-ChildItem -Path $SourceFolderPath -Recurse
        $Source = @($SourceItems | Select FullName,  PSIsContainer,
                                     @{Label='TargetItemURL';Expression={$_.FullName.Replace($SourceFolderPath,$TargetFolderSiteRelativeURL).Replace("\","/")}}, 
                                            @{Label='LastUpdated';Expression={$_.LastWriteTimeUtc.ToString()}})

        #Get All Files from the target document library - In batches of 2000
        $TargetFiles = Get-PnPListItem -List $TargetLibraryName -PageSize 2000
        $Target = @($TargetFiles | Select @{Label='FullName';Expression={$_.FieldValues.FileRef.Replace($TargetFolder.ServerRelativeURL,$SourceFolderPath).Replace("/","\")}},
                                                @{Label='PSIsContainer';Expression={$_.FileSystemObjectType -eq "Folder"}},
                                                    @{Label='TargetItemURL';Expression={$_.FieldValues.FileRef.Replace($Web.ServerRelativeUrl,"")}},
                                                        @{Label='LastUpdated';Expression={$_.FieldValues.Modified.ToUniversalTime().ToString()}})

        #Compare Source and Target and remove files which are not in the Source
        $Counter = 1
        $FilesDiff = Compare-Object -ReferenceObject $Source -DifferenceObject $Target -Property FullName, PSIsContainer, TargetItemURL, LastUpdated     
        $TargetDelta = @($FilesDiff | Where {$_.SideIndicator -eq "=>"})
        $TargetDeltaCount = $TargetDelta.Count

        #Check if Target Files Needs to be deleted
        If($TargetDeltaCount -gt 0)
        {
            Write-host "Found $TargetDeltaCount differences in the Target!" 
            Add-content $Logfile -value "Found $TargetDeltaCount differences in the Target!"   
    
            $TargetDelta | Sort-Object TargetItemURL -Descending | ForEach-Object {
                #Display Progress bar
                $Status  = "Removing Item " + $_.TargetItemURL +" ($($Counter) of $($TargetDeltaCount))"
                Write-Progress -Activity "Removing Items in the Target..." -Status $Status -PercentComplete (($Counter / $TargetDeltaCount) * 100)

                If($_.PSIsContainer)
                {
                    #Empty and Remove the Folder
                    $Folder  = Get-PnPFolder -Url $_.TargetItemURL -ErrorAction SilentlyContinue
                    If($Folder -ne $Null)
                    {
                        $Folder.Recycle() | Out-Null
                        Invoke-PnPQuery

                        Write-host "Removed Folder '$($_.TargetItemURL)'"
                        Add-content $Logfile -value "Removed Folder '$($_.TargetItemURL)'"
                    }
                }
                Else
                {
                    $File = Get-PnPFile -Url $_.TargetItemURL -ErrorAction SilentlyContinue
                    If($File -ne $Null)
                    {
                        #Remove the File
                        Remove-PnPFile -SiteRelativeUrl $_.TargetItemURL -Force
                        Write-host "Removed File '$($_.TargetItemURL)'"
                        Add-content $Logfile -value "Removed File '$($_.TargetItemURL)'"
                    }
                }
                $Counter++
            }
        }
        Else
        {
            Write-host "Found no diferences in the Target! Total Items in Source: $SourceItemsCount , Number Items in Target: $($Library.Itemcount)" 
            Add-content $Logfile -value "Found no differences in the Target! Items in Source: $SourceItemsCount ,  Number Items in Target: $($Library.Itemcount)" 
        }
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message 
        Add-content $Logfile -value "Error:$($_.Exception.Message)"
    }
    Finally {
       Add-content $Logfile -value "---------------------- Remove FileShare Delta Script Completed: $(Get-date -format 'dd/MM/yyy hh:mm:ss tt')-----------------"
    }
}

Function Sync-FileShareToSPO()
{
 param
    (
        [Parameter(Mandatory=$true)] [string] $SiteURL,
        [Parameter(Mandatory=$true)] [string] $SourceFolderPath,
        [Parameter(Mandatory=$true)] [string] $TargetLibraryName,            
        [Parameter(Mandatory=$true)] [string] $LogFile
    )

    Try {
        #Connect to PnP Online
        Connect-PnPOnline -Url $SiteURL -UseWebLogin

        #Call the function to Import New Files from Fileshare to SPO
        Import-FileShareToSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName -LogFile $LogFile

        #Call the function to Remove Files in SPO that are moved/deleted in Fileshare
        Remove-FileShareDeltaInSPO -SiteURL $SiteURL -SourceFolderPath $SourceFolderPath -TargetLibraryName $TargetLibraryName -LogFile $LogFile
    }
    Catch {
        Write-host -f Red "Error:" $_.Exception.Message
    }
    Finally {
       DisConnect-PnPOnline
    }
}

#Call the Function to Sync Files from Fileshare to SharePoint Online
Sync-FileShareToSPO -SiteURL "https://crescent.sharepoint.com/sites/Projects" -SourceFolderPath "\\FileServer\Reports" `
                            -TargetLibraryName "Documents" -LogFile "C:\Temp\SyncFileShare-LOG.log"
This script checks the timestamp of each file from the given fileshare and uploads it to SharePoint Online if they are updated in File share. Similarly, any file deleted in Fileshare will be deleted in SharePoint Online as well. It generates a log file also every time it runs. Please note, this script does only one-way sync from Fileshare to SharePoint Online. You can Schedule PowerShell script in Windows Task scheduler , so that it runs on schedule. E.g. for every 10 min.

You can also use this script to sync files from local drive to SharePoint by changing the SourceFolderPath parameter to any folder in your computer. E.g. "C:\Temp\Upload"

5 comments:

  1. Hi There, Thanks for putting effort into making these scripts available!
    I'm running into issues with larger files.... seems like there is a 250 MB limit?
    _Add-PnPFile : The underlying connection was closed: An unexpected error occurred on a send.
    https://github.com/pnp/PnP-PowerShell/issues/936

    Do you know of a way around this?

    ReplyDelete
  2. Hi there, I appreciate your work and have used some of your scripts for my batch jobs.

    I am based at India and this script has two issues which I would like to bring to your attention.

    In getting list section: Comparing dates with UTC format does not work. I got a workaround though as @{Label='LastUpdated';Expression={$_.LastWriteTimeUTC.ToString('MM-dd-yyyy HH:mm', [System.Globalization.CultureInfo]::InvariantCulture)}}) and @{Label='LastUpdated';Expression={$_.FieldValues.Modified.ToUniversalTime().ToString('MM-dd-yyyy HH:mm', [System.Globalization.CultureInfo]::InvariantCulture)}}).

    The script uploads files after this modification, still there is second issue which I am trying to figure out.
    After uploading files and folder, the "Compare Source and Target and remove files which are not in the Source" section starts and removes all files and folders it just uploaded.

    Maybe it's related to same date comparisons. I will let you know.

    Please keep up the good work.

    and

    ReplyDelete
    Replies
    1. Hello
      first - thanks for the good work Salaudeen
      I also have a problem with the date-format as i'm based in germany:

      Error: Cannot convert value "17.02.2021 08:44:11" to type "System.DateTime". Error: "String was not recognized as a valid DateTime."


      did you already found an answer for this?

      thanks to all!

      Delete
  3. Ensured File '\\millnl\storage\cad\logfiles\wijziging.txt' to Folder /Shared Documents
    Add-PnPFile : Server relative urls must start with SPWeb.ServerRelativeUrl
    At D:\_Mill\scripts\Sync-FileShareToSPO.ps1:84 char:30
    + ... $File = Add-PnPFile -Path $_.FullName -Folder $TargetFolderURL -V ...
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo : WriteError: (:) [Add-PnPFile], ServerException
    + FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Files.AddFile

    ReplyDelete
  4. Same Error as above. Server relative urls must start with SPWeb.ServerRelativeUrl

    ReplyDelete

Please Login and comment to get your questions answered!

Powered by Blogger.