SharePoint Online: Upload Large Files using PowerShell
Requirement: Upload large files to SharePoint Online using PowerShell.
PowerShell to Upload a large file to SharePoint Online
Suppose you try to upload files with > 250 MB file size to SharePoint Online using CSOM Files.Add method, or PnP PowerShell Add-PnPFile methods you’ll end up in errors:
- “The remote server returned an error: (503) Server Unavailable.”
- “Add-PnPFile : The request message is too big. The server does not allow messages larger than 262144000 bytes”
- Add-PnPFile: The underlying connection was closed: An unexpected error occurred on a send
- Add-PnPFile : The remote server returned an error: (404) Not Found
To mitigate the errors Here is how to upload a larger file to SharePoint Online using PowerShell:
#Load SharePoint CSOM Assemblies
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
#Parameters
$SiteURL = "https://crescent.sharepoint.com/sites/marketing"
$TargetFolderURL = "/sites/marketing/Shared Documents"
$FilePath = "C:\Users\Thomas\Downloads\WFx64.zip"
#Get Credentials to connect
$Cred = Get-Credential
Try {
#Setup the context
$Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL)
$Ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Cred.UserName,$Cred.Password)
#Get the file from disk
$FileStream = ([System.IO.FileInfo] (Get-Item $FilePath)).OpenRead()
#Calculate Target URL
$SourceFileName = Split-path $FilePath -leaf
$TargetFileURL = $TargetFolderURL+"/"+$SourceFileName
#Upload Large File to SharePoint Online
$Ctx.RequestTimeout = [System.Threading.Timeout]::Infinite
[Microsoft.SharePoint.Client.File]::SaveBinaryDirect($Ctx, $TargetFileURL, $FileStream,$True)
Write-host "File uploaded Successfully!" -f Green
}
Catch {
write-host "Error Uploading File: $($_.Exception.Message)" -foregroundcolor Red
}
However, when you upload a larger file from a slow network, it timeouts if the upload operation takes more than 30 minutes.
- 250 GB – File upload limit. Applies to each individual file
- 250 MB – File attached to a list item.
PowerShell to Upload Large Files in Chunks to SharePoint Online:
Instead of uploading a larger file in one single stretch, we can split and upload it in smaller chunks.
#Function to Upload Large File to SharePoint Online Library
Function Upload-LargeFile($FilePath, $LibraryName, $FileChunkSize=10)
{
Try {
#Get File Name
$FileName = [System.IO.Path]::GetFileName($FilePath)
$UploadId = [GUID]::NewGuid()
#Get the folder to upload
$Library = $Ctx.Web.Lists.GetByTitle($LibraryName)
$Ctx.Load($Library)
$Ctx.Load($Library.RootFolder)
$Ctx.ExecuteQuery()
$BlockSize = $FileChunkSize * 1024 * 1024
$FileSize = (Get-Item $FilePath).length
If($FileSize -le $BlockSize)
{
#Regular upload
$FileStream = New-Object IO.FileStream($FilePath,[System.IO.FileMode]::Open)
$FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$FileCreationInfo.Overwrite = $true
$FileCreationInfo.ContentStream = $FileStream
$FileCreationInfo.URL = $FileName
$Upload = $Docs.RootFolder.Files.Add($FileCreationInfo)
$ctx.Load($Upload)
$ctx.ExecuteQuery()
}
Else
{
#Large File Upload in Chunks
$ServerRelativeUrlOfRootFolder = $Library.RootFolder.ServerRelativeUrl
[Microsoft.SharePoint.Client.File]$Upload
$BytesUploaded = $null
$Filestream = $null
$Filestream = [System.IO.File]::Open($FilePath, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite)
$BinaryReader = New-Object System.IO.BinaryReader($Filestream)
$Buffer = New-Object System.Byte[]($BlockSize)
$LastBuffer = $null
$Fileoffset = 0
$TotalBytesRead = 0
$BytesRead
$First = $True
$Last = $False
#Read data from the file in blocks
While(($BytesRead = $BinaryReader.Read($Buffer, 0, $Buffer.Length)) -gt 0)
{
$TotalBytesRead = $TotalBytesRead + $BytesRead
If ($TotalBytesRead -eq $FileSize)
{
$Last = $True
$LastBuffer = New-Object System.Byte[]($BytesRead)
[Array]::Copy($Buffer, 0, $LastBuffer, 0, $BytesRead)
}
If($First)
{
#Create the File in Target
$ContentStream = New-Object System.IO.MemoryStream
$FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation
$FileCreationInfo.ContentStream = $ContentStream
$FileCreationInfo.Url = $FileName
$FileCreationInfo.Overwrite = $true
$Upload = $Library.RootFolder.Files.Add($FileCreationInfo)
$Ctx.Load($Upload)
#Start FIle upload by uploading the first slice
$s = new-object System.IO.MemoryStream(, $Buffer)
$BytesUploaded = $Upload.StartUpload($UploadId, $s)
$Ctx.ExecuteQuery()
$fileoffset = $BytesUploaded.Value
$First = $False
}
Else
{
#Get the File Reference
$Upload = $ctx.Web.GetFileByServerRelativeUrl($Library.RootFolder.ServerRelativeUrl + [System.IO.Path]::AltDirectorySeparatorChar + $FileName);
If($Last)
{
$s = [System.IO.MemoryStream]::new($LastBuffer)
$Upload = $Upload.FinishUpload($UploadId, $fileoffset, $s)
$Ctx.ExecuteQuery()
Write-Host "File Upload completed!" -f Green
}
Else
{
#Update fileoffset for the next slice
$s = [System.IO.MemoryStream]::new($buffer)
$BytesUploaded = $Upload.ContinueUpload($UploadId, $fileoffset, $s)
$Ctx.ExecuteQuery()
$fileoffset = $BytesUploaded.Value
}
}
}
}
}
Catch {
Write-Host $_.Exception.Message -ForegroundColor Red
}
Finally {
If($Filestream -ne $null)
{
$Filestream.Dispose()
}
}
}
#Connect to SharePoint Online site
Connect-PnPOnline "https://crescent.sharepoint.com/sites/marketing" -Interactive
$Ctx = Get-PnPContext
#Call the function to Upload File
Upload-LargeFile -FilePath "C:\Users\Thomas\Downloads\235021WFx64.rar" -LibraryName "Documents"
BTW, I’ve converted the method into PowerShell, described in https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/upload-large-files-sample-app-for-sharepoint
Looking for a 2-way sync between multiple remote on-premise locations, and 2-way sync from one location to Sharepoint (O365)
This script works for me only if the file is uploaded to the root “Documents” folder.
I can’t make it work for a subfolder.
Can you help me?
Yes! That’s how its coded. Refer How to Upload a File to a SubFolder in SharePoint Online? to upload a File to a Folder.
It worked but I don’t know why sometime it throws Number of Items Found in the Source:
Error:Attempted to divide by zero.
Is it because it can’t recognize the file extension? Thank
Could be due to Zero-sized files?
Good afternoon. Can you help with a solution for downloading a file that is located on an external resource? We need to upload 2 backup files 7 GB and 14 GB in size. At the input we receive a link to download these files. Which of your scripts would be most suitable in our case? I am not strong in powershell, but there is no other choice.
Try:
For me, the SharePoint timeout seems to occur after 5mins and not 30. I end up with 2GB out of 10GB file uploaded, as a note, uploading the file manually (file upload via browser) only takes 15mins in total.
Anyone else experience this?
PowerShell is one of my favorite freeways to upload from my server to SharePoint Online, but I really prefer using Gs RichCopy 360 to upload directly to SharePoint Online, it has an option to limit the connection while uploading which means, no bandwidth-consuming .