SharePoint Online: Upload Large Files using PowerShell
Requirement: Upload large files to SharePoint Online using PowerShell
PowerShell to Upload large file to SharePoint Online
If you try to upload files with > 250 MB file size to SharePoint Online using CSOM Files.Add method, or PnP PowerShell Add-PnPFile methods you'll end-up in errors:
- "The remote server returned an error: (503) Server Unavailable."
- "Add-PnPFile : The request message is too big. The server does not allow messages larger than 262144000 bytes"
- Add-PnPFile: The underlying connection was closed: An unexpected error occurred on a send
- Add-PnPFile : The remote server returned an error: (404) Not Found
To mitigate the errors, Here is how to upload a larger file to SharePoint Online using PowerShell:
#Load SharePoint CSOM Assemblies Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll" Add-Type -Path "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll" #Parameters $SiteURL = "https://crescent.sharepoint.com/sites/marketing" $TargetFolderURL = "/sites/marketing/Shared Documents" $FilePath = "C:\Users\Thomas\Downloads\WFx64.zip" #Get Credentials to connect $Cred = Get-Credential Try { #Setup the context $Ctx = New-Object Microsoft.SharePoint.Client.ClientContext($SiteURL) $Ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($Cred.UserName,$Cred.Password) #Get the file from disk $FileStream = ([System.IO.FileInfo] (Get-Item $FilePath)).OpenRead() #Calculate Target URL $SourceFileName = Split-path $FilePath -leaf $TargetFileURL = $TargetFolderURL+"/"+$SourceFileName #Upload Large File to SharePoint Online $Ctx.RequestTimeout = [System.Threading.Timeout]::Infinite [Microsoft.SharePoint.Client.File]::SaveBinaryDirect($Ctx, $TargetFileURL, $FileStream,$True) Write-host "File uploaded Successfully!" -f Green } Catch { write-host "Error Uploading File: $($_.Exception.Message)" -foregroundcolor Red }
However, when you upload a larger file from slow network, it timeouts if the upload operation takes more than 30 minutes.
PowerShell to Upload Large Files in Chunks to SharePoint Online:
Instead of uploading a larger file in one single stretch, we can split and upload in smaller chunks.
#Function to Upload Large File to SharePoint Online Library Function Upload-LargeFile($FilePath, $LibraryName, $FileChunkSize=10) { Try { #Get File Name $FileName = [System.IO.Path]::GetFileName($FilePath) $UploadId = [GUID]::NewGuid() #Get the folder to upload $Library = $Ctx.Web.Lists.GetByTitle($LibraryName) $Ctx.Load($Library) $Ctx.Load($Library.RootFolder) $Ctx.ExecuteQuery() $BlockSize = $FileChunkSize * 1024 * 1024 $FileSize = (Get-Item $FilePath).length If($FileSize -le $BlockSize) { #Regular upload $FileStream = New-Object IO.FileStream($FilePath,[System.IO.FileMode]::Open) $FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation $FileCreationInfo.Overwrite = $true $FileCreationInfo.ContentStream = $FileStream $FileCreationInfo.URL = $FileName $Upload = $Docs.RootFolder.Files.Add($FileCreationInfo) $ctx.Load($Upload) $ctx.ExecuteQuery() } Else { #Large File Upload in Chunks $ServerRelativeUrlOfRootFolder = $Library.RootFolder.ServerRelativeUrl [Microsoft.SharePoint.Client.File]$Upload $BytesUploaded = $null $Filestream = $null $Filestream = [System.IO.File]::Open($FilePath, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Read, [System.IO.FileShare]::ReadWrite) $BinaryReader = New-Object System.IO.BinaryReader($Filestream) $Buffer = New-Object System.Byte[]($BlockSize) $LastBuffer = $null $Fileoffset = 0 $TotalBytesRead = 0 $BytesRead $First = $True $Last = $False #Read data from the file in blocks While(($BytesRead = $BinaryReader.Read($Buffer, 0, $Buffer.Length)) -gt 0) { $TotalBytesRead = $TotalBytesRead + $BytesRead If ($TotalBytesRead -eq $FileSize) { $Last = $True $LastBuffer = New-Object System.Byte[]($BytesRead) [Array]::Copy($Buffer, 0, $LastBuffer, 0, $BytesRead) } If($First) { #Create the File in Target $ContentStream = New-Object System.IO.MemoryStream $FileCreationInfo = New-Object Microsoft.SharePoint.Client.FileCreationInformation $FileCreationInfo.ContentStream = $ContentStream $FileCreationInfo.Url = $FileName $FileCreationInfo.Overwrite = $true $Upload = $Library.RootFolder.Files.Add($FileCreationInfo) $Ctx.Load($Upload) #Start FIle upload by uploading the first slice $s = new-object System.IO.MemoryStream(, $Buffer) $BytesUploaded = $Upload.StartUpload($UploadId, $s) $Ctx.ExecuteQuery() $fileoffset = $BytesUploaded.Value $First = $False } Else { #Get the File Reference $Upload = $ctx.Web.GetFileByServerRelativeUrl($Library.RootFolder.ServerRelativeUrl + [System.IO.Path]::AltDirectorySeparatorChar + $FileName); If($Last) { $s = [System.IO.MemoryStream]::new($LastBuffer) $Upload = $Upload.FinishUpload($UploadId, $fileoffset, $s) $Ctx.ExecuteQuery() Write-Host "File Upload completed!" -f Green } Else { #Update fileoffset for the next slice $s = [System.IO.MemoryStream]::new($buffer) $BytesUploaded = $Upload.ContinueUpload($UploadId, $fileoffset, $s) $Ctx.ExecuteQuery() $fileoffset = $BytesUploaded.Value } } } } } Catch { Write-Host $_.Exception.Message -ForegroundColor Red } Finally { If($Filestream -ne $null) { $Filestream.Dispose() } } } #Connect to SharePoint Online site Connect-PnPOnline "https://crescent.sharepoint.com/sites/marketing" -UseWebLogin $Ctx = Get-PnPContext #Call the function to Upload File Upload-LargeFile -FilePath "C:\Users\Thomas\Downloads\235021WFx64.rar" -LibraryName "Documents"
BTW, I've converted the method into PowerShell, described in https://docs.microsoft.com/en-us/sharepoint/dev/solution-guidance/upload-large-files-sample-app-for-sharepoint
No comments:
Please Login and comment to get your questions answered!