Layer2 Data Provider for Office 365 Fast File Sync

The Layer2 Data Provider for Office 365 Fast File Sync can be used by the Layer2 Cloud Connector to connect to Microsoft SharePoint Online (Office 365, OneDrive for Business, Microsoft Teams, Office Groups, and similar) or SharePoint Server 2016 on-premises to access files and folders on SharePoint document libraries in a way which is optimized for fast file uploading. Basically, the provider comes with some feature limitations compared to the more general Layer2 Data Provider for SharePoint (CSOM) – but can therefore work up-to ten times faster.

It is a great choice for customers that need a very fast file sync, but don’t need any advanced features or support for older SharePoint versions. It can be used for fast file migration, backup, and sync between local file shares and SharePoint, or SharePoint to SharePoint. The provider in included with the Layer2 Cloud Connector since version 7.9.

For more detailed information about the differences, benefits and limitations of both providers please visit the comparison article page.

Layer2 Data Provider for Office 365 Fast File Sync: Specification

Connection String

A typical connection string for this provider looks like this:

Url=<SharePointSiteUrl>; Library=<LibraryName>; Authentication=<auth>

Here is the full list of connection string parameters for this provider:

Url

This is the URL of the SharePoint site-collection where the document library is located or the full URL to the view in the SharePoint web-site. When a full URL is used then the library, folder and view information is automatically retrieved from it. Otherwise the library name must be specified separately. The URL is mandatory.

Authentication

This setting specifies how the provider authenticates against the target server.

This setting is optional. If not provided, it will be Anonymous. The most commonly used method is Microsoft_Modern.

User

This part of the connection string specifies the username for the account which is used to authenticate against SharePoint. It needs to be specified for several authentication methods.

Password

This defines the password for the account which is used to authenticate. It needs to be specified for several authentication methods by typing it into the Connection String or Password field.

Library

The Library setting specifies the SharePoint document library to be used as the data entity. This could be the internal name, the display name, or the ID of the document library. This setting is mandatory when the full URL to the library is not used in the connection string.

Note: Custom lists, Calendar lists, etc., cannot be used here. Only document libraries.

View

The View setting is optional and can be used to define a specific set of sub-data of a document library to be synchronized. The view can be created and configured on the SharePoint portal as usual and then the view can be specified in the connection string. The View setting will accept the name of the view as well as the URL of the view’s .aspx-site or just the name of the view’s .aspx-site.

Optional Parameters 

The following parameters are optional and should be changed with care as misconfigurations can lead to serious performance issues (CPU as well as memory) or errors during the synchronization (for example, in case the batch sizes are too large and cannot be processed). On the other hand, environment-dependent, optimized settings for these parameters can greatly improve synchronization throughput by utilizing the host machine’s resources.

ReadingBatchSize

This setting specifies how many SharePoint items will be retrieved together from the server in one read request. By default, SharePoint allows reading up to 5000 items.

WritingBatchSize

This setting specifies how many SharePoint items will be committed together to the server in one update request. By default, the provider commits up to 100 items at once during one update request. In case this causes errors, the value can be reduced.

RequestTimeout

This setting specifies the time in milliseconds before a client-side timeout exception is raised. By default, this is set to 300000ms (5 minutes). This value may be increased in case of frequent timeouts.

UpdateMetadata

If no metadata is necessary, this can be set to ‘false’ which can speed up the upload-process. The default value is ‘true’.

ChunkSize

Large files are uploaded in chunks, as recommended by Microsoft. This setting specifies the chunk size in MB. The default value is set to 250MB.  

MaxFileSize

SharePoint has limitations about the maximum size of files. However, as this limit is not consistent among all SharePoint tenants, it is possible to change the maximum file size in MB that the Cloud Connector tries to upload with this setting. The default value is set to 2000MB (2GB) because that is the official limit for file uploads to SharePoint Online.

SmallFilesMaxSize

Small files are uploaded in batches. This setting specifies the maximum size in MB that will be sent in one batch. The default value is set to 200MB. As this upload method is the fastest, increasing this value still can speed-up the upload, but increases the risk of request size errors.

MediumFilesMaxSize

Files of medium size are uploaded in one single request. This setting specifies the maximum file size in MB that is uploaded using this method. The default value is set to 250MB, as recommended by Microsoft. As this upload method is faster than the chunked upload, increasing this value can speedup the upload, but increases the risk of request size errors.

SmallFilesThreadCount

For performance reasons, multiple files are uploaded simultaneously. This setting specifies how many small file uploads will be running at once. The default value is set to 3. Increasing this value can speed-up the upload, but leads to an increased memory consumption as each of these uploads may consume up to SmallFilesMaxSize MBs of memory.

MediumFilesThreadCount

For performance reasons, multiple files are uploaded simultaneously. This setting specifies how many medium file uploads will be running at once. The default value is set to 3. Increasing this value can speed-up the upload, but leads to an increased memory consumption as each of these uploads may consume up to MediumFilesMaxSize MBs of memory.

LargeFilesThreadCount

For performance reasons, multiple files are uploaded simultaneously. This setting specifies how many large file uploads will be running at once. The default value is set to 3. Increasing this value can speed-up the upload, but leads to an increased memory consumption as each of these uploads may consume up to ChunkSize MBs of memory.

RetryCount

In case of a transient error during a file upload, the provider will wait a few seconds and retry the upload. This parameter specifies how often an upload attempt is retried. The default value is 5.

RetryPeriod

In case of a transient error during a file upload, the provider will wait a few seconds and retry the upload. This parameter specifies how long the provider waits (in milliseconds) before attempting to retry. The default value is set to 5000ms (5 seconds).

Layer2 Data Provider for Office 365 Fast File Sync: Sample Connection Strings

• To connect to a list on SharePoint with current user access rights please use:
  Url=http://mydomain/sites/mysite/lists/myList/AllItems.aspx
• To connect to a Microsoft Office 365 list please use the following connection string:
  Url=https://mydomain.sharepoint.com/sites/mysite/lists/myList/AllItems.aspx; Authentication=Microsoft_Modern; User=myuser@mydomain.onmicrosoft.com; Password=myPwd

Mapping: writing Created and Modified timestamps

Usually SharePoint maintains the fields Created and Modified. When a list-item is created in a SharePoint-list or a file is added to a document-library, SharePoint automatically assigns the current date and time for the Created field. Likewise, whenever an entry changes SharePoint will update the Modified field.

However, if the fields are mapped, the dates of the source are written/synchronized instead.

Due to its intended internal usage, there are some things to consider when writing to those fields:

• The user, which is used to synchronize, needs to have Site-Collection administrator permissions; otherwise, Created and Modified are read-only.
• If checkout/checkin is enforced on a document-library the Modified field will always be overwritten with the current date on check-in.

Layer2 Data Provider for Office 365 Fast File Sync: Use Cases

• ​File Share to SharePoint connections: Can be used together with the Layer2 Data Provider for File System to sync any type of files or documents on a local file server, NAS or mounted Azure File Share with SharePoint libraries, such as used in SharePoint Online, SharePoint Server 2016, OneDrive for Business, Microsoft Teams, Office Groups, and others.
• SharePoint to SharePoint connections: Keep SharePoint libraries in sync, for example between online and on-premises, libraries on different sites, site collections, tenants, or installations.
• General: Can be used for migration or backup (one-way) or even two-way (permanent sync).

SharePoint Integration - Known Issues and Workarounds

• Synchronization to or from a SharePoint Asset Library is not supported.
• Versioning is not fully supported. If versioning is enabled on the document library, the provider might create multiple versions for a single update.
• SharePoint list attachments are not managed at this time and are not supported.
• For other SharePoint restrictions, see Software Boundaries and Limits for SharePoint 2013 and Blocked File Types.​​
• Editor, Author, Created, Modified, ContentType, and HtmlFileType field can only be written to by a site administrator. If you get errors that these fields are read-only and the account being used is a site administrator, it may be due to a known SharePoint issue where it denies acess if "AddAndCustomizePages" is set to Deny for the user. To fix this you must explicitly give "AddAndCustomizePage" permissions to the account. For more information, see this article.
• Switching a synchronization from using the fast file upload provider to using the generic SharePoint provider is supported. Switching the other way around, however, is not recommended.

Ready to FOR your next steps?