Copying files from multiple SharePoint sites onto Azure Data Lake (Gen 2) using Azure Logic Apps and Azure Data Factory
An interesting area I was reading up on recently was to Ingest data from a SharePoint folder to Azure Data Lake (Gen 2). There were quite a few interesting articles on this topic, some of which are listed below, Source: MSSQL tips https://www.mssqltips.com/sqlservertip/5893/transfer-files-from-sharepoint-to-blob-storage-with-azure-logic-apps/ Source: The Tech Platform https://www.thetechplatform.com/post/copy-files-from-sharepoint-to-an-azure-blob-storage-using-microsoft-flow Like these articles suggests, I too looked at the approach of using Logic Apps to do the same. However, I had multiple SharePoint sites and folders to ingest data from. The process was too tedious to be done manually and hence a solution through which this could be semi-automated was required. The solution is broken into 2 parts, Azure Logic App – Holds the steps to read files on a particular SharePoint folder and writes it into Azure Data Lake Storage (Gen 2). Azure Data Factory – Automates the process b