fully automated with AI Agents integrated :)

@Will Needham - In my project we have a requirement to implement event driven triggers. requirement as below, 1. files are placed by client in client shared drive which is out side of fabric. 2. for every new file entry in the share drive an event should trigger the data pipeline and take that new file from the share drive (which is outside of fabric) and process through medallion architecture and load into target system. disclaimer from customer: Azure is out of scope for this project. everything should be implemented inside the fabric environment. Workarounds: 1. we have written a python code to run from fabric notebook and trying to access the file which is in share drive. but not able to read files from outside of fabric. 2. we tried SAS token option, but that has to be created in the Azure portal which is out of scope for this project 3. Azure event hub is another approach, same issue azure is out of scope. Requesting anyone from the community to provide the suitable solution for this issue. many thanks in advance.

@Will Needham - below code snippet is working to load the file into lakehouse from python. import os import requests from dotenv import load_dotenv load_dotenv('.env') # # Step 1: Load connection details from environment variables # TENANT_ID = os.getenv('fabric_tenant_id') # CLIENT_ID = os.getenv('FABRIC_CLIENT_ID') # CLIENT_SECRET = os.getenv('FABRIC_CLIENT_SECRET') # LAKEHOUSE_URL = os.getenv('FABRIC_LAKEHOUSE_URL') # DESTINATION_PATH = os.getenv('FABRIC_FILE_DESTINATION_PATH') # LOCAL_FILE_PATH = os.getenv('FABRIC_LOCAL_FILE_PATH') # print("TENANT_ID:", TENANT_ID) # print("CLIENT_ID:", CLIENT_ID) # print("CLIENT_SECRET:", CLIENT_SECRET) # print("LAKEHOUSE_URL:", LAKEHOUSE_URL) # print("DESTINATION_PATH:", DESTINATION_PATH) # print("LOCAL_FILE_PATH:", LOCAL_FILE_PATH) TENANT_ID = "ce8255f5-8eda-41ae-8a43-ee9497702f4c" CLIENT_ID = "c1023f62-70f8-47ff-8efe-74a30fe28900" CLIENT_SECRET = ".zx8Q~xXUuoxj1qEbnqgurLuG3TZ8XK5nV1EtaSe" LAKEHOUSE_URL = "https://onelake.dfs.fabric.microsoft.com/Fabric_datapipeline1609/DP002_LH.Lakehouse/Files" DESTINATION_PATH = "Dataset_Account_Master_BusinessGlossary.xlsx" LOCAL_FILE_PATH = "D:\\Metadata-copilot\\Connect_Lakehouse_Python\\my_env\\Dataset_Account_Master_BusinessGlossary.xlsx" # Step 2: Authenticate using OAuth2 def get_access_token(tenant_id, client_id, client_secret): token_url = f"https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/token" token_data = { "grant_type": "client_credentials", "client_id": CLIENT_ID, "client_secret": CLIENT_SECRET, "scope": "https://storage.azure.com/.default" } headers = { "Content-Type": "application/x-www-form-urlencoded" } response = requests.post(token_url, data=token_data, headers=headers) # print(response.json()) # Debug the token response

@Pavan Kumar Okay, let me check

@Pavan Kumar after creating the username and password, still getting the TCUP/IP network issue. No response from the server. mostly server team need to check on whitelisting the host and port.

Congratulations winners💐