Purview is pretty neat, when I played around with it a while ago it seemed to definitely be lacking though (though I will say I'm definitely not super familiar with it). I saw the potential for sure though when it came to linking Fabric to it!

@Will Needham thanks for the theory! I definitely think you are spot on, and gave way more insight that I would have ever gotten from trying to figure it out myself haha. Totally makes more sense as to why some of these things stay in preview for so long. I have been using the lakehouse schemas for a while and haven't had too much issue in how I've been using them. I don't really use DFG2 much but decided to give it a whirl last week for a simple data cleansing and load from a few excel files instead of doing a whole spark notebook, but the lack of a schema destination was a bit off-putting. I do go back on the pipeline notifications as the major one for me personally. I haven't looked at the new outlook or teams activities for options, but they sure were a pain before, hence the Power Automate solution. It definitely is easier to have all the activities on one canvas though that's for sure!

@Will Needham look forward to what you find out and post!

I personally do not like Outlook activity in a pipeline since copying and moving it to another pipeline always gave me issues. I’ve found that creating a trigger and having it trigger a Power Automate flow and passing those specific pipeline parameters is best, although I think it might be limited to the event stream schema. I know for mine I send the pipeline url, report url (I have a Power BI report with specifics about each table and the error messages), and a database url (figured I might as well practice some KQL too). It’s not super hard to setup, but it definitely is more to stitch together using multiple services. I’d say it takes about 10 minutes for the email to send to me via Outlook from the Power Automate flow after a failure. I’m not sure if this could be sped up by altering the event stream (didn’t really check since my pipelines mostly run early AM). Don’t know if that helps or gives some ideas but thought I would share. More than happy to help out more if I can!

@Arlette Slachmuylder 100% agree with you. It’s out of preview now I think though. I don’t know if that means any changes though at the moment to both the Outlook or Teams activities.

Love the evaluate expression feature in the pipelines!

@Abhay Verma appreciate the summary!

If you're passing from pipeline to notebook you would need to set base parameters for the notebook activity in the pipeline. The code is a bit confusing though. You want to run a query from somewhere and then pass that into Spark?