Activity
Mon
Wed
Fri
Sun
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
What is this?
Less
More
Memberships
Learn Microsoft Fabric
12.7k members โข Free
4 contributions to Learn Microsoft Fabric
๐
Jun '24 โขย
Dimensional Modeling in Fabric (new series of articles)
Microsoft has released a series of new articles on dimensional modeling in the Data Warehouse: - Dimensional modeling in Microsoft Fabric Warehouse (intro) - Dimension tables - Fact Tables - Loading Tables
0 likes โข Jun '24
@Nikhil Joy Yes, we found we couldn't have recursive CTEs in Spark Notebooks when we were on a Synapse project. However, I did come across an article which allows for it in a PySpark cell. I am sorry that I don't have the link for it any more.
May '24 โขย
Hello
Hi, I am starting to evaluate Fabric as a potential upgrade to a reporting solution currently built in Synapse. I will have to do some rework anyway as Microsoft have dropped "Export to Datalake" in D365 F&O in favour of Synapse Link. I created a traditional data warehouse solution in Synapse, building dimension and fact tables in a medallion architecture. I also have Dev/Test/Prod environments, all parameterised so that I can move notebooks/pipelines using release pipelines in DevOps. All working reasonably well. In my evaluation of Fabric, I can get individual notebooks to run, but parameterising them seems to be a lot harder. I would see that medallion is using a couple of Lakehouses and then a Warehouse as the Gold layer. I would want to be able to transform from one Lakehouse to the next and then to the Warehouse, but connecting to each doesn't seem to be as easy as I might like. I know I am missing something. I have watched quite a few YouTube videos, but there isn't much coverage yet of that area. Does anyone know of any videos covering this kind of thing?
1 like โข May '24
Hi @Liam Shropshire-Fanning , many thanks for your thoughts. You've hit the same as I have. Getting data from one Lakehouse to another seems to be challenging in SQL. Mounting in PySpark seems to be able to do the job, but the upsert type approach then seems to be more tricky across lakehouses. I do think I can parameterise the Bronze/Silver/Gold Lakehouses and pass in from pipelines; I only do something fairly simple there and just create variables based on the Dev/Test/Prod rather than try to pass in via Release Pipelines. It's worked well enough in Synapse even though it is a bit crude. I must admit I hadn't thought of putting the lakehouse names in a utility file called by a %run. I'll give that some thought to see how that could work. Thanks
0 likes โข May '24
@Liam Shropshire-Fanning That is definitely a good thought. I hadn't considered chaining lots of ETL notebooks and running in series; it will save lots of spin up time, particularly for the "cheap" notebooks. I will have a try with putting my cheap notebooks in series as they will likely take less time to run anyway and then put the more transient notebooks in parallel via pipelines as I do now I was aware of the concurrency limit in Fabric. One of the big things I like in Fabric is the better spin up times which is my biggest performance hit and it would be great if Synapse allowed the notebooks to share the pools properly. Many thanks Graham
๐
May '24 โขย
What are your biggest pain points currently with Fabric?
Hey everyone, happy Monday! I'm currently planning out future content for the YouTube channel, and want to always produce the content that is most relevant/helpful to you! So, some questions: - What are your biggest pain points currently with Fabric? - Anything you're struggling to understand? - Things you think are important, but don't quite grasp yet? It's this kind of engagement that led to the Power BI -> Fabric series, and then the DP-600 series, so I thought I'd open it up to you again! Look forward to hearing from you - thank you! Here's some potential things: - Delta file format - integrating AI/ ML/ Azure AI / Open AI - copilot - Git integration / deployment pipelines / CI/CD - data modelling / implementing SCDs - medallion implementation - more advanced pyspark stuff - data pipelines - metadata driven workflows - dataflows (and optimising dataflows) - lakehouse achitectures - real-time - data science projects - semantic link - migrating semantic models - using python to manage semantic models - administration/ automation - fabric api - other...?
4 likes โข May '24
Transform data from DataLake to DataLake (eg Bronze to Silver) using SQL rather than PySpark. Parameterise Lake URLs to allow CI/CD Dev/Test/Prod Thank you
๐
May '24 โขย
๐ New joiner? Welcome! Start here ๐
Welcome to all new members, here's some links and information to help you get started! ๐ค๐๐ถ๐ฐ๐ธ ๐๐ถ๐ป๐ธ๐ ๐๐ผ ๐ด๐ฒ๐ ๐๐๐ฎ๐ฟ๐๐ฒ๐ฑ - For an introduction to this community โ Explore the Welcome Pack - New-ish to Fabric? โ Check out our Fabric Foundation module - Studying for the DP-600? โ Check out the DP-600 Module and the DP-600 category - Studying for the DP-700? โ Check out the DP-700 Module and the DP-700 category - Want to get hands-on? โ Check out Fabric Dojo ๐๐ผ๐ ๐๐ผ ๐ฒ๐ป๐ด๐ฎ๐ด๐ฒ ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐ฐ๐ผ๐บ๐บ๐๐ป๐ถ๐๐? - Share your knowledge and experience! Even if you're relatively new to Fabric, or the community, your opinion and experiences are valued here! A great way to earn your first point(s) is to introduce yourself in the thread below ๐๐ Thank you for engaging and joining us on this exciting learning journey! ๐ Will
9 likes โข May '24
Hi, I have been building databases and applications for 20plus years, primarily in a business planning space, but branching out to other things since. Nowadays, I am involved in data migration and business intelligence for D365 F&O. I am looking into Fabric as an upgrade or compliment to a Synapse platform.
1-4 of 4
Active 459d ago
Joined May 20, 2024
Cliffe Woods, Kent
Powered by