Aug 18 • Technical
Delta Incremental load with Pyspark
Hi all,
I’m writing Delta tables in a Spark notebook, partitioned by a date column.
Usually I do a full overwrite, but I’m thinking of switching to:
.option("partitionOverwriteMode", "dynamic")
Has anyone tested this option in Fabric?
I’d be curious to hear your feedback or gotchas.
Thanks!
3
2 comments
Delta Incremental load with Pyspark
skool.com/microsoft-fabric
Helping passionate analysts, data engineers, data scientists (& more) to advance their careers on the Microsoft Fabric platform.
Leaderboard (30-day)
1
+27
2
+23
3
+23
4
+16
5
+15
Powered by