Activity
Mon
Wed
Fri
Sun
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
What is this?
Less
More

Memberships

The Energy Data Scientist

513 members • Free

1 contribution to The Energy Data Scientist
New Online Course: How Big data from Smart Meters are processed efficiently
I’ve just published a new online course about Memory-Efficient Processing of Big Data. This course teaches real-world skills as they are used in practice. Smart meters measure the electricity-consumption data every hour, and store the information in CSV files. These files eventually become very large (big data). The new online course is called "Smart Meter Big Data Efficient Processing" and it is in the Classroom in 1.36. This online course teaches a Python methodology that is used by energy companies in practice to read extremely large datasets (Big Data). Without this technique such files cannot be read because they cause a memory (RAM) error. Companies that sell electricity to consumers are known as 'Retailers' or 'Suppliers'. Such companies have CSV files with hundreds of millions of rows, where each row is the hourly kWh electricity consumption. If they try to load these CSV files, their computers will run out of RAM and crash. So these companies process these files using Python iterators, which enable a memory-efficient and fast processing method. In this course, I show you the industry-standard solution: using Python Iterators to process Big Data in "chunks". See the attached image; this is analysed in detail in the course.
New Online Course: How Big data from Smart Meters are processed efficiently
1 like • Jan 17
This is great, thank you!
1-1 of 1
Sakeenah Aderinto
1
4points to level up
@sakeenah-aderinto-5303
Energy Enthusiast

Active 1h ago
Joined Dec 4, 2025