Marcel-Jan Krijgsman
Marcel-Jan Krijgsman is a senior data engineer with 27 years experience in data. Learning Python when switching his career to data engineering 9 years ago, these days it's part of his daily life. Whether he retrieves data from his weighing scale to store in a database, or updating his cycling dashboard on an e-ink display.
Session
It's 2026 and all of a sudden your regular solution to build a data pipeline in Azure or AWS seems to have gone out of favour pretty fast. We want to store our data outside of the reach of the next autocrat.
But what are the alternatives? In this session we'll discuss how you can build a data lakehouse in the European cloud. And we want more: we want PySpark, notebooks and data visualization. Is that all possible?
For this data lakehouse solution we start with Kubernetes and object storage. You'll be surprised how many European cloud providers offer these products. Now we'll use Nessie as catalog and Trino as query engine. With Iceberg, our open table format, we can already create our first table.
Next we'll run Jupyter Hub for shared notebooks. And now we can cooperate on writing PySpark code.
Great, but can we still use (local) PowerBI for data visualisation?
Yes, but that actually turns out a bit harder and for some reason expensive. We'll look at the alternatives for that.
This presentation is also suitable for visitors who are not data engineers or who have little knowledge of Kubernetes.