A new data format that could replace CSV and JSON. Two weeks ago, we talked about how something as simple as an empty field in a CSV can carry very different meanings — and how those differences quietly affect everything downstream. Right on cue, a new data format appeared in the
An empty field isn’t NULL. And that difference can quietly break your logic. The other day, I was preparing a CSV for a database import. The schema had two name fields — name_local and name_english. Downstream systems were supposed to use the local name if available, otherwise fall
MySQL’s hidden cutoff decides what’s visible — and what isn’t. The other day I tried to find a souvenir online. I typed I love NY into the shop’s search field — and got a vast load of results for all kind of products with "love" printed
Clean data isn’t always correct. Here’s how “beautiful lies” sneak into your reports. When we work with data, there’s one question that should never leave our mind: Can we trust it? It sounds obvious. But in practice, it’s one of the most neglected questions in everyday
Modern companies are held together with duct tape — and that duct tape is us. If you’re like me, you didn’t apply to be a “data person.” You weren’t trained in databases or machine learning. Yet here we are, every day, doing a job that has no name.
Sometimes faulty data has become valid data — and fixing it makes things worse. A data engineer told me a story the other day. They were doing some routine cleanup on a core database: migrations, column checks, the usual. Nothing fancy. While doing his work, he spotted a something — one column
If you've ever used a microphone too close to a speaker, you know the result: that awful, escalating screech. That’s an audio feedback loop — your voice gets amplified, cycles through the system, and spins out of control faster than it can be corrected. Now imagine the same
In many organizations, queries and formulas quietly carry logic that exists nowhere else. A user of my Tablecruncher app once came to me with a “simple” task. They needed to merge two CSV files with client data based on a few conditions. Nothing fancy — just a macro to automate the
Most CSV headaches start the moment your data stops being truly flat. CSV is the most common data format in the world. It’s simple, human-readable, and supported by almost every tool. And yet, it’s one of the most heavily misused formats out there. A CSV is built for
We’ve all seen the “10 quick tips for cleaning your data” posts. Most of them leave out the tricky parts — the places where you can quietly ruin your dataset without even noticing. A few that deserve more attention: 1. Format conversion is dangerous. Getting data into the format you
Some tools quietly earn a permanent place in your toolbox. KNIME is one of mine — a visual data workflow builder that’s saved me from endless spreadsheet wrangling and script debugging. KNIME started at the University of Konstanz in 2004 and has grown into a mature, open source platform for
We all know the best practices. Write clean code. Use libraries. Document everything. But here’s the thing: in the real world, things are messy — and I’m no exception. So today, let’s talk about my data engineering sins. Not as a confession, but as a mirror. Because chances