Clean data isn’t always correct. Here’s how “beautiful lies” sneak into your reports. When we work with data, there’s one question that should never leave our mind: Can we trust it? It sounds obvious. But in practice, it’s one of the most neglected questions in everyday
Modern companies are held together with duct tape — and that duct tape is us. If you’re like me, you didn’t apply to be a “data person.” You weren’t trained in databases or machine learning. Yet here we are, every day, doing a job that has no name.
Sometimes faulty data has become valid data — and fixing it makes things worse. A data engineer told me a story the other day. They were doing some routine cleanup on a core database: migrations, column checks, the usual. Nothing fancy. While doing his work, he spotted a something — one column
If you've ever used a microphone too close to a speaker, you know the result: that awful, escalating screech. That’s an audio feedback loop — your voice gets amplified, cycles through the system, and spins out of control faster than it can be corrected. Now imagine the same
In many organizations, queries and formulas quietly carry logic that exists nowhere else. A user of my Tablecruncher app once came to me with a “simple” task. They needed to merge two CSV files with client data based on a few conditions. Nothing fancy — just a macro to automate the
Most CSV headaches start the moment your data stops being truly flat. CSV is the most common data format in the world. It’s simple, human-readable, and supported by almost every tool. And yet, it’s one of the most heavily misused formats out there. A CSV is built for
We’ve all seen the “10 quick tips for cleaning your data” posts. Most of them leave out the tricky parts — the places where you can quietly ruin your dataset without even noticing. A few that deserve more attention: 1. Format conversion is dangerous. Getting data into the format you
Some tools quietly earn a permanent place in your toolbox. KNIME is one of mine — a visual data workflow builder that’s saved me from endless spreadsheet wrangling and script debugging. KNIME started at the University of Konstanz in 2004 and has grown into a mature, open source platform for
We all know the best practices. Write clean code. Use libraries. Document everything. But here’s the thing: in the real world, things are messy — and I’m no exception. So today, let’s talk about my data engineering sins. Not as a confession, but as a mirror. Because chances
Even the Library of Congress agrees: boring formats win in the long run. In our daily data work, we often treat the data we get as throwaway material. We read it, interpret it, combine it with other data, and finally reach our desired result. Then we toss the intermediate data
Numbers are easy for computers — that’s what we’ve been told. But try storing 9007199254740993 in Excel and you’ll get… a different number. What’s going on? The invisible conversion When I wrote the first version of my CSV editor Tablecruncher, one of my non-negotiables was: Don'
How do you know if two names refer to the same person? That’s the question behind a new EU regulation that will soon require European banks to warn customers during money transfers if the payee’s name doesn’t match the account holder. It’s meant to prevent fraud