Documenting a CSV or JSON file sounds trivial—until you try to do it properly. Most people assume the format will somehow explain itself, but neither CSV nor JSON gives you comments, metadata, or any built-in way to describe what your fields actually mean. And that’s exactly why half
A new data format that could replace CSV and JSON. Two weeks ago, we talked about how something as simple as an empty field in a CSV can carry very different meanings — and how those differences quietly affect everything downstream. Right on cue, a new data format appeared in the
An empty field isn’t NULL. And that difference can quietly break your logic. The other day, I was preparing a CSV for a database import. The schema had two name fields — name_local and name_english. Downstream systems were supposed to use the local name if available, otherwise fall
MySQL’s hidden cutoff decides what’s visible — and what isn’t. The other day I tried to find a souvenir online. I typed I love NY into the shop’s search field — and got a vast load of results for all kind of products with "love" printed
Clean data isn’t always correct. Here’s how “beautiful lies” sneak into your reports. When we work with data, there’s one question that should never leave our mind: Can we trust it? It sounds obvious. But in practice, it’s one of the most neglected questions in everyday
Modern companies are held together with duct tape — and that duct tape is us. If you’re like me, you didn’t apply to be a “data person.” You weren’t trained in databases or machine learning. Yet here we are, every day, doing a job that has no name.
Sometimes faulty data has become valid data — and fixing it makes things worse. A data engineer told me a story the other day. They were doing some routine cleanup on a core database: migrations, column checks, the usual. Nothing fancy. While doing his work, he spotted a something — one column
If you've ever used a microphone too close to a speaker, you know the result: that awful, escalating screech. That’s an audio feedback loop — your voice gets amplified, cycles through the system, and spins out of control faster than it can be corrected. Now imagine the same
In many organizations, queries and formulas quietly carry logic that exists nowhere else. A user of my Tablecruncher app once came to me with a “simple” task. They needed to merge two CSV files with client data based on a few conditions. Nothing fancy — just a macro to automate the
Most CSV headaches start the moment your data stops being truly flat. CSV is the most common data format in the world. It’s simple, human-readable, and supported by almost every tool. And yet, it’s one of the most heavily misused formats out there. A CSV is built for
We’ve all seen the “10 quick tips for cleaning your data” posts. Most of them leave out the tricky parts — the places where you can quietly ruin your dataset without even noticing. A few that deserve more attention: 1. Format conversion is dangerous. Getting data into the format you
Some tools quietly earn a permanent place in your toolbox. KNIME is one of mine — a visual data workflow builder that’s saved me from endless spreadsheet wrangling and script debugging. KNIME started at the University of Konstanz in 2004 and has grown into a mature, open source platform for
We all know the best practices. Write clean code. Use libraries. Document everything. But here’s the thing: in the real world, things are messy — and I’m no exception. So today, let’s talk about my data engineering sins. Not as a confession, but as a mirror. Because chances
Even the Library of Congress agrees: boring formats win in the long run. In our daily data work, we often treat the data we get as throwaway material. We read it, interpret it, combine it with other data, and finally reach our desired result. Then we toss the intermediate data
Numbers are easy for computers — that’s what we’ve been told. But try storing 9007199254740993 in Excel and you’ll get… a different number. What’s going on? The invisible conversion When I wrote the first version of my CSV editor Tablecruncher, one of my non-negotiables was: Don'
How do you know if two names refer to the same person? That’s the question behind a new EU regulation that will soon require European banks to warn customers during money transfers if the payee’s name doesn’t match the account holder. It’s meant to prevent fraud
Some years ago, I consulted for a company with recurring website issues — broken features, slow performance, inconsistent analytics. After a bit of digging, the root cause became clear: they had layered so many user tracking systems into their site that it had become fragile and unreliable. When I asked why
This is issue #007 of The Missing Header Data problems don’t just happen in IT departments. Even in civil engineering, interpreting data correctly is crucial — and mistakes can cause enormous trouble. In 2003, a bridge called Hochrheinbrücke, connecting Germany and Switzerland across the river Rhine, was built. The project
This is issue #006 of The Missing Header Did you ever get an email with a sentence like: We meet at the café" You probably guessed the sender meant a coffeehouse. But where's this letter garbage coming from? This is a classic symptom of broken character encoding.
This is issue #005 of The Missing Header About a year ago, I asked the AI search engine Perplexity a simple question: “How many otters live in Europe?” The answer? 350. Now, the European otter is endangered in some regions — but not that endangered. There are far more than just
This is issue #004 ofThe Missing Header — formerly known as the Tablecruncher Newsletter* Recently, I got a CSV file with column names like state, amount, and the ever-popular date. Even after looking at the content for a while, their meaning wasn’t obvious. What kind of “date” are we talking
This is already issue #003 ofThe Missing Header — formerly known as the Tablecruncher Newsletter* Opening a CSV file in Excel feels easy. You double-click, it loads, and voilà — your data. Or so you think. But what actually happens under the hood can quietly corrupt your data without you ever noticing.
Welcome to issue #002 of The Missing Header — formerly known as the Tablecruncher Newsletter* A user emailed me, rather frustrated. They had a CSV full of customer data — and a clear goal: get rid of duplicates. They opened it in Excel, clicked Remove Duplicates, and… nothing happened. Why? Because every
Hey there – and welcome to the very first issue of The Missing Header! Formerly known as the Tablecruncher Newsletter, I’ve rebranded it to avoid confusion with the software project. Same author, same scope — still all about solving messy data problems. I’m really glad you’re here. We’ll