Why you can’t trust your computer — even with numbers

Numbers are easy for computers — that’s what we’ve been told. But try storing 9007199254740993 in Excel and you’ll get… a different number.

What’s going on?

The invisible conversion

When I wrote the first version of my CSV editor Tablecruncher, one of my non-negotiables was:

Don't touch the data.
If a file says 0.000123456789012345, then that’s what the user sees.

This sounds obvious. But lots of software quietly disagrees. If it sees a column of numbers, it says: “Hey, these are numbers — let’s convert them to floating point!”

Why? Because floating point numbers are fast. Great for math.
But that speed comes at a cost: limited precision.

Why it breaks

Skipping all the boring implementation details, what makes numbers so complicated is that there are just so many of them.
Between any two real numbers, there’s an infinite number of other numbers. But your computer only has a finite amount of space — typically 64 bits per number.
So it’s obvious: you can’t store every number exactly. You only get a best-effort approximation.

A real example: Excel

Try this:

A1: 0,000123456789012345
A2: 1
A3: =A1+A2

Then format the cells to show 20 digits.

The last digits are off — that’s the IEEE 754 limit.

Now try:

A4: =A3-A2

You’d expect the original value again. But Excel gives you:
0,00012345678901226600.

This isn’t just rounding. It’s accumulated error.
In finance, that can mean real money. In auditing, it can mean lawsuits.

And it gets worse. Try entering: 9007199254740993.

Excel changes it to: 9007199254740990

Because 9007199254740993 isn’t representable in 64-bit floats. So Excel picks the nearest number it can represent — in this case, 9007199254740990. And it does this without asking.

It’s not just Excel

Python and JavaScript have similar problems:

>>> 0.1 + 0.02
0.12000000000000001

>>> 9007199254740993.0 > 9007199254740992.5
False

Yes, even comparisons can go sideways when numbers are just… wrong.

What to do

Text-based formats like CSV and JSON store numbers as strings.
That’s good: it keeps them intact. But the moment you open the file in Excel, or run code that auto-parses it into floats, you’re back in danger.

Unless you’re explicitly doing math, treat numbers — especially IDs or money — as text or integers. And never trust software defaults. Always check how all the tools and scripts in your workflow handle numbers internally.

That’s why Tablecruncher leaves your data exactly as it is — no guessing, no auto-converting just because something “looks like a number.”

But the moment you sort a column or compare values, some kind of conversion has to happen. And that’s where the float problems creep in again. Yes, even in Tablecruncher.

Very often, you can’t avoid floats completely — but you can know where the cracks are.

Thanks for reading,
Stefan

🧮 The Missing Number

754 – the number of the technical standard for floating-point arithmetic.
Used by Excel, Python, JavaScript, and most of the tools you work with.

PS: This is issue #010 of The Missing Header. You may be receiving this newsletter because you subscribed to the Tablecruncher Newsletter some time ago. I’ve rebranded it to avoid confusion with the software project. Same author, same scope — still all about solving messy data problems.

[1] Floating-point arithmetic may give inaccurate results in Excel:
https://learn.microsoft.com/en-us/troubleshoot/microsoft-365-apps/excel/floating-point-arithmetic-inaccurate-result