FluffyBench

Why can identical-looking text fail a comparison?

A name can look exactly right and still fail a search or duplicate check. The hidden difference may be how its letters are stored—and normalization can make equivalent spellings match.

FluffyBench · 5 min read · Reviewed

What does an exact text comparison read?

It reads the stored values behind the letters, so matching shapes don't guarantee matching text.

Think of a letter's shape as the printed letter, its Unicode code point as a catalog number, and its bytes as the package used to store it.

The printed letter Å points to Unicode code point U+00C5, which is encoded as UTF-8 bytes C3 85.
The shape, the numbered value, and the saved bytes are different layers of the same text. Sources and review notes

Unicode assigns numbered values called code points to text. A font draws their shapes; an encoding, such as UTF-8 or UTF-16, writes them as bytes. When a search misses a name that looks right, the code points are a useful place to look.

How does normalization make equivalent spellings match?

Normalization gives equivalent spellings a shared form, so they can pass an exact comparison.

Å can be stored as a precomposed letter, an Angstrom sign, or A followed by a combining ring. These are canonically equivalent spellings: NFC joins them into the same precomposed letter; NFD separates them into the same letter-and-ring sequence.

Å: U+00C5 → NFC U+00C5; Å: U+212B → NFC U+00C5; Å: U+0041 + U+030A → NFC U+00C5. All NFC results match. Under NFD, all become U+0041 + U+030A.
Different input sequences become identical when the same normalization form is applied to each.

NFC and NFD are standard forms, and either settles this difference. The important part is using the same form on both sides. You're changing how equivalent text is written down, much like putting the same address into a consistent layout before checking for duplicates.

Doing it in FluffyBench's Unicode Inspector

FluffyBench's Unicode Inspector reveals the code points in your text and makes a normalized copy.

Inspect one spelling at a time. These captures use the same Å spellings as the comparison above; short text updates automatically.

  1. FluffyBench Unicode Inspector: From Text, To Code points.
    FluffyBench Unicode Inspector: From Text, To Code points.
    Step 1. From stays on Text. Under To, choose Code points.
    1. From: Text is the input.
    2. To: choose Code points.
  2. Unicode Inspector Input holding A plus a combining ring, with Input and Result tabs.
    Unicode Inspector Input holding A plus a combining ring, with Input and Result tabs.
    Step 2. Paste A plus a combining ring into Input. On a phone, open Result to read its values.
    1. Paste A plus the ring.
    2. Input is for pasting; Result is for reading.
  3. Code points result with separate U+0041 and U+030A rows.
    Code points result with separate U+0041 and U+030A rows.
    Step 3. A plus the ring has separate rows: U+0041 and U+030A.
    1. Read the letter and ring as separate values.
  4. Code points result for precomposed Å, showing U+00C5.
    Code points result for precomposed Å, showing U+00C5.
    Step 4. Back on Input, replace the text with precomposed Å, then open Result: its row is U+00C5.
    1. Input takes the precomposed Å; Result shows its values.
    2. Read U+00C5 for the precomposed Å.
  5. Code points result for Angstrom sign Å, showing U+212B.
    Code points result for Angstrom sign Å, showing U+212B.
    Step 5. Repeat with the Angstrom sign Å: its row is U+212B. Similar shapes can hide different values.
    1. Input takes the Angstrom sign Å; Result shows its values.
    2. Read U+212B for the Angstrom sign Å.

Keep the original text and use a normalized copy for the comparison. Here, NFC turns A plus a ring into Å and leaves the precomposed Å unchanged.

  1. FluffyBench Unicode Inspector: From Text, To NFC.
    FluffyBench Unicode Inspector: From Text, To NFC.
    Step 1. Under To, choose NFC to give equivalent spellings the same composed form.
    1. From: Text is the input.
    2. To: choose NFC.
  2. Original A plus combining ring in Input, reading 2 characters and 3 bytes.
    Original A plus combining ring in Input, reading 2 characters and 3 bytes.
    Step 2. Paste A plus the ring. Input stays as pasted; the NFC result updates automatically.
    1. Input still has the letter plus its ring.
    2. Open Result for the NFC copy.
  3. NFC result Å, reading 1 character and 2 bytes.
    NFC result Å, reading 1 character and 2 bytes.
    Step 3. The NFC result is precomposed Å. Its readout is 1 character.
    1. NFC has joined this letter and ring.
    2. The NFC copy has 1 character.
  4. Precomposed Å in Input, reading 1 character and 2 bytes, with Result tab visible.
    Precomposed Å in Input, reading 1 character and 2 bytes, with Result tab visible.
    Step 4. Back on Input, replace the text with precomposed Å. It already has 1 character; open Result for its NFC copy.
    1. Paste the precomposed Å.
    2. This input already has 1 character.
    3. Open Result for this spelling.
  5. Unchanged NFC result Å with 1 character readout and Copy and Download controls.
    Unchanged NFC result Å with 1 character readout and Copy and Download controls.
    Step 5. The second NFC result is also Å, unchanged. Copy takes the result to your clipboard; Download saves it as a UTF-8 text file.
    1. The second NFC copy is also Å, unchanged.
    2. Copy copies the NFC text; Download saves a UTF-8 text file.

Need to inspect the code points in your text?

When won't normalization make text match?

Normalization follows defined equivalences; it doesn't treat every pair of similar-looking letters as the same letter.

NFKC and NFKD also apply compatibility mappings, which can replace ligatures and fullwidth letters with plain ones. Choose them when you want those distinctions removed; Latin and Cyrillic lookalikes remain different under every normalization form.

Canonical: U+00C5 and U+0041 + U+030A match under NFC. Compatibility: U+FB01 becomes fi, and U+FF21 becomes A under NFKC or NFKD. Latin a (U+0061) and Cyrillic а (U+0430) stay distinct.
A shared form can settle an equivalent spelling, remove a compatibility distinction, or leave different letters distinct.

Why can saved text have different bytes?

UTF-8 and UTF-16 package the same code points differently, so matching text can still produce different files.

The grinning face 😀 is one code point. It takes 4 bytes in each encoding below; UTF-16 groups those bytes into 2 code units, with BE and LE choosing the byte order within each unit.

U+1F600: UTF-8 F0 9F 98 80; UTF-16BE D8 3D DE 00; UTF-16LE 3D D8 00 DE. Each occupies 4 bytes here. UTF-16 has 2 two-byte code units.
Same emoji, different byte sequences; the boxes group UTF-16 bytes by code unit.

For hexadecimal byte views and conversions, use FluffyBench's Base encoder. Unicode Inspector handles code points and normalization. Base encoder handles the UTF-8 and UTF-16 byte representations, including conversion back to text.

What does a byte order mark add to a file?

A byte order mark, or BOM, is a leading marker separate from the first letter.

A BOM uses the code point U+FEFF. In UTF-16 it marks the byte order; in UTF-8, which has no byte-order choice, it marks the encoding.

A without / with a BOM: UTF-8 41 / EF BB BF 41; UTF-16BE 00 41 / FE FF 00 41; UTF-16LE 41 00 / FF FE 41 00.
The marker adds bytes ahead of A without changing A's own bytes.

For saved files, the useful comparison depends on your question: do you need the same text, or the exact same file bytes?

Same text: identify each file's encoding and BOM, decode each with its own encoding and account for the marker, normalize equivalent spellings to the same form when appropriate, then compare the text. Exact file: compare the original bytes; encoding and BOM differences count as differences.
Text comparison decodes the packaging; exact file comparison keeps every byte in the question.
Search FluffyBench