Unicode inspection
Invisible Unicode characters: inspect before removing
Two strings can look identical and still compare differently. A zero-width space can break search. A no-break space can change wrapping. A direction control can change how mixed scripts are displayed. Some invisible characters are accidental residue, while others are required for correct language shaping or emoji presentation.
The safe order is inspection, classification, selective removal, and visual review. Deleting every non-printing character is not a safe default.
Before
alpha[U+200B]beta price[U+00A0]10
After
alphabeta price 10
Five different categories
TextClean reports the character name, Unicode code point, category, count, and first positions. It distinguishes zero-width characters, direction controls, variation selectors, Unicode tag characters, and special spaces because the risks are different.
A zero-width space may have been inserted by a website. A zero-width joiner or non-joiner may be part of valid shaping. A variation selector can choose the text or emoji form of a character. Right-to-left marks and isolates can keep punctuation and numbers in the correct visual order.
- Zero-width spaces, joiners, soft hyphens, word joiners, and byte order marks.
- Left-to-right and right-to-left marks, embeddings, overrides, and isolates.
- Variation selectors used by text and emoji presentation.
- Unicode tag characters used in certain sequences.
- No-break, figure, thin, hair, narrow, and mathematical spaces.
Why remove all is dangerous
Arabic, Persian, Urdu, Hebrew, and mixed-direction text can depend on controls that are invisible by design. Indic scripts use combining behavior, and emoji sequences can depend on joiners and variation selectors. Removing a character because it has no visible glyph can change spelling, ordering, or presentation.
Inspection in TextClean does not modify the source. Special spaces, zero-width characters, and direction controls have separate actions. Variation selectors and tag characters are removed only by the broader remove-all action.
A practical debugging method
When search, sorting, duplicate detection, or validation behaves unexpectedly, compare a short failing sample. Record the reported code points and positions. Change one control at a time, then test the destination again.
The zero-width category includes joiners and non-joiners that can be required for Persian and other scripts. Do not remove the whole category from multilingual text unless a competent reader has checked the result.
If the text contains a language you cannot read, do not use visual similarity as proof. Ask a competent reader or retain the original controls.
Normalization is not deletion
Unicode normalization can make canonically equivalent sequences consistent. For example, an accented letter may be stored as one code point or as a base letter plus a combining mark. NFC normalization can make those forms comparable without stripping the accent.
NFKC is more aggressive and can replace compatibility forms. TextClean uses NFC by default and makes destructive removal a separate choice.
Check before using the result
- Code point and category
- Writing direction
- Emoji presentation
- Search and string comparison
- Appearance in the final application