Detecting and Removing Zero-Width Characters in Code

The characters that cause trouble

A handful of code points do most of the damage. They render as nothing but are very much present in the byte stream.

Note the nuance with U+200D: it is a legitimate part of emoji sequences, so a blind strip can corrupt emoji. A careful tool distinguishes context, or at least warns. This is the same list our consumer guide covers for non technical users.

Detection in Python

Detection is deterministic: walk the string and check each character against the set. Report the name and index so the output is actionable, not just a count.

class="syn-keyword">import unicodedata

INVISIBLE = {
    "\u200b": "ZERO WIDTH SPACE",
    "\u200c": "ZERO WIDTH NON-JOINER",
    "\u200d": "ZERO WIDTH JOINER",
    "\u2060": "WORD JOINER",
    "\ufeff": "BYTE ORDER MARK",
}

class="syn-keyword">def scan(text):
    hits = []
    class="syn-keyword">for i, ch class="syn-keyword">in enumerate(text):
        class="syn-keyword">if ch class="syn-keyword">in INVISIBLE:
            hits.append((i, INVISIBLE[ch], hex(ord(ch))))
    class="syn-keyword">return hits

class="syn-keyword">for idx, name, code class="syn-keyword">in scan(open("input.txt", encoding="utf-8").read()):
    print(f"position {idx}: {name} ({code})")

Removal and normalisation

Stripping the characters is the easy part. The step people forget is normalisation: converting to Unicode NFC so that look-alike characters composed from multiple code points collapse to a canonical form. Without it, two visually identical strings can still fail an equality check.

class="syn-keyword">import unicodedata

STRIP = dict.fromkeys(0x200b, None)  # start; extend below
STRIP = {ord(c): None class="syn-keyword">for c class="syn-keyword">in ["\u200b","\u200c","\u200d","\u2060","\ufeff"]}

class="syn-keyword">def clean(text):
    text = text.translate(STRIP)          # remove invisibles
    text = unicodedata.normalize("NFC", text)  # canonicalise
    class="syn-keyword">return text

cleaned = clean(open("input.txt", encoding="utf-8").read())
open("output.txt", "w", encoding="utf-8").write(cleaned)

The same thing in JavaScript

For browser or Node tooling, the logic is identical. A regular expression covering the ranges plus a normalize call does the job.

class="syn-keyword">function clean(text) {
  // remove zero-width and BOM, then normalise
  class="syn-keyword">const stripped = text.replace(/[\u200B-\u200D\u2060\uFEFF]/g, "");
  class="syn-keyword">return stripped.normalize("NFC");
}

class="syn-keyword">function scan(text) {
  class="syn-keyword">const re = /[\u200B-\u200D\u2060\uFEFF]/g;
  class="syn-keyword">const hits = [];
  class="syn-keyword">let m;
  class="syn-keyword">while ((m = re.exec(text)) !== null) hits.push(m.index);
  class="syn-keyword">return hits;
}

The edge case: emoji and combining marks

A naive strip of the zero width range can corrupt valid text, and this is where careless tools cause new bugs while fixing old ones. The zero width joiner, U+200D, is a legitimate and essential part of emoji sequences: a family emoji or a profession emoji is several code points glued together with joiners. Strip them blindly and you shatter the emoji into its components. Combining marks raise a related issue, where a character and its accent are separate code points that must stay together. The safe approach is to be deliberate: remove the pure formatting characters, but treat U+200D with care, and always normalise afterward so composed and decomposed forms converge.

class="syn-keyword">import unicodedata, regex

class="syn-keyword">def clean_safe(text):
    # remove zero-width space/BOM/word-joiner, but preserve ZWJ class="syn-keyword">in emoji
    text = regex.sub(r"[\u200B\u200C\u2060\uFEFF]", "", text)
    # only drop ZWJ when it is NOT between emoji
    text = regex.sub(r"(?<!\p{Emoji})\u200D(?!\p{Emoji})", "", text)
    class="syn-keyword">return unicodedata.normalize("NFC", text)

If you do not need to preserve emoji, the simpler strip is fine. The point is to make that choice consciously rather than discover it in a bug report.

Performance at scale

Detection is linear in the length of the text, so it is cheap, but at scale two things matter. First, compile your regex once and reuse it rather than rebuilding it per call. Second, prefer translate tables or a single compiled pattern over character by character Python loops when you process large volumes, since the constant factor adds up across millions of documents. For most workloads the naive version is more than fast enough; reach for the optimised path only when profiling says you need it.

Putting it in CI

Because detection is deterministic, it belongs in automation. A pre-commit hook or CI check that fails when invisible characters appear in source, configuration, or structured data will catch a whole class of maddening, invisible bugs before they ship. The exact same primitive powers our scan endpoint, so you can call the hosted version instead of maintaining your own list if you prefer.

What this does not do

Be clear with your users about scope. Removing zero-width characters cleans one specific kind of marker. It does not touch a model's statistical watermark, which lives in token choices and cannot be read or removed with any guarantee by an outside tool. Conflating the two is the most common misconception in this area, and a technical audience is exactly the group that should get it right.

A shippable CLI

To make this useful day to day, wrap the logic in a small command line tool that reads from a file or standard input, prints a report, and exits non zero if anything was found, which is what lets it gate a commit. The pieces are all above: the invisible set, a scan that records positions, and a clean that strips and normalises. Adding argument parsing and an exit code turns the primitive into something you can drop into a pre-commit hook or a CI job, so invisible characters are caught automatically rather than discovered in a confusing bug report weeks later.

class="syn-keyword">import sys

class="syn-keyword">def main():
    data = sys.stdin.read()
    hits = scan(data)
    class="syn-keyword">if hits:
        class="syn-keyword">for i class="syn-keyword">in hits:
            print(f"hidden character at position {i}", file=sys.stderr)
        sys.exit(1)   # fail CI
    sys.stdout.write(clean(data))

class="syn-keyword">if __name__ == "__main__":
    main()

Frequently asked questions

Will removing zero-width characters change my visible text? No. They have no width, so stripping them leaves the readable text identical, though you should normalise to NFC afterward for reliable matching.

Is it safe to strip U+200D everywhere? Not blindly. The zero width joiner is part of emoji sequences, so a naive strip can break emoji. Handle it deliberately if your text contains emoji.

Does this remove AI watermarks? No. It removes hidden Unicode markers only. A statistical text watermark lives in token choices and cannot be read or removed by this method with any guarantee.

Keep reading: Build a C2PA inspector · Attacks on text watermarks · Text API reference