Remove Duplicates Not Working? Why Identical-Looking Lines Stay
Duplicate lines that will not go away usually differ by case, a trailing space, an invisible character or a curly apostrophe. How to find and fix each one.
The short answer
If two lines look the same but a duplicate remover keeps both, they are not identical character for character. The usual culprits are different capitalisation, a space at the end of one line, an invisible character copied from a web page or spreadsheet, or two kinds of apostrophe. In the Remove Duplicate Lines tool, untick Case sensitive and keep Trim whitespace ticked to fix the first two; the others need a quick clean-up first, explained below.
How the tool decides that two lines are duplicates
It helps to know exactly what the tool compares, because it is stricter than your eye.
The counter under the button shows how many lines were removed and how many unique lines remain. Note that dropped empty lines are included in the "removed" number, so a list with blank lines between items can show more removals than you expected.
Cause 1: Different capitalisation
"Apple", "apple" and "APPLE" are three different lines to a computer. With Case sensitive ticked, all three survive. Untick it and run the tool again:
| Input | Case sensitive ticked | Case sensitive unticked |
|---|---|---|
| Apple, apple, APPLE | 3 lines kept | 1 line kept (Apple) |
This matters most for email addresses, which are almost always treated as case-insensitive by mail providers, and for keyword lists exported from different tools.
Cause 2: A space or tab at the end of a line
A trailing space is invisible, but "apple " (with a space) and "apple" are different strings. With Trim whitespace ticked, the tool strips spaces and tabs from both ends of every line before comparing, so these become duplicates.
If you unticked Trim whitespace because you need indentation kept, the trailing-space problem comes back. Tick it again, or remove only trailing spaces with Find and Replace (tick Use regex, find [ \t]+(?=\n|$) and leave Replace with empty, or clean the list in a spreadsheet with TRIM).
Trimming also removes a non-breaking space at the start or end of a line. Those often arrive when you copy from a web page or Word.
Cause 3: Two spaces in the middle
Trimming only touches the ends of a line. "red apple" (two spaces) and "red apple" (one space) are still different. This happens a lot with text copied from PDFs.
Fix: in Find and Replace, tick Use regex, find {2,} (a space followed by {2,}) and replace it with a single space. Then run the duplicate remover again.
Cause 4: Invisible zero-width characters
Some characters take up no space at all. The most common is the zero-width space (Unicode U+200B), which appears in text copied from websites, chat apps and some CMS editors. A line containing one looks exactly like a normal line, and trimming does not remove it.
How to spot it:
- Click at the end of the word and press the left arrow key. If the cursor seems to "stick" for one press without moving, there is an invisible character there.
- Paste the line into the Word Counter. If "apple" shows 6 characters instead of 5, something hidden is in it.
Fix: in Find and Replace with Use regex ticked, find [\u200B-\u200D\uFEFF] and replace with nothing. This removes zero-width spaces, zero-width joiners and the byte-order mark.
Cause 5: Curly and straight quotes or apostrophes
Word processors and phones turn ' into ’ automatically. So "it's" typed on a laptop and "it’s" typed on a phone are two different lines. The same goes for "double quotes" versus “curly quotes”, and for a hyphen (-) versus an en dash (–).
Fix: use Find and Replace to change ’ to ' (and “ ” to ") across the whole list before removing duplicates.
Cause 6: Look-alike letters from another alphabet
Rare, but it happens with lists scraped from the web or with deliberately disguised spam: the Cyrillic letter "а" looks the same as the Latin "a". A line such as "аpple" written with a Cyrillic first letter will never match "apple". If one stubborn pair remains after every other fix, retype the line by hand and run the tool again.
Cause 7: Windows line endings
Files saved on Windows end each line with two characters (a carriage return and a line feed). If Trim whitespace is ticked, the extra carriage return is removed and you will not notice. If you untick it, the last line of a file (which often has no line ending) can fail to match the same text on an earlier line. Keep Trim whitespace ticked unless you have a specific reason not to.
A clean-up order that fixes almost everything
When a list comes from several sources, run these steps in order:
Worked example
Here is a list of six email addresses pasted from two spreadsheets:
Riya@example.comriya@example.com(trailing space)amit@example.comamit@example.com(with a hidden zero-width space after "com")neha@example.comNEHA@example.com
With the default settings (case sensitive, trim on), nothing is removed and all six lines remain. Trimming does strip the space after the second riya, but "Riya" and "riya" still differ in case. Untick Case sensitive and two lines go (the second riya and NEHA), leaving four. Both amit lines are still there, because one contains a hidden character. After removing zero-width characters with Find and Replace, the second amit line becomes a true duplicate, and the final list is three unique addresses.
When to use a spreadsheet instead
The tool works on whole lines. If your data has several columns and you want to remove rows where only the email column repeats, a spreadsheet is the better choice. See remove duplicates in Excel, Google Sheets or online for when each method fits. To find which items appear in one list but not another, rather than removing repeats, use the Text Diff Checker.
Privacy
The comparison runs in your browser. The text you paste is not uploaded, which makes it fine for customer lists and email addresses.
Frequently asked questions
Why does remove duplicates keep lines that look the same?
Because they are not identical. The usual differences are capital letters, a trailing space, an invisible zero-width character, a curly versus straight apostrophe, or two spaces in the middle of the line.
How do I remove duplicates ignoring upper and lower case?
Untick Case sensitive before you click Remove Duplicates. Apple, apple and APPLE are then treated as one line, and the first version in your list is kept.
Does the tool remove blank lines?
Yes. Empty lines are always dropped, and they are included in the removed count shown after processing.
How do I find invisible characters in a line?
Paste the line into a character counter. If a five-letter word shows six characters, there is a hidden character. Remove zero-width characters with a regex find and replace on [\u200B-\u200D\uFEFF].
Does removing duplicates change the order of my list?
No. The first copy of each line stays where it was. The order changes only if you tick Sort result.
Put this guide into practice with our free online tool — no signup required.
Open tool