๐งน Duplicate Line Remover
Remove duplicate lines from a block of multi-line text, keeping only each item's first occurrence. Useful for cleaning up a list, or removing duplicates from an email list or product name list.
How to use
- Enter the text you want to de-duplicate, one item per line.
- Toggle "Case-insensitive" and "Trim leading/trailing whitespace" as needed.
- The de-duplicated result appears automatically. Use the "Copy" button to copy it.
How the calculation works
This tool splits the input into lines and checks each line, from top to bottom, against the lines already seen. It uses a Set (a collection of unique values): a line seen for the first time is kept and added to the set, and any later identical line is removed. Because of this, the remaining lines stay in their original order. Unlike the "Remove duplicates" option in the text sorter, the order does not change, which suits data where order matters, such as logs or lists of names. The number of removed lines is shown as well. When "Trim leading/trailing whitespace" is on (the default), leading and trailing spaces are stripped before comparing, and the output uses the stripped form. With "Case-insensitive" on, "Apple" and "apple" are treated as the same line, and whichever spelling appeared first is kept.
Worked example
Input apple banana apple cherry banana apple Output apple banana cherry (3 lines removed) The second and later "apple" and "banana" lines are removed, and the rest stay in the order they first appeared (apple โ banana โ cherry).
Things to be aware of
- An empty line counts as a line too. If there are several empty lines, only the first one is kept.
- With whitespace trimming on, indentation at the start of the kept lines is also removed. Turn it off to keep indentation.
- Full-width and half-width characters (such as "๏ผก" and "A") are treated as different lines. Normalise them first with the full-width/half-width converter if needed.
- Your text is processed in the browser and is never sent anywhere.
FAQ
Is the original line order preserved?
Yes โ each item stays in the order it first appeared, and only later duplicate occurrences are removed.
Are lines that differ only by leading/trailing whitespace treated as duplicates?
With "Trim leading/trailing whitespace" on, differences in leading/trailing whitespace are ignored when checking for duplicates. With it off, whitespace differences make lines count as distinct.
Can I see how many lines were removed?
Yes โ the number of removed lines is shown next to the options.