Duplicate Line Remover, Clean Up Lists in One Click
This duplicate line remover strips repeated lines from any list or block of text instantly. Choose case sensitivity, trim whitespace, keep first or last occurrence, and sort the result, no sign up needed. This duplicate line remover is free and needs no signup.
How this duplicate line remover decides what counts as the same line
Removing duplicate lines sounds like a one liner, but the definition of duplicate is the whole problem. Is Apple the same as apple? Does trailing whitespace count? If a line repeats three times, do you keep the first copy or the last? This duplicate line remover exposes every one of those decisions as a toggle instead of guessing, then processes the result with a hash map lookup that keeps performance linear even on large pastes. Half a million lines take about a second and a half.
The whole operation runs client side with a JavaScript Map as the lookup table. Nothing you paste is sent anywhere.
The dedupe algorithm
/\r?\n/, which handles both Windows and Unix line endings. If trimming is enabled, each line has leading and trailing whitespace stripped before anything else happens. If ignore blank lines is enabled, empty lines are filtered out entirely. line.toLowerCase() if it is off. This key, not the original text, is what gets compared. Map. Because Map lookup is O(1), checking whether a key already exists does not require rescanning prior lines, which is what keeps this fast on thousands of lines. A third choice, None, drops every line that appears more than once and keeps only lines that appear exactly once. Intl.Collator, which orders text the way a human reader expects rather than by raw character codes, and puts item2 before item10. Leave it off and the original order is preserved exactly. A worked deduplication
Input:
Output with case sensitive off, keep first, no sort:
| Option | Effect on the example above |
|---|---|
| Case sensitive on | Banana and banana are now treated as different lines, so both survive |
| Keep last occurrence | Output keeps the second apple and the second banana instead of the first |
| Sort result | Output reorders to apple, Banana, cherry by locale comparison |
"apple" and "apple " with a trailing space are different keys and both survive. Turn trimming on and they collapse into one. This trips people up when a file was exported with trailing spaces on every line, since the tool will look like it did nothing until trim is enabled.What else the tool reports
Duplicate list with counts
Any key that occurred more than once is listed separately with its exact occurrence count, built from a frequency map computed alongside the dedupe pass, so you can see what was actually removed without diffing the input yourself.
Line count summary
Original line count, unique line count, duplicates removed and empty lines removed are shown as pills above the result, giving you a quick sanity check that the operation did what you expected.
- MDN on the Set object covers the native alternative to a hash map for uniqueness checks in JavaScript.
- MDN on localeCompare explains the locale aware ordering rules that Intl.Collator also follows when sort is enabled.
- Hash table background on why key lookups here stay close to constant time regardless of input size.
Cleanup jobs this handles
Cleaning an email list exported from two different signup forms, merging two CSV columns of tags without repeats, tidying a log file where the same error line got written on every retry, deduplicating a keyword list before pasting it into an ad platform, and clearing out repeated entries in a bookmarks or URL export before importing it somewhere else. The case sensitive and trim toggles matter most when the duplicates come from two different sources that formatted the same data slightly differently.
How to remove duplicates from a list
Paste the list with one item per line. The tool keeps the first copy of each line and drops the repeats.
Two lines only match when they are identical, so a trailing space or a different capital letter hides a duplicate. Trim spaces and ignore case to catch those.
FAQ: Duplicate Line Remover
Yes, by default the cleaned result keeps lines in their original order, it simply removes the repeats, it doesn't reorder anything unless you turn on the Sort Alphabetically option. If order matters for your list, for example a sequence of steps or a prioritized list, leave sorting off.
If a line appears multiple times, "Keep first occurrence" preserves the version that appeared earliest in your list and removes later repeats, which is the more common choice. "Keep last occurrence" does the opposite, useful when later entries in a list represent more up to date or corrected versions of earlier ones, such as a log file where the newest entry for a given key should win.
Turn on case sensitivity when capitalization is meaningful in your data, for example a list of exact code identifiers, product SKUs, or proper nouns where "NYC" and "nyc" should be treated as genuinely different values. Leave it off (the default) for general text like names or common words, where "Apple" and "apple" almost always mean the same thing.
Yes, this is one of the most common uses. Paste one email address per line, keep case sensitivity off since email addresses are not case sensitive by convention, and enable whitespace trimming to catch accidental extra spaces copied from spreadsheets, which is a frequent cause of "duplicate" entries that look identical but aren't caught by exact matching.
Yes, when enabled, any line that is completely empty (or empty after whitespace trimming) is dropped from the output entirely rather than being counted as a "duplicate empty line." Turn this off if blank lines are meaningful spacing you want preserved; every blank line is then kept where it was and never treated as a duplicate.
There's no hard limit built into the duplicate line remover itself, it can comfortably handle lists with tens of thousands of lines since processing happens with a simple, efficient lookup rather than comparing every line against every other line. Extremely massive pastes, in the range of hundreds of thousands of lines, may feel slower depending on your device's available memory.
No, this duplicate line remover matches exact duplicates only (after any trimming and case rules you apply), it does not detect "fuzzy" near duplicates like typos, reordered words, or slightly different phrasing that still mean the same thing. For that kind of similarity detection you would need a dedicated fuzzy matching tool rather than an exact line comparison.
No, all processing happens locally in your browser using JavaScript, nothing you paste is sent to a server, logged, or stored anywhere.
Yes. Set Keep Which Occurrence to None. Every line that appears more than once is removed completely, including its first copy, and only lines that appear exactly once are left. This is handy for finding the unique entries between two merged lists.
Open the file, copy its contents into the tool above and copy the cleaned list back.
Select the column, open the Data tab and choose Remove Duplicates.
Put one address per line and switch on the option to ignore case, since email addresses are not case sensitive.