📝 Text & Content Free Forever

Duplicate Line Remover, Clean Up Lists in One Click

This duplicate line remover strips repeated lines from any list or block of text instantly. Choose case sensitivity, trim whitespace, keep first or last occurrence, and sort the result, no sign up needed. This duplicate line remover is free and needs no signup.

Your Text
Options
About This Tool

How this duplicate line remover decides what counts as the same line

Removing duplicate lines sounds like a one liner, but the definition of duplicate is the whole problem. Is Apple the same as apple? Does trailing whitespace count? If a line repeats three times, do you keep the first copy or the last? This duplicate line remover exposes every one of those decisions as a toggle instead of guessing, then processes the result with a hash map lookup that keeps performance linear even on large pastes. Half a million lines take about a second and a half.

The whole operation runs client side with a JavaScript Map as the lookup table. Nothing you paste is sent anywhere.

The dedupe algorithm

Step 1 Split and clean Input is split on /\r?\n/, which handles both Windows and Unix line endings. If trimming is enabled, each line has leading and trailing whitespace stripped before anything else happens. If ignore blank lines is enabled, empty lines are filtered out entirely.
Step 2 Build a comparison key For every line, the tool computes a key: the line itself if case sensitive matching is on, or line.toLowerCase() if it is off. This key, not the original text, is what gets compared.
Step 3 First seen wins, unless you ask for last The tool walks the lines (in reverse if you chose keep last occurrence) and records the first and last position of every key in a Map. Because Map lookup is O(1), checking whether a key already exists does not require rescanning prior lines, which is what keeps this fast on thousands of lines. A third choice, None, drops every line that appears more than once and keeps only lines that appear exactly once.
Step 4 Optional alphabetical sort If sorting is enabled, the deduplicated result is sorted with Intl.Collator, which orders text the way a human reader expects rather than by raw character codes, and puts item2 before item10. Leave it off and the original order is preserved exactly.
// key based dedupe, adapted from the tool source var first = new Map(), last = new Map(), counts = new Map(); lines.forEach(function(line, i){ var key = caseSensitive ? line : line.toLowerCase(); counts.set(key, (counts.get(key) || 0) + 1); if (!first.has(key)) first.set(key, i); last.set(key, i); }); var result = lines.filter(function(line, i){ return (keepLast ? last : first).get(keyOf(line)) === i; });

A worked deduplication

Input:

apple Banana apple cherry banana

Output with case sensitive off, keep first, no sort:

apple Banana cherry
OptionEffect on the example above
Case sensitive onBanana and banana are now treated as different lines, so both survive
Keep last occurrenceOutput keeps the second apple and the second banana instead of the first
Sort resultOutput reorders to apple, Banana, cherry by locale comparison
Trimming changes what counts as a duplicate, not just how the line looks. If trim is off, "apple" and "apple " with a trailing space are different keys and both survive. Turn trimming on and they collapse into one. This trips people up when a file was exported with trailing spaces on every line, since the tool will look like it did nothing until trim is enabled.

What else the tool reports

Duplicate list with counts

Any key that occurred more than once is listed separately with its exact occurrence count, built from a frequency map computed alongside the dedupe pass, so you can see what was actually removed without diffing the input yourself.

Line count summary

Original line count, unique line count, duplicates removed and empty lines removed are shown as pills above the result, giving you a quick sanity check that the operation did what you expected.

Case sensitive toggle Trim whitespace toggle Ignore blank lines toggle Keep first, last or no repeated line Optional alphabetical sort
  • MDN on the Set object covers the native alternative to a hash map for uniqueness checks in JavaScript.
  • MDN on localeCompare explains the locale aware ordering rules that Intl.Collator also follows when sort is enabled.
  • Hash table background on why key lookups here stay close to constant time regardless of input size.

Cleanup jobs this handles

Cleaning an email list exported from two different signup forms, merging two CSV columns of tags without repeats, tidying a log file where the same error line got written on every retry, deduplicating a keyword list before pasting it into an ad platform, and clearing out repeated entries in a bookmarks or URL export before importing it somewhere else. The case sensitive and trim toggles matter most when the duplicates come from two different sources that formatted the same data slightly differently.

How to remove duplicates from a list

Paste the list with one item per line. The tool keeps the first copy of each line and drops the repeats.

Two lines only match when they are identical, so a trailing space or a different capital letter hides a duplicate. Trim spaces and ignore case to catch those.

Common Questions

FAQ: Duplicate Line Remover

Yes, by default the cleaned result keeps lines in their original order, it simply removes the repeats, it doesn't reorder anything unless you turn on the Sort Alphabetically option. If order matters for your list, for example a sequence of steps or a prioritized list, leave sorting off.

If a line appears multiple times, "Keep first occurrence" preserves the version that appeared earliest in your list and removes later repeats, which is the more common choice. "Keep last occurrence" does the opposite, useful when later entries in a list represent more up to date or corrected versions of earlier ones, such as a log file where the newest entry for a given key should win.

Turn on case sensitivity when capitalization is meaningful in your data, for example a list of exact code identifiers, product SKUs, or proper nouns where "NYC" and "nyc" should be treated as genuinely different values. Leave it off (the default) for general text like names or common words, where "Apple" and "apple" almost always mean the same thing.

Yes, this is one of the most common uses. Paste one email address per line, keep case sensitivity off since email addresses are not case sensitive by convention, and enable whitespace trimming to catch accidental extra spaces copied from spreadsheets, which is a frequent cause of "duplicate" entries that look identical but aren't caught by exact matching.

Yes, when enabled, any line that is completely empty (or empty after whitespace trimming) is dropped from the output entirely rather than being counted as a "duplicate empty line." Turn this off if blank lines are meaningful spacing you want preserved; every blank line is then kept where it was and never treated as a duplicate.

There's no hard limit built into the duplicate line remover itself, it can comfortably handle lists with tens of thousands of lines since processing happens with a simple, efficient lookup rather than comparing every line against every other line. Extremely massive pastes, in the range of hundreds of thousands of lines, may feel slower depending on your device's available memory.

No, this duplicate line remover matches exact duplicates only (after any trimming and case rules you apply), it does not detect "fuzzy" near duplicates like typos, reordered words, or slightly different phrasing that still mean the same thing. For that kind of similarity detection you would need a dedicated fuzzy matching tool rather than an exact line comparison.

No, all processing happens locally in your browser using JavaScript, nothing you paste is sent to a server, logged, or stored anywhere.

Yes. Set Keep Which Occurrence to None. Every line that appears more than once is removed completely, including its first copy, and only lines that appear exactly once are left. This is handy for finding the unique entries between two merged lists.

Open the file, copy its contents into the tool above and copy the cleaned list back.

Select the column, open the Data tab and choose Remove Duplicates.

Put one address per line and switch on the option to ignore case, since email addresses are not case sensitive.