Remove Duplicate Lines

Paste a list and remove repeated lines without changing the order of the rest.

Loading tool…

About the Remove Duplicate Lines

Unlike sort | uniq, this keeps the original order: each line is compared with a set of lines already seen, and repeats are dropped. Keep first keeps the first copy in its position; Keep last keeps the last copy, useful for logs or config where later entries override earlier ones.

Ignore case treats Apple and apple as the same line, and Trim spaces before comparing ignores leading and trailing whitespace, which often hides duplicates in data pasted from spreadsheets. In both cases the kept line is output as written. Remove empty lines drops blank lines entirely. The status line shows how many duplicates were removed.

The comparison uses a JavaScript Set, so lists with hundreds of thousands of lines are processed in a fraction of a second, all in your browser. Typical uses: cleaning email lists, keyword lists, .gitignore entries or log lines.

Example: with Ignore case on, the list apple, banana, Apple, cherry becomes apple, banana, cherry.

How to use it

  1. Paste your lines or open a text file.
  2. Choose keep first or keep last.
  3. Set the ignore-case, trim and empty-line options.
  4. Copy or download the result.

Frequently asked questions

Does it change the order of lines?
No. Lines stay in their original order; only repeats are removed. Use Sort Lines if you also want them sorted.
How do I remove duplicates in the terminal?
awk '!seen[$0]++' file keeps the first copy in order. sort -u file also dedupes but sorts the output.
Are lines with different spacing duplicates?
With “Trim spaces before comparing” on, leading and trailing spaces are ignored. Spaces inside a line always count.

Related tools