How to Remove Duplicate Lines from Text (Fast and In-Browser)

The problem: messy lists full of repeats

Duplicate lines creep in everywhere: a mailing list stitched together from three exports, a log file with the same error a thousand times, a list of dependencies pasted from two package.json files, a CSV column copied into a text editor. Before you can count, import, or act on that data, you need one clean copy of each line.

How deduplication works

The efficient way to remove duplicates is a hash set: walk the lines once, remember every value you have seen, and keep only the first occurrence of each. This is O(n) and, crucially, preserves the original order, unlike sorting-based approaches.

function dedupe(text, { trim = true, caseSensitive = true } = {}) {
  const seen = new Set();
  const out = [];
  for (let line of text.split('\n')) {
    const value = trim ? line.trim() : line;
    const key = caseSensitive ? value : value.toLowerCase();
    if (!seen.has(key)) {
      seen.add(key);
      out.push(value);
    }
  }
  return out.join('\n');
}

The two options matter more than they look. Trimming makes apple and apple (with a trailing space) count as the same line. Case-insensitivity makes Apple and apple collapse together, which is useful for emails but wrong for case-sensitive data like tokens.

Command-line equivalents

If you live in a terminal, two classic commands do the job:

# Sort and remove duplicates (order NOT preserved):
sort -u emails.txt

# Remove duplicates and KEEP original order:
awk '!seen[$0]++' emails.txt

Note that uniq on its own only removes adjacent duplicates, which is why it is almost always paired with sort. The awk one-liner is the order-preserving equivalent of the JavaScript above.

A practical walkthrough

Imagine you merged two newsletter exports and have 4,300 email addresses with an unknown number of overlaps. Paste the whole list into the Remove Duplicate Lines tool, enable trimming and case-insensitive matching, and you instantly get the unique set, which you can then count to see how many real subscribers you have. Because it runs locally, you are not uploading a list of personal email addresses to a random website.

Common pitfalls

• Invisible whitespace. Trailing spaces, tabs, and Windows \r line endings make identical-looking lines differ. Trim, or normalize line endings first.
• Case sensitivity. Deduping usernames or file paths case-insensitively can silently merge two distinct records. Only enable it when the data really is case-insensitive.
• Order versus sorting. If you sort to dedupe, you lose the original order. Use a set-based method when order carries meaning, like a changelog.
• Blank lines. Decide whether to keep one blank line or drop them all; deduping treats every blank line as the same value.

Related tools

Deduping and sorting are often the same chore. The Sort Text Lines tool orders your list alphabetically or numerically after, or instead of, removing repeats. And when the duplicates are actually substrings you want to swap out, Find and Replace gives you pattern-based edits across every line.

Conclusion

Removing duplicate lines is a set operation with two dials, trimming and case-sensitivity, that decide what "the same line" means. Get those right and the rest is instant. The Remove Duplicate Lines tool applies both in your browser, so even sensitive lists never leave your machine.