Remove Duplicate Lines

What it does

Paste a list — email addresses, keywords, URLs, names, log lines — and the tool keeps the first occurrence of each line and drops the rest. The original order is preserved unless you ask for it to be sorted.

The options

• Ignore case — treats Alice and alice as the same line. Essential for email addresses and tags; wrong for anything case-sensitive, like passwords or IDs.
• Ignore surrounding spaces — trims each line before comparing, so  apple and apple  match. This catches the most common source of duplicates that “look identical”: invisible trailing whitespace from a spreadsheet export.
• Drop blank lines — removes empty lines entirely rather than keeping one.
• Sort A–Z — orders the result alphabetically, using natural ordering so item2 comes before item10.
• Show duplicates only — inverts the tool. Instead of the de-duplicated list, you get just the values that appeared more than once. Useful for auditing rather than cleaning.

Where duplicates come from

Almost always a merge: two exports concatenated, a copy-paste that overlapped, a form that was submitted twice. The counts under the result — lines in, lines out, removed — tell you how bad the overlap was, which is often the actual thing you wanted to know.

Common questions

Is my list uploaded?

No. The de-duplication runs in your browser on this page. For lists containing email addresses or customer data, that matters.

How large a list can it handle?

Tens of thousands of lines comfortably. Very large files — hundreds of thousands of lines — may make the browser tab briefly unresponsive while it works.

Does it remove duplicate words, or only whole lines?

Whole lines. To de-duplicate words, put one per line first — the Text Cleaner can help you reshape a block of text.