Compare lines and keep only one copy of identical lines. You can choose whether to ignore leading/trailing spaces. All processing happens locally in your browser — data is never uploaded.
Features
- Paste multi-line text and remove duplicate lines in one click.
- Choose Ignore leading/trailing spaces (default) or Keep leading/trailing spaces.
- Filter empty lines and count how many times each duplicate appears.
- Side-by-side unique results and duplicate list; light end-of-line annotations after running.
Use Cases
- List cleaning: Merge emails, phones, usernames, or SKUs from multiple sources before import.
- Keywords & tags: Build a deduplicated word list from search terms or hashtags.
- Log & ID cleaning: See which lines repeat and how often each appears.
How to Use
- Paste text (one item per line) in the input box.
- Pick space handling: default is ignore leading/trailing spaces; or keep them.
- Click Deduplicate. Empty lines are filtered and duplicates removed; the unique result is copied to the clipboard automatically.
- The input switches to an annotation view: later occurrences show
(出现第 N 次)and fade; trimmed lines note ignored leading/trailing spaces; empty lines show(忽略空行). Click the annotation view to edit again. - Below, view Deduplicated results and Duplicates (
Content (Count)side by side). Use sort to cycle original / ascending / descending (display order of unique results only). - Copy via Copy unique results or each panel’s copy control; Reset clears everything.
Notes
- Full-line matching: only completely identical lines count as duplicates; case, full-width/half-width, and in-line spacing differences stay distinct.
- With Keep leading/trailing spaces, lines that differ only by edge spaces are not merged.
- Sorting changes only how unique results are shown, not dedupe logic or occurrence counts.
Related Keywords
Text Deduplication, Online Text Deduplication, dedupe, ignore spaces, occurrence count, Sanye Hub, Online Tools, 10000