Back to Free Tools
FREE TOOLS
NDJSON Record Deduplicator
Remove duplicate whole NDJSON object records while retaining first order and exact numeric token spelling.
Pasted NDJSON, One UTF-8 NDJSON file
deduplicated.ndjson, ndjson-deduplication-report.json
Browser only
Runs locally in your browser. Your files, pasted content and generated output are not uploaded. Reset clears this tool’s working data.
This tool needs JavaScript. If it does not load, refresh the page and try again.
How to use this tool
- Enter the input and explicit options shown. A selected UTF-8 file wins its corresponding pasted text.
- Select Deduplicate records and inspect the complete stated conventions.
- Download deduplicated.ndjson, ndjson-deduplication-report.json.
Examples
{"a":1} and {"a":1.0} are one whole-record identity; first spelling remains.
Limitations
- At most1 MiB UTF-8,10000 records,depth40,200000 combined parse/identity visits,8 MiB cumulative identity keys,5 MiB combined artifacts and30 seconds.
- Object records only,one initial BOM,LF/CRLF and final LF required. Blank records,bare CR,interior BOM,duplicate members or scalar roots refuse.
- Whole-record identity ignores object order, respects arrays/decoded strings and exact mathematical decimals. First numeric token spelling remains in compact LF output; original line whitespace and selected-field deduplication are outside this version.
Related tools
No items found.