[Ilugc] uniq is not working for unicode text

  • From: arunprakash.pts@xxxxxxxxx (Arun Prakash)
  • Date: Fri, 25 Jul 2014 16:12:26 +0530

Hello,


uniq is not working for the unicode text.

We are collecting tamil words to build a tamil spellchecker using hunspell.
We need to remove duplicate words from the collection.

The uniq is not working.

Is there any other way to find duplicate words from unicode file?


Why not do it in Libreoffice Calc (
http://milospjanic.blogspot.com/2011/10/how-to-remove-duplicates-in-libreoffice.html
).

-- 
Regards,
Arun Prakash

Other related posts: