Use sort -u
On Fri, Jul 25, 2014 at 4:02 PM, Shrinivasan T <tshrinivasan at gmail.com>
wrote:
I have a text file with tamil words.
one word per line.
cat test.txt
?????
??????.
?????
Let us sort this.
cat test.txt | sort
??????.
?????
?????
Let us use uniq for this.
cat test.txt | sort | uniq
??????.
?????
?????
uniq is not working for the unicode text.
We are collecting tamil words to build a tamil spellchecker using hunspell.
We need to remove duplicate words from the collection.
The uniq is not working.
Is there any other way to find duplicate words from unicode file?
Thanks.
--
Regards,
T.Shrinivasan
My Life with GNU/Linux : http://goinggnu.wordpress.com
Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com
Get CollabNet Subversion Edge : http://www.collab.net/svnedge
_______________________________________________
ILUGC Mailing List:
http://www.ae.iitm.ac.in/mailman/listinfo/ilugc
ILUGC Mailing List Guidelines:
http://ilugc.in/mailinglist-guidelines
