I have a text file with tamil words.
one word per line.
cat test.txt
?????
??????.
?????
Let us sort this.
cat test.txt | sort
??????.
?????
?????
Let us use uniq for this.
cat test.txt | sort | uniq
??????.
?????
?????
uniq is not working for the unicode text.
We are collecting tamil words to build a tamil spellchecker using hunspell.
We need to remove duplicate words from the collection.
The uniq is not working.
Is there any other way to find duplicate words from unicode file?
Thanks.
--
Regards,
T.Shrinivasan
My Life with GNU/Linux : http://goinggnu.wordpress.com
Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com
Get CollabNet Subversion Edge : http://www.collab.net/svnedge
