Hello Mr. Shrinivasan,
You might want to look into the IMS Open Corpus Workbench (CWB)
<http://cwb.sourceforge.net/>. It "is a collection of open-source tools for
managing and querying large text corpora (ranging from 10 million to 2
billion words) with linguistic annotations. Its central component is the
flexible and efficient query processor CQP."
1. Full query capability (using regular expressions) through the command
line.
2. A simpler query syntax for beginners, using wildcard characters.
2. For non-technical users there is a graphical interface (CQPweb) or the
Windows version.
Possible limitations:
1. Even though it has Unicode support (UTF-8 encoding), all usage and
discussions I see are about European languages.
2. Your corpus seems to be about 7 or 8 billion words. This claims to
handle 2 billion words.
Online demos are here. <http://cwb.sourceforge.net/demos.php>
SourceForge project is here. <https://sourceforge.net/projects/cwb/>
Mailing list archives are here.
<http://liste.sslmit.unibo.it/pipermail/cwb/>
Best,
R. Asokan
On Fri, Oct 7, 2016 at 5:07 PM, Shrinivasan T <tshrinivasan@xxxxxxxxx>
wrote:
Hi,
Many tamil scholars are looking for a search engine for tamil literatures.
They often look for the following things.
1. search for any word in all literature. highlight the line of
occurrence, if possible one line above and below.
2. frequency of any given words
3. major used, minor used words by any given author
The literature are available in text format here.
http://www.projectmadurai.org/pmworks.html
There are people who scraps tamil websites regularly.
They have around 180 GB of tamil in plain text format.
When they do a grep for any word it tools 8-10 hours on normal desktop.
I think we can use bigdata tools for them.
Can we use elasticsearch/druid for their purpose?
How to import the plaintext to these tools?
share your thoughts on this.
--
Regards,
T.Shrinivasan
My Life with GNU/Linux : http://goinggnu.wordpress.com
Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com
Get Free Tamil Ebooks for Android, iOS, Kindle, Computer :
http://FreeTamilEbooks.com
_____________________________________
ILUGC List: //www.freelists.org/list/ilugc
ILUGC Web: http://ilugc.in/
