Hi,
the code is now published at:
freedict/tools : ding2tei-haskell-rewrite : /importers/ding2tei/
Web:
https://github.com/freedict/tools/tree/ding2tei-haskell-rewrite/importers/ding2tei
Regards,
Einhard
On 29/08/2020 03:13, Einhard Leichtfuß wrote:
Hi all,--
I have started working on the importer a while ago. In fact, it has
become the subject of my bachelor's thesis.
The latter is also the reason for that I neither published any code yet
nor contacted you again earlier, since I was unsure what I was allowed
to in the context of my bachelor's thesis.
Once I have write acces to the tools repository, I will publish the
current state of my work. (Unless you were to agree with me that a
separate git repository is more suitable.)
Unfortunately, the program is not yet able to produce any TEI XML from
the Ding source. The only thing it does is to transform (parts of) the
Ding into an internal datastructure and print the number of lines
identified.
I do have some questions. A lot, in fact. I hope not to overwhelm you
with them. Just ignore some of them, if they are to many.
Since this is now my bachelor's thesis, I need to ask you to refrain
from giving me any coding-specific advice (or else I'd have to cite you
in my bachelor's thesis). Please also do not publish any changes to my
code before I submit my bachelor's thesis (september or october 2020).
Note that I currently target version 1.8.1 exclusively.
Questions:
A) TEI
A.1) TEI Lex-0. Have I understood correctly that it is a good idea to
follow this standard [0]? E.g.
* a) <gram type="gender"/> instead of <gen/>.;
* b) <usg> with @type (and possibly @norm)
A.2) Verb & Transitivity annotation.
* In a HowTo [1], it is suggested to use v,vt,vi,vti, i.e., merge
all such information into a single token.
* In an example [2], I see "<pos>v</pos><subc>tr</subc>", which
would also adhere to TEI Lex-0, in contrast to the former.
? So, which to use? (I prefer the latter, if that matters.)
A.3) IPA Pronunciation. The current deu-eng.tei in the Freedict
repository contains <pron> tags. I assume that these were
autogenerated, since the Ding does not contain such information.
If I am right, how can I have that information autogenerated?
A.4) Normalization of usage annotations
* Recommended by TEI Lex-0.
* different languages (e.g. "[Sprw.]" ~ "[prov.]")
* same language (e.g. "[coll.]" ~ "[slang]")
? Should they be normalised to a single label?
? Should they be normalised to some standard labels?
* ISO 12620 [4,5,6] (full standard only commercially available)
* The usage of @norm in <usg> might render that less an issue.
A.5) Quantified (or similar) usage annotations
* Ex.: "mainly Am."
* Ex.: "bes. Süddt.", "especially Am."
? How to represent the determiner?
A.6) Dialect / language annotations.
a) Ex.: "[Br.]", "[Am.]", "[Ös.]", "[Sächs.]"
b) Ex.: "[South Africa]", "[Hessen]", "[Berlin]", "[Wien]"
d) Ex.: "[French]", "[Lat.]"
? Represent as <usg type="geographic">?
* According to TEI Lex-0: "marker which identifies the place or
region where a lexical unit is mainly used"
* Matches c) only.
? Separate d)? And represent how?
A.7) Abbreviations.
a) Headwords, which are annotations.
* rare
b) Annotated on headwords.
? How to represent in TEI?
* The TEI documentation contains an example [7] with both
<form type="abbrev"> and <form type="full">, in the same
<entry>.
* I remember though that within the Freedict project multiple
<form> tags inside <entry> are frowned upon.
* The TEI documentation also offers <abbr> and <expan>, possibly
grouped in <choice> [8], though these does not feel like the
correct tags to me.
B) Linguistics
B.1) {num} / numeral. The Ding annotates with {num} both numerals (e.g.
"acht") and other words expressing quantity, e.g.
- "jedermann; alle ohne Unterschied {pron} {num}",
- a pronoun expressing a quantity
- "beide {pron} {num}".
- either a pronoun expressing a quantity or a quantifier
? I guess the above two examples should not be annotated with pos
"num" in TEI, correct?
? Should I consider them mistakes? Or encode them differently?
I.e.., pron+num -> something
B.1.1) {Quantifikator} / quantifier. There is exactly one such
annotation in the Ding, for "einiges" - which, if I undestand
correctly isn't even a quantifier, but rather in the same
category as "jedermann" (see B.1).
C) Transfer (Ding -> TEI)
C.1) Inline annotations. In the Ding, some annotations are inside a
larger expression / phrase (as delimited by <;>. <|>, <::>) and
only refer to part of it.
? Can/should such annotations be represented in TEI?
* I am primarily concerned with grammar and usage annotations
here (potentially also flected forms - unsure, whether such
occur annotated inline).
* Note that suffixing annotations do not necessarily apply to the
whole expression, but I guess I should assume that.
* Alternatively, if I were to distinguish simple entries (e.g.,
single words) from example expressions/phrases, I could assume
/ require that simple expressions only have suffixing
annotations applying to it as a whole, while phrase annotations
might be entirely dropped.
* Example phrases need a headword they belong to. According to
the above definition, this is not always the case:
* Ex.: "Alternativkontur {f} der Schultern (Reifen) :: bead \
seat optional contour"
C.2) Annotations in parentheses (<()>). These have a multitude of
meanings.
a) Prefix annotations. These should usually represent optional
prefixes.
b) Infix annotations.
i) optional infix
ii) description of / annotation on some preceding semantical
entity, which does not need to be a single word.
* Difficult to identify such an entity.
c) Suffix annotations.
i) optional suffix
ii) description of / annotation on some preceding semantical
entity, which is not the whole expression.
iii) description of / annotation on the whole expression
* Probably more likely when non-()-annotations precede the
()-annotation in question.
? Which of the potential semantics can be represented in TEI, and
if so, how?
? How should I treat annotations that I cannot represent in TEI or
properly extract from the Ding?
* I guess, when part of a phrase, they can just be literally
retained.
C.2) "<>". This separator indicates that the surrounding to entities
(usually single words) may be swapped.
* Potential representations
a) Leave as is, as part of the string.
b) Duplicate the element.
c) Explicit representation (possible?).
* Note: "<>" does not only occur in phrases.
* Ex.: "to file away <> sth."
? Which representation do you prefer / which are possible?
* For non-phrases (see above example) there would probably
ideally be different <colloc> elements.
C.3) "/" (alternative).
* weak slash (usually surrounded by whitespace, separates more
than single words)
* Difficult to identify scope.
* strong slash (not surrounded by whitespace, separates single
words)
* double slash (separates slash-expressions, e.g.: "a/b//c/d")
* Contexts:
a) Ex.: "etw./jdn. <verb>"
? Represent as several <colloc> elements (possible?)?
b) As part of a phrase.
* Potential representations: similar to those in C.2).
? Which?
C.4) "to" prefix for verbs.
? Drop, correct?
? Or only hide somehow (keep as part of the editorial view)?
C.5) Usage [literary].
* One occurrence: "sword; blade [literary]"
* Denotes a <usg type="textType"> (see TEI Lex-0 spec).
* Not equivalent to [lit.], which denotes a usage domain.
* Similar to [poet.]
? Identify?
C.6) Retaining of non-dictionary (typographic, lexical) [3] data?
- Ex.: <;>, <::>, <~>.
C.7) Grouped annotations.
* Ex: "[formal/Am.]", "{
* Grouped []-annotations seem to always represent a disjunction
(regardless of the separator: </>, <;>, <,>)
? Treat differently to separate annotations?
* Ex.: "[formal] [Am.]".
* For {}-annotations, it seems ok to ignore grouping.
C.8) Multiple genders
* Ex.: "Anwesende {m,f}"
* In fact, I would not consider "Anwesende" a base masculine form
(only when used with a definite article).
* Ex.: "Avis {m,n}"
? Simply two gender annotations in a single gramGrp?
C.9) [.]
a) usage annotation
* Ex.: "Sagenit {m} [min.]"
* identifiable by contained keywords
* If relying on that one might miss new keywords in later
versions of the Ding.
b) optional word / expression
* Currently transformed (during preprocessing) to (.).
* Ex.: "a clutch of [persons]" -> "a clutch of (persons)"
c) alternative, like </> but with preference
* Currently transformed to </> (strong slash).
* Ex.: "ruby [red] copper ore" -> "ruby/red copper ore"
* Options to avoid ambiguity.
i) Transform as described above, some information lost.
ii) Keep the information.
* I'd probably invent some new Ding-syntax and adapt the
preprocessing accordingly.
? How to represent b) and c) in TEI?
iii) Rely on (future) phrase identification and assume / require
that in non-phrases, only proper usage annotations occur.
? Which option?
C.10) <.>
* According to the outdated Ding spec [9], they represent
wrong/old forms.
* In practice:
a) alternative (valid) form / synonym
* Ex.: "Supplemental Restraint System /SRS/ <air bag>"
* Ex.: "Acethylen {n} [chem.] <Azetylen>"
* In fact, "Acethylen" seems to be a misspelling of the
other.
* Ex.: "Ausrichtung {f} (nach) <Harmonisierung>"
b) corresponding singular form
* Ex.: "Lurche {pl} (Amphibia) (zoologische Klasse) [zool.] \
<Lurch>"
* quite frequent
c) wrong form
* Ex.: "occurrence <occurrance>"
* rare
* difficult to distinguish from a) in particular.
* Options
i) Manual separation
* 717 occurences of <.>. I am possibly too lazy.
ii) Assume always a)
iii) Assume a), unless b) can be inferred using some heuristic.
* heuristic: a {pl}-annotated group of headwords, followed
by a <.>-word that "looks like" the singular form of one
of the headwords.
? Which option?
? How to represent a), b), c) (unless dropped)?
D) Meta
D.1) I chose AGPL3+ as license (more restricted than GPL3; can be
combined with GPL3 into AGPL3, but not the other way round). Do
you have any issue with that?
Notes:
1) The table 5.1 in the HowTo on Writing an FD dictionary [1] is broken.
Sorry again for the multitude of questions; I would appreciate your
answers to at least some of them.
Regards,
Einhard
[0]
https://dariah-eric.github.io/lexicalresources/pages/TEILex0/TEILex0.html#index.xml-body.1_div.7_div.1
[1]
https://github.com/freedict/fd-dictionaries/wiki/FreeDict-HOWTO-%E2%80%93-Writing-A-FreeDict-Dictionary
[2]
https://github.com/freedict/fd-dictionaries/blob/master/shared/lg1-lg2/lg1-lg2.tei
[3] https://www.tei-c.org/release/doc/tei-p5-doc/en/html/DI.html#DIMV
[4] https://en.wikipedia.org/wiki/ISO_12620
[5] https://en.wikipedia.org/wiki/Register_(sociolinguistics)#ISO_standard
[6] https://www.iso.org/standard/69550.html
[7]
https://www.tei-c.org/release/doc/tei-p5-doc/en/html/ref-lbl.html#index-egXML-d53e79101
[8] https://www.tei-c.org/release/doc/tei-p5-doc/en/html/CO.html#CONAAB
[9] https://dict.tu-chemnitz.de/doc/syntax.html
