Not all tokens can be rendered as words easily. Their context may affect the way they are to be pronounced. For example in the utterance
On May 5 1985, 1985 people moved to Livingston.
the tokens "1985" should be pronounced differently, the first as
a year, "nineteen eighty five" while the second as a quantity "one
thousand nine hundred and eighty five". Numbers may also be pronounced
as ordinals as in the "5" above, it should be "fifth" rather than
"five".
Also, the pronunciation of certain words cannot simply be found from their orthographic form alone. Linguistic part of speech tags help to disambiguate a large class of homographs, e.g. "lives". A part of speech tagger is included in Festival and discussed in POS tagging. But even part of speech isn't sufficient in a number of cases. Words such as "bass", "wind", "bow" etc cannot by distinguished by part of speech alone, some semantic information is also required. As full semantic analysis of text is outwith the realms of Festival's capabilities some other method for disambiguation is required.
Following the work of yarowsky96 we have included a method for identified tokens to be further labelled with extra tags to help identify their type. Yarowsky uses decision lists to identify different types for homographs. Decision lists are a restricted form of decision trees which have some advantages over full trees, they are easier to build and Yarowsky has shown them to be adequate for typical homograph resolution.
Festival offers a method for assigning a token_pos feature to
each token. It does so using Yarowsky-type disambiguation techniques.
A list of disambiguators can be provided in the variable
token_pos_cart_trees. Each disambiguator consists of a regular
expression and a CART tree (which may be a decision list as they have the
same format). If a token matches the regular expression the CART tree
is applied to the token and the resulting class is assigned
to the token via the feature token_pos. This is done
by the Token_POS module.
For example, the follow disambiguator distinguishes "St" (street and saint) and "Dr" (doctor and drive).
("\\([dD][Rr]\\|[Ss][tT]\\)"
((n.name is 0)
((p.cap is 1)
((street))
((p.name matches "[0-9]*\\(1[sS][tT]\\|2[nN][dD]\\|3[rR][dD]\\|[0-9][tT][hH]\\)")
((street))
((title))))
((punc matches ".*,.*")
((street))
((p.punc matches ".*,.*")
((title))
((n.cap is 0)
((street))
((p.cap is 0)
((p.name matches "[0-9]*\\(1[sS][tT]\\|2[nN][dD]\\|3[rR][dD]\\|[0-9][tT][hH]\\)")
((street))
((title)))
((pp.name matches "[1-9][0-9]+")
((street))
((title)))))))))
Note that these only assign values for the feature token_pos and
do nothing more. You must have a related token to word rule that
interprets this feature value and does the required translation. For
example the corresponding token to word rule for the above disambiguator
is
((string-matches name "\\([dD][Rr]\\|[Ss][tT]\\)")
(if (string-equal (item.feat token "token_pos") "street")
(if (string-matches name "[dD][rR]")
(list "drive")
(list "street"))
(if (string-matches name "[dD][rR]")
(list "doctor")
(list "saint"))))
Festival offers some support for building disambiguation trees. The basic method is to find all occurrences of a homographic token in a large text database, label each occurrence into classes, extract appropriate context features for these tokens and finally build an classification tree or decision list based on the extracted features.
The extraction and building of trees is not yet a fully automated process in Festival but the file festival/examples/toksearch.scm shows some basic Scheme code we use for extracting tokens from very large collections of text.
The function extract_tokens does the real work. It reads the
given file, token by token into a token stream. Each token is tested
against the desired tokens and if there is a match the named features
are extracted. The token stream will be extended to provide the
necessary context. Note that only some features will make any sense in
this situation. There is only a token relation so referring to words,
syllables etc. is not productive.
In this example databases are identified by a file that lists all
the files in the text databases. Its name is expected to be
bin/DBNAME.files where DBNAME is the name of
the database. The file should contain a list
of filenames in the database e.g for the Gutenberg texts the
file bin/Gutenberg.files contains
gutenberg/etext90/bill11.txt
gutenberg/etext90/const11.txt
gutenberg/etext90/getty11.txt
gutenberg/etext90/jfk11.txt
...
Extracting the tokens is typically done in two passes. The first pass extracts the context (I've used 5 tokens either side). It extracts the file and position, so the token is identified, and the word in context.
Next those examples should be labelled with a small set of classes which identify the type of the token. For example for a token like "Dr" whether it is a person's title or a street identifier. Note that hand-labelling can be laborious, though it is surprising how few tokens of particular types actually exist in 62 million words.
The next task is to extract the tokens with the features that will best distinguish the particular token. In our "Dr" case this will involve punctuation around the token, capitalisation of surrounding tokens etc. After extracting the distinguishing tokens you must line up the labels with these extracted features. It would be easier to extract both the context and the desired features at the same time but experience shows that in labelling, more appropriate features come to mind that will distinguish classes better and you don't want to have to label twice.
Once a set of features consisting of the label and features is created it is easy to use wagon to create the corresponding decision tree or decision list. wagon supports both decision trees and decision lists, it may be worth experimenting to find out which give the best results on some held out test data. It appears that decision trees are typically better, but are often much larger, and the size does not always justify the the sometimes only slightly better results.