Task #06: Word-Sense Disambiguation of Prepositions

Mailing list

We've set up a mailing list to facilitate discussion and information exchange about this task.

To join the discussion group on this task, enter your e-mail here:

After you've signed in you can browse the e-mail discussion on the task.

Datasets and Formats

The set of prepositions included in the SemEval-2007 task is drawn from The Preposition Project. The prepositions include the 17 most common prepositions, as well as 20 additional very common prepositions, as shown in the accompanying table. This table shows the number of senses for each preposition and the total number of instances that have been tagged (disambiguated) in The Preposition Project. Except for below, beyond, and near, all these prepositions are included in the SemEval-2007 test set, with approximately two-thirds of the instances used for training and the remaining one-third as the test set. The training and test sets will be generated randomly from the available instances.

Each preposition included in the task will have a training set and a test set. Each instance is a sentence drawn from the FrameNet sentences and is included exactly as transcribed there, except that the target preposition to be disambiguated is tagged with a head element. The set of instances for each preposition will be contained in an XML-valid file, with the top element lexelt (e.g., <lexelt item="for.p">). Each instance will consist of a sentence identified as the context; the training set will also contain an answer element, as in the following example.

<instance id="for.p.fn.331141" docsrc="FN">

<answer instance="for.p.fn.331141" senseid="2(2)"/>

<context>

He used to work as a policeman and , on the few occasions when he was in extreme danger , automatically did whatever had to be done without any thought <head>for</head> his own safety .

</context>

</instance>

The sense identifier is a number assigned in The Preposition Project using the sense inventory from the Oxford Dictionary of English. The full sense inventory for each preposition will be provided and will include a mnemonic semantic relation/role name, and properties of the preposition complement and the head or attachment point of the preposition. The answer key for the training set will include not only the sense identifier, but also a FrameNet frame and frame element name, as assigned by the FrameNet lexicographers. Participants may use the FrameNet sentence identifier to make use of other information generated by the FrameNet lexicographers.

A trial dataset has been prepared for the instances and senses of the prepositions below, beyond, and near. This dataset consists of sense inventories (*.defs.xml), instance files (*.sents.xml), and answer keys (*.sents,key). The sense inventories were generated from data available in the The Preposition Project. The help file from the Online TPP is included in the trial dataset.

The sense inventories include some extraneous information from the data files used in TPP. Essentially, the relevant information for each sense is contained in S (sections) tags; each sense has a sense identifier (senseid). ODE data are contained primarily is definition fields (df), examples (ex and exg), and grammar groups (gg). TPP generated data are contained in the following fields (which are described in detail in the accompanying help file): srtype (semantic relation or role name), qsyn (Quirk syntax), qpar (Quirk paragraph), cprop (complement properties), aprop (attachment properties), frfes (Frame::Element pairs), opreps (other prepositions - short), fepreps (other prepositions - long), srel (sense relations), and com (comments). Some of these fields may be empty for some senses; minimally, there will be information in srtype, cprop, and aprop fields. An important consideration for the SemEval task is that senses in the sense inventory that do not have an accompanying set of frfes will have no instances in the sentences. In many cases, but not all, the TPP lexicographer provided a treatment of a preposition’s senses (including near and beyond, but not below); these treatments are included in the trial dataset. In the case of more common prepositions, these treatments may be somewhat elaborate.

Additional information about each sense is also available from The Preposition Project, as shown in the Online TPP. The data file used in the Online TPP will be available to participants.

Evaluation

The task will be evaluated in the same manner as previous Senseval lexical sample tasks, following the same methodology for evaluation (including the use of the same evaluation scripts, with sense tagging available for both fine-grained and coarse-grained disambiguation). All the necessary resources are already available to potential participants. Systems will be evaluated against the manually annotated unseen test sets, using the following measures. Accuracy is defined as the percentage of correctly classified instances in the whole set. Precision, recall, and balanced f-score will then be used to assess performance with respect to each annotation category. An evaluation script will be provided. We encourage supervised and unsupervised systems as well as partial submissions.

Baseline disambiguations of the prepositions have been prepared. The first baseline picks the first sense for each preposition, specifically, sense 1(1). The second baseline picks the most frequent sense in the training data. Three tables show these results. The first shows the first sense baseline. The second table shows the most frequent sense, as determined from the training data. The third table shows the most frequent sense baseline. The baseline tables show the preposition, its number of senses, and the fine-grained and coarse-grained precision scores when the specified sense is selected. The frequency table shows the most frequent sense for each preposition, as well as the frequency with which it appears in the training data. (Please note that, since the selection of the training and test sets was performed randomly to achieve a two-thirds, one-third split, the frequencies in the test data may not mirror those in the training data.)

Download area

This section will contain evaluation software, useful scripts, complementary materials, baseline systems, etc. but not the datasets proper. The datasets will be available at the main site for download.

System and Results

This section will be completed after the competition.

References

Litkowski, K. C. and O. Hargraves (2005). "The Preposition
Project", Proceedings of the Second ACL-SIGSEM Workshop on The
Linguistic Dimensions of Prepositions and their Use in Computational
Linguistics Formalisms and Applications, April 19-21, Colchester,
England: University of Essex (pdf)
Litkowski, K. C. and O. Hargraves (2006). "Coverage and
Inheritance in the Preposition Project", Proceedings of the
Third ACL-SIGSEM Workshop on Prepositions, April 3, Trento,
Italy (pdf)

 For more information, visit the SemEval-2007 home page.