Shared Annotation Task

The EventStory workshop will include a session at the end of the day to discuss a group annotation exercise. Based on the discussion of the exercise at the 2017 workshop (paper) and partially inspired by the goals of DARPA’s AIDA program (paper), this year our exercise will focus on conflicting storylines and the effects of varying viewpoints/sources.

A challenge to developing and maintaining an understanding of events is that users are often receiving information from many disparate sources, reflecting different genres, modalities, as well as different stances on and accounts of what are potentially the same events. We invite participants to consider...

1. What data sources provide provenance/stance information (e.g., metadata, bylines)?
2. What linguistic and/or rhetorical phenomena evidence a particular stance in the account of an event (e.g., hedging, speaking verbs)? How cross-linguistically general/unique are these?
3. What lexical resources might be particularly helpful in storing such phenomena?
4. What annotations might be needed to train systems to recognize that varying/conflicting storylines reference the same event or events?
5. What annotations might aid in the assessment of an account’s veracity or in our confidence of an event sequence?
6. What annotation schemes currently exist and what gaps might be present in these?
7. What downstream effect might inclusion/exclusion of provenance/stance features have on other applications (e.g., question-answering)?
8. How important/relevant is it for the events/stories research communities to address these issues?

Please consider these questions with respect to the sample documents from the event-event relations corpus (Hong et al. 2016 paper) and the vaccination corpus (link TBD).

• MH17 document 1
• MH17 document 2
• MH17 document 3


• Vaccination document 1
• Vaccination document 2

Please prepare the following types of annotation for discussion in the final block of the EventStory workshop:

1. Underline and provide a unique # ID to indicators of...
   
   • Stance-holders
   • The topic/subject of their stance
   • The actual stance/claim
   • Textual triggers/evidence of stance-taking
2. Separately in the text document or in a spreadsheet, indicate the relations between your underlined indicators (see example below); add Author and implicit information as needed.

EXAMPLE (strictly illustrative of possibilities -- not exhaustive):

Ukrainian authorities have released what they-0 say-1 are intercepted phone conversations-2 between pro-Russian separatists and what appear-3 to be Russian military officers-4 saying-5 that separatists shot down Malaysia Airlines flight MH17-6.

Stance-holder	Topic/Subject	Stance/Claim	Triggers
0		2	1
Author	Veracity of (2)		1, 3
4		6	5

Additional annotation could relate to veridicality. For example:

• Indicate speaker’s degree of certainty regarding their description of event
• Underline and tag phrase(s) related to their certainty
• Provide event ID that links that phrase to the event (E.g. "I think I saw them enter the building".)

Please send annotations and observations on questions 1-8 to Susan.Brown@colorado.edu before the workshop, and we will incorporate them into our plans for the discussion session.