WikiQuestions
A large question dataset generated from Wikipedia sentences
Pipeline
Preprocessing
The input for the QG-system is a Wikipedia dump with recognized Freebase Easy entities. The input is preprocessed by annotating each entity with its category. Entity mentions are then presented in the format [entity name|entity category|original word in the sentence]. Additionally, flawed entity recognition is improved by merging identical entities that appear sequentially in the sentence (e.g. An [An_American_in_Paris|Composition|American] in [An_American_in_Paris|Composition|Paris] becomes the single entity [An_American_in_Paris|Composition|An American in Paris]). The following sentence from the preprocessed Wikipedia dump will be used as running example to explain the system pipeline.
In 1989 , at the age of eight , [Anna_Kournikova|Person|Kournikova] began appearing in junior tournaments , and by the following year , was attracting attention from [Spartak_Tennis_Club|unknown|tennis] scouts across the [World_TeamTennis|Organisation|world] .
Parsing
The Wikipedia entity-sentences are parsed using the spaCy dependency parser. Entity mentions are replaced by the entity name which is treated by the parser as a single word. The parse is saved in CoNLL-2007 format with an additional column for entity mentions as shown in the example below:
address word tag head rel entity ... 10 Anna_Kournikova NNP 11 nsubj ("Anna_Kournikova", "Person", "Kournikova", 10) 11 began VBD 0 root None 12 appearing VBG 11 xcomp None 13 in IN 12 prep None 14 junior JJ 15 amod None 15 tournaments NNS 13 pobj None ...
Question Generation
For question generation, the parsed sentences are first preprocessed. Preprocessing steps include:
- Removal of clauses that are not directly dependent on the main clause and are separated from it by comma or semicolon (a clause is considered directly dependent on the main clause if it contains a subject, object or a preposition whose head is the root node of the sentence)
- Recovery of pronouns that were recognized as entities but are supposed to appear as simple pronouns in the question
- Validity check of the parse (e.g. is the root node a verb, does a subject exist, ...)
The example sentence after the preprocessing looks like this:
In 1989 , at the age of eight , [Anna_Kournikova|Person|Kournikova] began appearing in junior tournaments
The system then selects an entity within the sentence for which to ask the question. This answer entity is either the subject of the sentence (nsubj / nsubjpass), a direct object (dobj) or an object of a preposition (pobj). In addition to entity mentions, years or months can also be selected as answer entity since they are not marked as entities in the Wikipedia dump. For the example sentence, two entities are selected as answer entities. An object of a preposition and the subject:
1989
[Anna_Kournikova|Person|Kournikova]
Next, the question word is determined. Which question word is chosen depends on the category of the answer entity as well as its role in the sentence. E.g. if it is the subject of the sentence and its category is in a list of selected categories such as Person or Fictional Character, the question word is Who. The table below gives an overview of which question word is chosen when.
| nsubj / nsubjpass | dobj | pobj | |
|---|---|---|---|
| Who-category | Who | Who | - |
| What-category | What Which <type> | What Which <type> | - |
| Location | What Which <type> | What Which <type> | Where |
| Year / Month | - | - | When |
As can be seen in the table, for answer entities that fall into a What-category, Which <type> can also be chosen as question word. The type that is inserted for <type> is taken from the list of Freebase types of the answer entity. However, only certain handpicked types are inserted. The entity Lamborghini for example has the Freebase types Automobile Company, Business Operation, Consumer company, Employer, Literature Subject, Manmade Object, Organisation, Thing and Topic. However, a question that starts with e.g. Which manmade object would rarely be encountered in real life. Therefore, if Lamborghini is the answer entity for a question, aside from the question word What only the question words Which automobile company, Which business operation and Which organisation will be chosen.
The question words for the answer entities of the example sentence are:
1989 --> "When"
[Anna_Kournikova|Person|Kournikova] --> "Who"
In the following, various transformations are performed to turn the sentence into a question. If the answer entity is the subject of the sentence, these transformations are straightforward. First, every word that depends on the subject is removed from the sentence (e.g. adjectives). The subject itself is replaced by the selected question word and set as first word of the question. Then, everything that appears after the subject in the sentence is appended to the new question string except for some words that appear directly before the predicate and only make sense in the context of the previous sentences, e.g. also, however, then. The subject question of the example sentence is:
Who began appearing in junior tournaments ?
Finally, some post-processing is performed. This includes:
- Counteract flawed entity recognition by merging identical entities that appear sequentially in the question string
- Add missing possessive s for entities where the original word is a possessive pronoun
- Remove dangling hyphens or conjunctions
- Filter out questions that do not contain any entity
Since our example subject question does not contain any entity, it is filtered out.
For object questions, the transformations are more complex. First, as for subject questions, every word that depends on the object is removed from the sentence. In addition, if the object is the object of a preposition, this preposition is removed as well. The object itself is then replaced by the question word and set as first word of the question. The predicate is determined from the parse and a lemmatizer (WordNet Lemmatizer) is used to get the infinitive of the predicate. If no auxiliary verbs exists in the original sentence and the predicate is no form of to be, one of the auxiliary verbs do, does and did is chosen depending on the tense, person and number of the predicate. The auxiliary verb is appended to the question, followed by the subject and its dependencies. Next, possible additional auxiliary verbs are appended, as well as the infinitive of the predicate and negations, particles (e.g. shut down, hand over) and adverbial modifiers (e.g. publicly broadcast, recorded live) that belong to the predicate. Now, if the answer entity appears before the predicate in the sentence, everything following the predicate is appended to the question. This results in the following object question for our example sentence:
When did [Anna_Kournikova|Person|Kournikova] begin appearing in junior tournaments ?
If the answer entity appears after the predicate, everything between the predicate and the answer entities head is appended. That way, if the sentence structure of our example sentence had been different, the outcome would still be the same:
[Anna_Kournikova|Person|Kournikova] began appearing in junior tournaments in 1989 at the age of eight.
--> When did [Anna_Kournikova|Person|Kournikova] begin appearing in junior tournaments ?
Finally, clausal complements (e.g. He reported that they had left, She supported what they did), open clausal complements (e.g. He considered him honest, She likes to swim) and prepositional phrases (in 2018, at the Eiffel Tower) are appended if they are not already part of the question. To finalize the questions, the same post-processing as for subject questions is performed.
Using this system, 44,045,711 questions are generated from 140,879,948 Wikipedia sentences.
Filtering
Once the questions are generated, several filters are applied to improve the quality of the final questions. The following table shows the filter criteria together with the number of questions that are discarded if solely the corresponding filter is applied over all generated questions.
| Filter | # discarded questions | % discarded questions |
|---|---|---|
| Freebase-Connection-Filter Entities in the question don't have a connection in Freebase (directly or via one mediator entity) | ~16,375,174 | ~36.7% |
| Uppercase-Filter Question contains uppercase letters outside of entity mentions (often means wrong entity recognition) | 15,126,563 | 33.9% |
| Lowercase-Filter Question contains entities whose original word consists only of lowercase letters (often means wrong entity recognition) | 14,491,569 | 32.5% |
| MID-Question-Filter An entity name in the question can't be mapped to any Freebase MID | 11,070,738 | 24.8% |
| 2-Entities-Filter Question contains more than two entities | 8,547,368 | 19.1% |
| MID-Answer-Filter Answer entity name can't be mapped to any Freebase MID | 5,097,149 | 11.4% |
| It-Answer-Filter Original word of the answer entity is "it" (often means wrong entity recognition) | 3,562,766 | 8.0% |
| Answer-in-Question-Filter Answer entity appears in the question | 3,551,238 | 8.0% |
| Comma-Filter Question contains a comma | 2,912,552 | 6.5% |
| Context-Word-Filter Question contains a word that needs the context of previous sentences in order to make sense (e.g. they, there, these, ...) | 1,989,359 | 4.5% |
| It-Question-Filter Original word of an entity in the question is "it" | 1,814,137 | 4.1% |
After the filter process, the question dataset consists of 4,390,597 questions.