INSEMTIVES - Incentives for Semantics

OntoTube

OntoTube is a two player quiz game for annotating videos. Both players have to answer questions about the video. The answers they provide are used for data generation.

Data output : Semantic annotations using a simplified domain ontology.
Validation: Consensus, majority.
Contact : katharina.siorpaes@sti 2 .at
PopVideo

PopVideo is a two multi player game for annotating videos. All the players watch the same movie clip simultaneously and have to type in what they see. They earn points for similar answers. Additionally, the closer the answers are in respect to time the more points they earn.

Data output : Annotations of the objects in a video.
Validation: Consensus, majority.
Contact : support@gwap.com
Videotaggame

Yahoo 's Videotaggame works basically like PopVideo except that it allows to re - watch the video clip afterwards and see who entered what and when.

Data output : Annotations of the objects in a video.
Validation: Consensus, majority.
Contact : http://labs.yahoo.com/contact
TAG - The Annotation Game

The Annotation Game works similar as PopVideo.

Data output : Annotations of the objects in a video.
Validation: Consensus, majority.
Contact : annotationgame@gmail.com
Waisda

Waisda works like TAG and PopVideo. However, it uses live video shows to be annotated.

Data output : Annotations of the objects in a video.
Validation: Consensus, majority.
Contact : waisda@beeldengeluid.nl

Google Image Labeler

The Google image labler is a two player game. In each game round, both players get to see the same picture and have to type in the objects they see. As soon as they agree on a term they earn points and proceed to the next picture. Additionally, frequent answered words are put on a black list. Typing in a blacklisted word is not rewarded.

Data output : Annotations describing the objects in an image .
Validation: Consensus, majority.
Contact : http://www.google.com/support/bin/request.py
ESP Game

Works like the Google image labeler, except for a nicer interface.

Data output : Annotations of the objects in an image .
Validation: Consensus, majority.
Contact : support@gwap.com
Squigl

Squigl is a two player game for locating the position of objects in images . Both players get to see the same image and an object. They get a short period of time to outline the object on the image . The more similar their path is the more points they get. Objects are taken from the tags retrieved by the ESP game.

Data output : Locations of the objects in an image .
Validation: Consensus, majority.
Contact : support@gwap.com
Matchin

Matchin is a two player game for finding beautiful images . Both players get to see two images and have to agree on which they like better.

Data output : The "beauty" of images .
Validation: Consensus, majority.
Contact : support@gwap.com
Flip It

Flip It is a game for pointing out similarly looking images . It is based on the well known Memory game . Players have to identify images that show similarly looking objects out of 16 images that are covered like in Memory. To do so they are allowed to use a so called "life line". A life line reveals all images for a short period amount of times.

Data output : Annotations of the objects in the video.
Validation: Majority.
Contact : support@gwap.com
Magic Bullet

Magic Bullet is a game used to streamline the robustness evaluation of CAPTCHAs. CAPTCHAs are typically evaluated by machine learning approaches, however they usually require sample images which are already annotated with their meaning in order to train them. This is where Magic Bullet comes into play, the game uses its players to annotate CAPTCHAs and, hence, let them create the valuable training sets. The game is played by two teams and shows to each team a CAPTCHA segment. The first team which is able to agrees on the meaning of the CAPTCHA gets awarded with points. Magic Bullet could also be used as a game to improve the players typing skills.

Data output: Annotations of images
Validation: Consensus, majority
Contact: Jeff.Yan@ncl.ac.uk
TagCaptcha

TagCaptcha is a CAPTCHA system with a purpose. CAPTCHAs are simple tests used to check if an actor is human or not. In the case of TagCaptcha, CAPTCHAs constitute of the task of describing the content of images. If the actor is able to describe the content of the image successfully, he passes the test otherwise not. Image recognition is simple for the human brain, however, machines still have lots of troubles with this task. Nowadays, CAPTCHAS are typically randomly generated words which are displayed in a distorted way and solving them has no additional benefits. TagCaptcha, however, shows images/photos and let humans describe them. If several actors provide the same description to an image, the description is accepted as truth. This way a set of images is annotated as side-effect by the TagCaptcha CAPTCHA system, which can be considered as an additional benefit.

Data output: Annotation of images
Validation: Majority
Contact: donn.morrison@unige.ch
Picture This

Picture This is a two player game with the purpose of semantically annotating images. If a player starts the game he is matched with another player or if not available with a bot. Both players see a textual query and a set of images. The players have to select an image which suits best the given query and if both players select the same image they get points.

Data output: Annotations of images
Validation: Consensus, majority
Contact: pauben@microsoft.com

HerdIt

HerdIt is a multi player game for annotating audio files. It consists of several different mini games which are all about parts of music clips. All players have to listen to the same tune and afterwards answer questions about it. The more similar your answer is compared to the others the more points you get.

Data output : Annotations describing audio files.
Validation: Consensus, majority.
Contact : lukeinusa@gmail.com
Tag a tune

Tag a tune is a two player game. Both players get to hear a tune which is either the same or a different piece of music. They have to describe what they are hearing and based on these descriptions decide whether you are listening to the same tune or not.

Data output : Annotations describing audio files.
Validation: Consensus, majority.
Contact : support@gwap.com

Verbosity

Verbosity is a two player game for describing concepts with other phrases. One player gets a concept which they have to describe by using pre-defined phrases. The other player has to guess the concept based on the descriptions of their partners. After each game round these two roles are swapped, i.e. the guesser becomes the describer and vice versa.

Data output : Descriptions of concepts.
Validation: Consensus, majority.
Contact : support@gwap.com
Phrase Detective

Phrase Detective is a game that aims to identify relations words and phrases. A player gets different questions about these relationships in one or more sentences.

Data output : Relationships between words and phrases.
Validation: Validation by asking other players , majority.
Contact : jchamb@essex.ac.uk
PhraTris

PhraTris is a game which uses its players for annotating sentences with syntactic dependencies. The purpose of the game is, hence, to classify words in sentences according to their syntactical classes (e.g. adjective, adverb, noun, numeral, etc.). The game is based on the Tetris concept, but instead of blocks, which must be arranged by the player, words must be arranged. The arrangement should form sentences which make sense. Validation is done by consensus and majority.

Data output: Annotation of sentences with syntactic dependencies
Validation: Consensus, majority
Contact: http://galoap.codeplex.com/team/view
Concept Game on Facebook

Concept Game is another GWAP (Game With A Purpose) which tries to gather common sense knowledge from players. The game works like a slot machine, showing three slots, each slot shows a part of a sentence. The slot machine gives the player a sentence and he/she has to decide if the presented sentence makes sense or not. The game is based on time, which runs out if the player is to slow. Time is increased if the player is able to detect a sentence which makes sense. The game validates the player arrangements by majority and consensus. Furthermore, it uses a scoreboard (monthly reseted), a hall of fame, monetary prizes and (player-)leveling mechanics including associated avatars as incentives.

Data output: Players annotate a common sense dataset
Validation: Majority
Contact: Amaç Herdağdelen (amac@herdagdelen.com) and Marco Baroni
The FACTory (CycGame)

The FACTory is a GWAP with the goal to use player input in order to train an AI about common sense knowledge. The AI produces questions and the players answer them with either: (1) True, (2) False, (3) Don't know or (4) Doesn't make sense. Validation is done by player consensus and majority. Each round consists of several questions which the player has to answer. The player gets points for answering questions, most for questions which are consensually answered with others. A high-score board is used to give the player an additional incentive to play the game.

Data output: Players annotate a common sense dataset
Validation: Majority
Contact: info@cyc.com
WhoKnows?

In WhoKnows?, players evaluate the accuracy of DBpedia knowledge answering questions automatically extracted from DBpedia triples. The players are provided with questions that they should answer correctly and as fast as possible. Each question has a closed set of answers, including correct and incorrect choices. Correct choices are rewarded. The faster a player answers the questions the more points he gets.

Data output : Validated DBpedia assertions, ranking of facts..
Validation: Consensus, majority.
Contact : hpi-info@hpi.uni-potsdam.de

Page Hunt

Page Hunt is a game with the purpose to index web sites and hence to improve the search index of a search engine (Microsoft Bing). The player gets assigned a random web site and is asked to describe it with keywords. The game then shows the player the top 5 page hits for the entered keywords and depending on how high ranked the previously assigned web page is in the result set, points get assigned to the player. The higher ranked the site is, the more points the player gets awarded. Page Hunt also uses a bonus system to give the player additional points for different occasions, as for instance if he does not use certain keywords to describe a web site (tabu words). The player has also the option to skip an assigned web site and hence gets assigned another one. It is also possible to mark pages as flawed if they are not rendered properly.

Data output : Annotation of web sites
Validation: Majority
Contact : phfeed@microsoft.com

OntoPronto

OntoPronto is a two player quiz game for building Proton grounded domain ontology. Players get to read the abstract of a random Wikipedia article and have answer questions about the nature of this article in respect to Proton.

Data output : A Proton grounded domain ontology.
Validation: Consensus, majority.
Contact : katharina.siorpaes@sti 2 .at
Virtual Pet Game

Virtual Pet Game is game a which aims at constructing a semantic network that encodes common knowledge (a Chinese ConceptNet equivalent). The game is built on top of PPT, a popular Chinese Bulletin Board System which is accessible through a terminal interface. Each player has a pet which he should take care of in order to satisfy its needs, otherwise it could die. In order to take care of the pet (e.g. buy food etc.), the player has to earn points. To get points he has to ask and answer questions to his pet. The pet, in this game, is just a substitute for other players which receive the questions/answers and respond or validate them. This way a common sense network is constructed behind the game. Question and answers are given by players using given templates (e.g. subject, predetermined relation, object)

Data output: Semantic network which encodes common knowledge
Validation: Majority
Contact: a33kuo@gmail.com
Rapport Game

The purpose of the Rapport Game is the same as the one of the Virtual Pet Game. Both games exploit player labor in order to construct a semantic network that encodes common knowledge (a Chinese ConceptNet equivalent). Rapport Game, however, is built on top of FaceBook and uses direct interaction between its players. The players ask other players, in which they are interested in, questions. The other players answer them and the answers are evaluated by majority. In order to give players further incentives rapport points are earned by specific player action such as asking/answering questions. Based on the delivered facts the game constructs in the background a semantic network of common knowledge. Question and answers are given by players using given templates (e.g. subject, predetermined relation, object).

Data output: Semantic network which encodes common knowledge
Validation: Majority
Contact: a33kuo@gmail.com
Guess What?!

Guess What?! is a semantic game with a purpose which uses its players for the creation of formal domain ontologies from Linked Open Data. Each game session is based on a seed concept. This concept was previously chosen manually (but could also be randomly selected). The game algorithm then takes the seed concept (e.g. banana) and tries to find a matching URI in DBpedia, Freebase and OpenCyce. Then as much information as possible about the concept is gathered (e.g. superclass, object properties) by querying related (linked) RDF repositories. The result is cached locally for faster access. The resulting labels/URIs are analyzed by simple natural language processing in order to identify expressions which can be translated into logical operators and braking down complex descriptions into small fragments. The small fragments are then weighted by a generality and confidence value. If this is done a game session can start and fragments (e.g. fruit, yellowish, elongated) are presented to the players in each round. The rounds start with the most general fragment and in each round a more specific fragment is connected to it through a logical operator (e.g. fruit and yellowish and elongated). In each round the players have to guess a concept which is described by the fragments. The information provided by the players can be used to create an ontology.

Data output: Formal domain ontologies
Validation: Consensus, majority
Contact: voelker@informatik.uni-mannheim.de

Wordhunger Freebase

Wordhunger is a web application game that maps concepts from Wordnet synonym sets to cocepts of Freebase datasets. Players simply map one concept after the other and earn points for doing so.

Data output : Mappings from Wordnet synonym sets to Freebase dataset concepts.
Validation: Majority.
Contact : paritosh@cs.northwestern.edu
SpotTheLink

SpotTheLink is a two player quiz game for aligning ontologies, e.g. DBpedia and Proton. Both players get a concept from the one ontology. In the first step they have to agree on a matching concept of the other ontology and in the second step they have to agree on the relationship between these concepts.

Data output : A SKOS mapping between the ontologies' concepts.
Validation: Consensus, majority.
Contact : katharina.siorpaes@sti 2 .at

Collabio

Collabio is a social tagging game within the online social network Facebook that encourages friends to tag one another. Collabio’s approach of incentivizing members of the social network to generate information about each other produces personalizing information about its users. We report usage log analysis, survey data, and a rating exercise demonstrating that Collabio tags are accurate and augment information that could have been scraped online.

Data output : Basically a set of triples (friend, tag, individual) which associate a tag given by a friend to an individual
Validation: Individual can manage/validate tags, collective tagging
Contact : http://research.microsoft.com/en-us/um/redmond/groups/cue/collabio/

Wordhunger WikipediaGame

WikipediaGame is a Wiki racing game. One user creates a "race", i.e. a rout of links between two Wikipedia articles. Players have to find the route with as few clicks as possible.

Data output : -
Validation: -
Contact : paritosh@cs.northwestern.edu
WikipediaMaze

Another Wikiracing game that works similar as WikipediaGame.

Data output : -
Validation: -
Contact : micah@codingcontext.com
Wikispeedia

A third Wikiracing game that uses its results for calculating the semantic distance between two Wikipedia articles.

Data output : Semantic distance between two articles.
Validation: Majority.
Contact : west@cs.mcgill.ca

Games for Semantic Content Creation

Games for video annotation

Games for image annotation

Google Image Labeler

Games for Audio Annotation

Games for text annotation

Games for Web site annotation

Games for ontology building

Games for ontology alignment

Games for social-tagging

Wikiracing Games