How Do You Know What Each Turk Hit Is Worth

Mturk Qualifications are a useful way to go on tabs on workers and restrict workers' ability to work on your HITs based on previous work they have done. Mturk provides a number of built-in qualifications, like a worker's land of origin, their overall HIT credence charge per unit, or the number of HITs they have had approved. However, mturk too allows requesters to create their ain qualifications, which can have any value from 0 to 100 inclusive. New workers tin then exist screened according to whether their qualification score is greater than, less than, equal to, or not equal to whatsoever given score for a qualification. This means that you lot tin can create essentially arbitrary conditions for including or excluding workers.

Unfortunately, mturk does non provide a qualification exam equivalent to "Worker has/does not have a value for this qualification". You can test whether a worker has a particular score, but you cannot test for if they take a score at all. And once a qualification is added to an HIT, workers cannot participate in that Striking unless they havesome value for that qualification. Workers can, however, asking that a qualification be granted to them. This can happen two ways:

1) workers tin search qualifications. This is useful if a worker knows that a specific qualification will be required in guild to complete an HIT they have their centre on. Alternatively, this method allows workers to browse qualifications and request any or all of the qualifications available to them.

2) Perhaps more commonly: when a worker sees an Hit they are not qualified for, as long as the qualification is i created by the requester, and not 1 of mturk'due south built-in qualifications (which depend on a worker'due south location and operation), the worker will run into a "request qualification" button in place of the "have Hitting" push. If they expand the hit box, they volition too see a description of the qualification and their value for that qualification (see screenshot below).

(Automatically) Granting Qualifications

On the requester side, qualification requests are, by default, added to a queue. Requests for qualification must be manually approved by the requester, and a value must be manually added each fourth dimension a qualification is granted.

There are ii unlike ways to deal with granting qualifications: by automatically granting qualifications, or by creating a qualification test, which creates a specific qualification examination Hitting blazon and automatically assigns workers a qualification value based on examination results. Unfortunately, neither of these options is bachelor via the requester GUI -- to create either of these qualification types, you'll need to utilize the command line.

Using the command line with mturk requires an Amazon Web Services business relationship, which can be created at aws.amazon.com. You might fifty-fifty have one set initially merely past setting upwardly an mturk requester account. In improver to an AWS account, you will need to download either the Mechanical Turk Command Line Interface (documentation here), or a library for a programming language to interact with AWS, like the python boto repository (documentation here). You will have to either create or update a qualification type using either method (the command line interface requires you to create a file, while the boto repository is but a office with a number of different arguments). For either method, you are going to want to take the value autogranted=truthful, and fix a value for the qualification when information technology is automatically granted (either as a single line in your CLI file, or as an argument in your boto command -- encounter here and here, respectively).

Updating Qualification Scores to exclude Workers

Using these tools, you lot can set up a qualification to exist automatically granted to workers who request it, allowing them to take your Hit. But what virtually afterward they've done it, and you lot want to exclude them from participating in farther HITs?

I method is to reassign values to every worker by paw, which tin be managed by going to the results folio for a batch, clicking on each worker'due south ID, and assigning them a new score individually. Notwithstanding, this is an arduous procedure.

Using a batch .csv

Some other solution is to batch upload a .csv (comma separated values) file containing updated scores for all theworkers to whom yous wish to assign the qualification. For more details on doing this, see p. 39 of this .pdf. This can also be relatively arduous, as you must either catechumen a results file to the format required for uploading new qualification scores, or friction match workers from batches with your master listing of all workers (which can be obtained in the requester GUI past going to "manage", and then "workers", then "Download .csv".

Programmatic methods

Finally, you can update workers' scores past using either the CLI or boto.

Using the CLI

Using the CLI, y'all tin can update qualification scores for workers using a tab-delimited file with only the worker ID and the score you wish to assign to the worker (that makes it very easy to create these files in excel, especially if you desire to assign the same score to many workers). Run into hither for more detailed instructions on the specific commands and file types involved with this procedure.

To perform the same operation using boto, you would only write a for loop which loops through a list of worker ID's and uses the function

update_qualification_score (

qualification_type_id , worker_id , value

)

to update your scores.

How Do You Know What Each Turk Hit Is Worth

Source: http://www.josephjaywilliams.com/mturk/how-to-prevent-workers-from-participating-in-experiments-multiple-times/creating-auto-grant-qualifications-to-exclude-workers-from-participating-multiple-times

Share this post