Show Less
Restricted access

Testing ESL Sociopragmatics

Development and Validation of a Web-based Test Battery


Carsten Roever, Catriona Fraser and Catherine Elder

Testing of second language pragmatics has grown as a research area but still suffers from a tension between construct coverage and practicality. In this book, the authors describe the development and validation of a web-based test of second language pragmatics for learners of English. The test has a sociopragmatic orientation and strives for a broad coverage of the construct by assessing learners’ metapragmatic judgments as well as their ability to co-construct discourse. To ensure practicality, the test is delivered online and is scored partially automatically and partially by human raters. We used the argument-based approach to validation, which showed that the test can support low-stakes decisions about learners’ knowledge of sociopragmatics in English.
Show Summary Details
Restricted access

5 Validity and Validation


Tests are designed for a purpose, most generally to provide score users with information about test takers, and this information is then employed to inform decisions. For example, test takers are given a test of ESL pragmatics and the scores are then used to design an instructional treatment to improve their performance in making requests in formal settings. In evaluating such a use of test scores, the core question is its defensibility (Messick, 1989): can the evidence collected by means of the test support the decisions that it is meant to inform? To use a slightly outrageous example: having someone run the 100 yard dash and then using their time to infer how much they know about requests in formal settings, is obviously absurd since running speed is not conceivably related to knowledge of requests. In other words, inferring knowledge of requests from running speed is not a defensible inference.

While the example of running speed and pragmatic knowledge is quite obvious, judging the validity of inferences or decisions based on test scores is usually much more complicated. Take a test like TOEFL: it is designed to collect evidence of test takers’ English language ability and support decisions about admission to English-dominant tertiary academic settings. Evaluating the validity of inferences and decisions based on TOEFL scores requires a thorough evaluation of how the TOEFL arrives at its scores, what it covers, and to what extent the information it provides is useful for informing the decisions that end users...

You are not authenticated to view the full text of this chapter or article.

This site requires a subscription or purchase to access the full text of books or journals.

Do you have any questions? Contact us.

Or login to access all content.