Show Less
Restricted access

Testing ESL Sociopragmatics

Development and Validation of a Web-based Test Battery


Carsten Roever, Catriona Fraser and Catherine Elder

Testing of second language pragmatics has grown as a research area but still suffers from a tension between construct coverage and practicality. In this book, the authors describe the development and validation of a web-based test of second language pragmatics for learners of English. The test has a sociopragmatic orientation and strives for a broad coverage of the construct by assessing learners’ metapragmatic judgments as well as their ability to co-construct discourse. To ensure practicality, the test is delivered online and is scored partially automatically and partially by human raters. We used the argument-based approach to validation, which showed that the test can support low-stakes decisions about learners’ knowledge of sociopragmatics in English.
Show Summary Details
Restricted access

7 Methodology

← 72 | 73 →7 Methodology


This chapter begins with a description of the test methods used in this study including a description of the instrument types. This is followed by a detailed description of the item development, and an account of the development and refinement of the test for online use. The chapter also provides an extensive description of the pilot test before outlining the procedures and methods used for the main test with the final form of the test.

Since we set out to test sociopragmatic knowledge, we developed instruments which would allow us to test learners knowledge of how cultural norms and values affect language use and how conventional courses of social interaction progress. We planned to test both productive and receptive abilities but only in offline mode, i.e., not including spoken interaction. The test included four general types of instruments: appropriateness judgments, receptive detection and productive correction of sociopragmatic violations, receptive judgment and productive completion of extended exchanges, receptive judgment of speech styles, as well as C-tests as a proficiency measure.

The first item type, Appropriateness Judgments, is also the most common type in the literature on probing sociopragmatic knowledge (e.g., Bardovi-Harlig & Dörnyei, 1998; Liu, 2006; Matsumura, 2001, 2003). While common, this item type is not unproblematic in terms of scoring since native speakers rarely agree entirely on what is (in)appropriate in a given situation. We developed appropriateness judgment items consisting of a situation prompt, which explains the setting and the relationship between the interlocutors,...

You are not authenticated to view the full text of this chapter or article.

This site requires a subscription or purchase to access the full text of books or journals.

Do you have any questions? Contact us.

Or login to access all content.