Show Less
Restricted access

Testing ESL Sociopragmatics

Development and Validation of a Web-based Test Battery


Carsten Roever, Catriona Fraser and Catherine Elder

Testing of second language pragmatics has grown as a research area but still suffers from a tension between construct coverage and practicality. In this book, the authors describe the development and validation of a web-based test of second language pragmatics for learners of English. The test has a sociopragmatic orientation and strives for a broad coverage of the construct by assessing learners’ metapragmatic judgments as well as their ability to co-construct discourse. To ensure practicality, the test is delivered online and is scored partially automatically and partially by human raters. We used the argument-based approach to validation, which showed that the test can support low-stakes decisions about learners’ knowledge of sociopragmatics in English.
Show Summary Details
Restricted access

8 Results

← 108 | 109 →8 Results


In this section, we will report results in a general overview and specifically as backings in the validation argument.

We first ran reliability for the ESL group with all items in the test except Appropriateness Correction items 2, 5, 8, and 10, which did not require correction since the target utterance was appropriate. We obtained an overall reliability of Cronbach’s alpha = .800. When considering item statistics, however, we found four items with very low discrimination, which lead to decreased alpha (see Table 15 below):

Table 15: Item reliability

Appropriateness Judgment item 6 was an over-polite apology item, and the other detrimental items were pragmatically appropriate Appropriateness Choice items, i.e., items that did not require a correction. We deleted these under-performing items and re-ran reliability with these five items excluded and attained reliability of alpha = .807 with no item impairing reliability (see Table 16):

Table 16: Item reliability without under-performing items

You are not authenticated to view the full text of this chapter or article.

This site requires a subscription or purchase to access the full text of books or journals.

Do you have any questions? Contact us.

Or login to access all content.