Invited Symposium
Presentation 4
Language teacher development in relation to TOEFL family assessments
Veronika Timpe-Laughlin
Research scientist at - Educational Testing Service in Princeton, NJ.
Veronika Timpe-Laughlin is a research scientist at Educational Testing Service in Princeton, NJ. She received her Ph.D. in Applied Linguistics/TESOL in 2013 and her M.A. Ed. degree in teaching English and German in 2009—both from TU Dortmund University (Germany). She also holds an M.A. degree in Language Testing from Lancaster University (England). Since joining ETS in 2013, Veronika has conducted research in L2 pragmatics, young learner assessment, teacher education, and task-based language teaching (TBLT). Her research has recently been published in Intercultural Pragmatics, Language Teaching, Language Testing, and System.
Renka Ohta
Research Project Manager in the Center for Language Education and Assessment Research at Educational Testing Service.
Renka Ohta is a Research Project Manager in the Center for Language Education and Assessment Research at Educational Testing Service. She earned her Ph.D. in Foreign Language and ESL Education at the University of Iowa in 2018. Before working at ETS, she taught English as a second language at a community college in Cedar Rapids, Iowa, and was involved in professional development programs for K-12 teachers of English language learners in Iowa schools. Before pursuing graduate studies, she taught English at a high school in Japan. Her research interests include integrated writing assessment and teacher development.
Invited Symposium 2
Invited Symposium 1
Language learning and assessment innovations in the TOEFL family
Abstract
The TOEFL iBT® test is an assessment of academic English language proficiency used for university admissions and other purposes by more than 11,000 institutions in more than 150 countries worldwide. Since its initial launch in 1964, the TOEFL® test has undergone major revisions motivated by advances in theories of language learning and improvements in English teaching practices. The most recent revision, the TOEFL iBT test, contains numerous innovative design features—including in particular integrated tasks that engage multiple skills—to simulate language use in academic settings and reflect the reading, listening, speaking, and writing demands of real-world academic environments. This alignment between the design of an assessment and best practices in language teaching and learning is a core value underlying all tests in the TOEFL Family of Assessments. The TOEFL® Young Students Series (the TOEFL Primary® and TOEFL Junior® tests) was designed to provide useful formative and summative information to teachers and learners of English in school settings. The TOEFL ITP® program offers institutions an affordable test for placement and progress monitoring within English training programs. The newest addition to the family, the TOEFL Essentials® test, combines the convenience of at-home testing with reliable measurement of relevant language skills and abilities for academic and other purposes, in a student-friendly format.
At ETS, we understand that our English language assessments, and related products and services, should be designed to support language teaching and learning in a variety of educational settings. Accordingly, we engage in on-going research and innovation in response to the evolving needs of language teachers and learners worldwide. In this symposium, ETS researchers present on four recent projects that demonstrate this goal. Following an introduction to the teaching, learning, and assessment orientation of the TOEFL Family, presentations will address: (a) the application of vertical scaling methodology across all tests in the TOEFL Family of Assessments to help teachers and learners understand English proficiency development in relation to test scores; (b) the development of a brand new English proficiency test in response to critical emerging needs of test takers and score users; (c) research that explores the impact of test use on young learner educational practices; and (d) the creation of two new English language teacher development courses to support improved teaching practices in specific educational sectors. The symposium concludes with ample opportunity for questions and discussion.
Introduction
A TOEFL perspective on language teaching,learning, and assessment
John Norris
Senior Research Director of the Center for Language Education and Assessment Research at Educational Testing Service.
John Norris is Senior Research Director of the Center for Language Education and Assessment Research at Educational Testing Service. Prior to joining ETS, he worked at Georgetown University, University of Hawai'i at Mānoa, and Northern Arizona University. He began his career as an English teacher, and he completed his Ph.D. in Second Language Acquisition at the University of Hawai‘i. John’s research focuses on language education (task-based language teaching), assessment, program evaluation, and research methods. His most recent book (with Xiaoming Xi) is Assessing academic English for higher education admissions. John speaks German, Portuguese, and Spanish, and he is currently acquiring Japanese.
Presentation 1
The TOEFL learning journey: Facilitating test selection and score interpretation through application of vertical scaling methodology
.
Spiros Papageorgiou
Managing Senior Research Scientist in the Center for Language Education and Assessment Research at Educational Testing Service (ETS).
Spiros Papageorgiou is a Managing Senior Research Scientist in the Center for Language Education and Assessment Research at Educational Testing Service (ETS). Prior to joining ETS, he worked at the University of Michigan, where he was responsible for leading the development of English assessment programs. Spiros received his doctoral degree in linguistics specializing in language testing from Lancaster University, UK. His publications cover topics such as standard setting, score reporting and interpretation, and listening assessment. Spiros is the coeditor (with Kathleen M. Bailey) of the volume Global Perspectives on Language Assessment: Research, Theory, and Practice (2019).
Lixiong Gu
A Director of Psychometrics and Data analysis in Psychometric Analysis and Research at Educational Testing Service (ETS) in Princeton, New Jersey.
Lixiong Gu is a Director of Psychometrics and Data analysis in Psychometric Analysis and Research at Educational Testing Service (ETS) in Princeton, New Jersey. He holds an M.S. degree in Educational Psychology from Beijing Normal University and Ph.D in Measurement and Quantitative Methods from Michigan State University at East Lansing. Lixiong’s research interests include psychometric issues in language assessments, test equating and linking, and computerized adaptive testing.
Presentation 2
The TOEFL Essentials test: A new assessment to meet evolving test-taker and score-user needs
Larry Davis
Research scientist in the Center for Language Education and Assessment Research at Educational Testing Service (ETS) in Princeton, New Jersey.
Larry Davis is a research scientist in the Center for Language Education and Assessment Research at Educational Testing Service (ETS) in Princeton, New Jersey. He holds an M.A. degree in TESOL from the University of Minnesota and Ph.D in Second Language Studies from the University of Hawai’i at Manoa. Larry has taught English at the university level in China and the U.S. and taught university-level teacher training courses. His research interests include all aspects of the assessment of speaking, including development of technology-enhanced speaking tasks, assessment of spoken interaction, creation of rubrics, rater cognition, and automated evaluation of speaking ability.
Shoko Sasayama
Associate Research Scientist at Educational Testing Service.
Shoko Sasayama is an Associate Research Scientist at Educational Testing Service. She completed a Ph.D. in Linguistics at Georgetown University. Prior to joining ETS, Shoko worked as a faculty development specialist and as Assistant Managing Director of an English language program at the University of Tokyo. Her research focuses primarily on the role of cognition in communicative task design for language learning and assessment, including her award-winning publication, “Is a ‘complex’ task really complex? Validating the assumption of cognitive task complexity” (Modern Language Journal, 2016). At ETS, Shoko leads projects on assessment task design, language learning product innovation, and teacher training.
Presentation 3
Investigating the impact of the TOEFL Young Students Series tests on teaching and learning
Mikyung Kim Wolf
managing principal research scientist at Educational Testing Service.
Mikyung Kim Wolf is a managing principal research scientist at Educational Testing Service. Her research areas span technology-enhanced language assessments, formative assessment, and validity issues in assessing K-12 English language learners in the U.S. and in global contexts. Mikyung has published widely including two edited books, English Language Proficiency Assessments for Young Learners and Assessing English Language Proficiency in U.S. K-12 Schools. She is a chair of the International Language Testing Association (ILTA)’s Language Assessments for Young Learners SIG. She also serves as an associate editor of Language Assessment Quarterly. Her recent co-authored article, Investigating the Benefits of Scaffolding in Assessments of Young English Learners: A Case for Scaffolded Retell Tasks won the 2019 ILTA Best Article Award. Mikyung received a B.A. in English language and literature and M.A. in psycholinguistics from Korea University, and a Ph.D. in applied linguistics with specialization in language assessment from UCLA.
Presentation 4
Language teacher development in relation to TOEFL family assessments
Veronika Timpe-Laughlin
Research scientist at - Educational Testing Service in Princeton, NJ.
Veronika Timpe-Laughlin is a research scientist at Educational Testing Service in Princeton, NJ. She received her Ph.D. in Applied Linguistics/TESOL in 2013 and her M.A. Ed. degree in teaching English and German in 2009—both from TU Dortmund University (Germany). She also holds an M.A. degree in Language Testing from Lancaster University (England). Since joining ETS in 2013, Veronika has conducted research in L2 pragmatics, young learner assessment, teacher education, and task-based language teaching (TBLT). Her research has recently been published in Intercultural Pragmatics, Language Teaching, Language Testing, and System.
Renka Ohta
Research Project Manager in the Center for Language Education and Assessment Research at Educational Testing Service.
Renka Ohta is a Research Project Manager in the Center for Language Education and Assessment Research at Educational Testing Service. She earned her Ph.D. in Foreign Language and ESL Education at the University of Iowa in 2018. Before working at ETS, she taught English as a second language at a community college in Cedar Rapids, Iowa, and was involved in professional development programs for K-12 teachers of English language learners in Iowa schools. Before pursuing graduate studies, she taught English at a high school in Japan. Her research interests include integrated writing assessment and teacher development.
Invited Symposium 2
National Policy, Consequences,
and English Learning:
A Case of the TEPS-Family Tests
Abstract
Learning-oriented assessment (LOA) is primarily concerned with how to make assessment inform and guide learning and instruction. For this reason, LOA approaches require the development of not only appropriate tests designed to serve these purposes but also methodologies and procedures to extract specific information indicating where the learners currently stand and what they need to progress to the next stage of learning. The concept of effect-driven testing (Fulcher & Davidson, 2007) can be very useful in designing and developing an LOA system. In effect-driven testing, the intended effect of a test is clearly specified right from the beginning of a test design process and the test is structured and built with the very effect in mind. In fact, such test effects can also be conceptualized at many different levels of assessment evaluation and validity investigation.
Since the beneficial impact of assessment on learning is a crucial element in evaluating the results of LOA, one way to understand LOA is to investigate it from the perspective of consequential validity, particularly in terms of washback. Oftentimes, investigation of consequential validity and washback requires an expanded view of test effect, i.e., a macro-perspective on test effects, which can also be closely interconnected with the evaluation of language testing policies in a particular society. This is very true in the South Korean context where domestic and international English proficiency tests are used interchangeably for various admission and employment purposes. So it is very important to examine the impact of the domestic and international tests in tandem on English language learning and learners in South Korea along with various assessment variables/factors that can moderate or mediate such impact.
With this as a backdrop, the purpose of this symposium is to investigate and discuss the interrelationships among national testing policies, consequences of English proficiency testing, and English language learning in the South Korean context, with a focus on the fairness of score conversion between the TEPS family of tests and international tests and other related issues. The symposium consists of four related presentations followed by a discussion. The first presentation gives an overview of governmental, regulatory systems of English proficiency tests in Korea by reviewing clauses in relevant laws/decrees and discussing the limitations and problems of the systems. The second one reports a large-scale score comparability study undertaken to develop a new score conversion table for the TEPS and TOEIC. The third presents the results of a content comparison study and a test-taker survey that were done in support of the score conversion study. The fourth reports results of a research project conducted to develop a new intermediate-level English proficiency test targeting secondary school students and civil service exam candidates in South Korea, highlighting the difficulty of the new test compared to those of TEPS and TOEIC. The four presentations will be followed by the discussant’s critique and a Q&A session.
Presentation 1
National Certification and Score Conversion Between Domestic and International Tests in Korea: Legal and Policy Issues
Yong-Won Lee
YoungYu Yang
Abstract
National policies on certification tests can have a huge impact on English language tests administered in a particular country, which may in turn influence English language learners’ attitude toward these tests. Where there are both domestic and international tests competing for market share, it is critically important to ensure that passing scores are determined in a fair and equitable manner across these tests, which require careful supervision/management of score conversion relationships among these tests by a credible public body or a relevant governmental organization. There are two major legal apparatuses currently in place to regulate English proficiency tests used in South Korea: (a) Framework Act on Qualifications (jagyeokgibonbeop) enforced by the Korea Research Institute for Vocational Education and Training under the auspice of the Ministries of Education and Labor and (b) Decree on Public Officials Appointment Examinations (gongmuwonimyongsiheomyeong) enforced by the Ministry of Personnel Management. The first one is primarily concerned with national certification/regulation of domestically-developed tests, whereas the second one has to do with determining English ability certification tests appropriate for civil service examinations, along with their passing score(s). One critical problem of the current systems, however, is that international tests, such TOEFL and TOEIC, are located outside the purview of such national certification/regulation systems while enjoying the privilege of becoming dominant English ability certification tests for civil service examinations in the South Korean context. Even worse, there is no regulatory procedure enforced to periodically review and evaluate the fairness of score conversion between domestic and international tests as well as the adequacy of cut scores established for each of these tests. With this as a backdrop, the main purposes of this presentation is: (a) to give an overview of these governmental, regulatory systems by reviewing clauses in relevant laws/decrees, (b) expose the limitations and problems of the systems, with a focus on test revision history, score trend analyses of these tests for the past two decades, and fairness of score conversion relationships among them, and finally (c) suggest possible ways to reform these certification/regulatory systems.
Presentation 2
Score Comparability of TEPS and TOEIC
Euijin Lim Guemin Lee Heesung Jun
Jiwon Choi Youmin Hong Nagap Park
Abstract
Scores from the TEPS (Test of English Proficiency developed by Seoul National University) and the TOEIC (Test of English for International Communication) are widely used in Korea for various purposes, including employment, admission, and certification. However, there is more than one score conversion relationship between these two tests, causing confusion among test takers. Since being established in 2004 and 2011, moreover, the conversion relationships have not been updated for at least 10 years and up to 17 years. This is problematic, as both tests have undergone revision and the characteristics of the test-taker populations may have also shifted within this time. As a result, users of one test may have been disadvantaged over those of the other.
With this as a background, a large-scale score comparability study was undertaken to develop a new score conversion table for the TEPS and TOEIC. Using a single group design, data were collected from a total of 1,415 test takers who had taken both tests. Log-linear pre-smoothing and equipercentile methods were used for linking, with cross validation across subgroups.
According to the results of this study, TEPS test takers need to obtain higher scores than TOEIC test takers when the old conversion tables are used. In fact, at a certain cut-score point widely used, the score disadvantage has been shown to be up to 102 points on the 600-point TEPS score scale. In our presentation, we will describe the methods and results of the score comparability study in greater detail and propose the social implications of the final score conversion table.
Presentation 3
Evaluating TEPS-TOEIC Score Conversion: Content Comparison and Test-Taker Survey
Heesung Jun Jiwon Choi Youngmi Lee
Yoo-Ree Chung Yong-Won Lee
Abstract
The TEPS (Test of English Proficiency developed by Seoul National University) and the TOEIC (Test of English for International Communication) are two standardized tests of English language proficiency whose scores are widely used for various purposes in Korea. To ensure that the uses of scores from these two tests are fair and valid, it is important to establish an accurate and unbiased score conversion relationship between the two tests that does not advantage the users of one test over those of the other.
In our presentation, we question the accuracy of the TEPS-TOEIC score conversion relationship that was first established more than 15 years ago and is still being adopted by major decisions makers in Korea today. As a precursor to a large-scale score comparability study conducted in 2020 (Lee, Lim, Hong, & Park, 2021), we carried out a content comparison study and a test-taker survey to triangulate the findings.
We conducted an expert survey of five language testing specialists regarding the comparative difficulty of the content of the two tests. The reading comprehension sections of the two tests were also analyzed and compared using measures of passage difficulty. Furthermore, 500 test takers who have taken both TEPS and TOEIC were surveyed regarding their perceptions of the score conversion table currently being used for the civil servant selection exam.
In our presentation, we will focus on discussing the results of both the content comparison and test-taker survey along with their implications, particularly in terms of the need to regularly update score conversion relationships for the fairness and validity of test score use.
Presentation 4
Using the Delphi Method to Inform the Development of a Glocalized English Proficiency Test for Korean EFL Learners
Euijin Lim Heesung Jun Jiwon Choi
Youngmi Lee Yoo-Ree Chung Yong-Won Lee
Abstract
The TEPS (Test of English Proficiency developed by Seoul National University) is a general-purpose English test designed to assess the English proficiency of Korean EFL learners at a diverse range of proficiency levels. An important strength of this test lies in its excellent discriminating power for advanced-level English language learners, as is well evidenced by the wide use of its scores for undergraduate and graduate admissions at major universities and professional graduate schools in South Korea. While the usefulness of TEPS scores in such contexts has been proven, test-taker feedback over the years as well as needs analysis have consistently pointed to a growing need for a glocalized test with good discriminating power among intermediate-level test takers. With this as a background, a research project was undertaken to develop a new intermediate-level English proficiency test targeting secondary school students, civil service exam candidates, and job seekers in Korea.
The project was conducted in three stages: (a) preliminary research, (b) creation of a test specification and pilot test form, and (c) pilot testing. First, preliminary research included domain and needs analyses, a Delphi study with 19 language education/assessment experts, and a review of existing language tests and their score reports. Second, based on the results of the preliminary research, a tentative test design framework was created for the new test, and then a pilot test form was assembled. Third, the pilot test form was administered to over 1,000 high school and adult test takers, who were also asked to complete a post-test survey questionnaire. In addition, an expert survey was conducted with 17 English education/assessment experts and 13 high school English teachers regarding the format, content, and difficulty level of the pilot test.
In this presentation, we will focus on demonstrating how the results of our preliminary research, particularly the Delphi study, were utilized in the early stages of test development to make important decisions on various aspects of the new English proficiency test. We will also discuss how the results from the Delphi study can support the validity argument for the use of scores from the new test. We will conclude with implications and suggestions for test developers interested in finding out about what kinds of preliminary research can be done to inform and guide the test development process in its early stages.
Presenters
Yong-Won Lee
Professor of English linguistics in the Department of English Language and Literature at Seoul National University
Yong-Won Lee is a professor of English linguistics in the Department of English Language and Literature at Seoul National University. His research interests include multi-faceted analyses of writing and speaking assessments, diagnostic language assessment, and automated essay and speech evaluation.
Special professor in the School of Communications at Dankook University.
YoungYu Yang
YoungYu Yang is a special professor in the School of Communications at Dankook University. He worked for more than thirty years as a reporter for Joongang Ilbo Daily Newspaper.
Guemin Lee
Professor in the Department of Education in the College of Education at Yonsei University
Guemin Lee is a professor in the Department of Education in the College of Education at Yonsei University. His research interests include scaling, equating, college admission testing, generalizability, and item response theories.
Euijin Lim
Euijin Lim is an assistant professor in the Department of Education at Chinju National University of Education. Her research interests are equating, item response theory, and generalizability theory.
Assistant professor in the Department of Education at Chinju National University of Education
Principal researcher at the TEPS Center, Language Education Institute, Seoul National University
Youngmi Lee is a principal researcher at the TEPS Center, Language Education Institute, Seoul National University, where she works on the production and quality control of the TEPS and its family of tests.
Youngmi Lee
Yoo-Ree Chung
Senior researcher at the TEPS Center, Seoul National Univeristy
Senior researcher and Head of the Test Research and Development at the TEPS Center, Language Education Institute, Seoul National University
Heesung Jun is a senior researcher and Head of the Test Research and Development Department at the TEPS Center, Language Education Institute, Seoul National University, where she works on the development and validation of English proficiency tests, including the TEPS and its family of tests. Her research interests include integrated writing assessment and computer-assisted language testing.
Heesung Jun
Yoo-Ree Chung is a senior researcher at the TEPS Center, Seoul National University. She has earned a Ph.D. degree in applied linguistics at Iowa State University and recently taught courses in statistics for linguistics, Korean language assessment, and academic writing at Yonsei University. Her research interests include test validation and computer-assisted language testing.
Senior Researcher and Head of the Measurement and Statistical Research Department at the TEPS Center, Langauge Education Insitute, Seoul National University
Jiwon Choi is a senior researcher and Head of the Measurement and Statistical Research Department at the TEPS Center, Language Education Institute, Seoul National University, where she works on scoring and quantitative research for the TEPS and its family of tests. Her research interests are test equating, scoring, and item response theory.
Jiwon Choi
Nagap Park
Graduate student in the Department of Education in the College of Education at Yonsei University
Nagap Park is a graduate student in the Department of Education in the College of Education at Yonsei University. His research interests include item response theories, statistical learning, and educational evaluation.
Graduate student in the Department of Education in the College of Education at Yonsei university
Youmin Hong is a graduate student in the Department of Education in the College of Education at Yonsei university. Her research interests are item response theory, computerized adaptive testing, and equating.
