FIXED FORM vs. ADAPTIVE TEST DESIGN IN LANGUAGE PROFICIENCY TESTING
Monday, October 17, 2016 by David Bong
There are two fundamental test designs in language testing: Fixed Form, and Adaptive. In a fixed form test, every test taker receives the same items (aka questions) and can be delivered on paper or computer. An adaptive test can only be delivered on a computer. The computer algorithm scores each item as the test taker answers it, then based on the score of that item or cluster of items, the computer algorithm adjusts the level of the next item or cluster of items to the next “challenge” level. Until the era of computers, it was not possible to do adaptive testing on a large scale. It is still not possible to apply computerized adaptive test design for speaking and writing tests on a large scale. But it is now quite easy and common to use adaptive testing for reading and listening tests.
There are many advantages to adaptive tests. Because the level of difficulty of the items adjusts to the level demonstrated by the test taker, s/he is consistently challenged by items without being overwhelmed by overly difficult items, or bored by easy ones. With an adaptive test, the test taker can rise to the top of the scale based on her/his performance. In an adaptive test, test takers will take various paths through the test so that cheating is much more difficult (yes, it does happen anywhere). Because the adaptive algorithm hones in on the actual level of the test taker quickly, it is able to deliver more items across a wide range of levels that are at or close to the test taker’s skill, increasing the confidence that the final score is accurate - all in a shorter amount of time than a fixed form test. In Avant STAMP, we also use the adaptive reading score to determine the level of prompts we deliver for the writing prompts, and the adaptive listening score in the same way for the speaking prompts, thus making the productive sections semi-adaptive and more appropriately leveled for each test taker.
One test design strategy that Avant adopted early on is the Computer Adaptive Multi-Stage Test (CAMST) design pioneered by Dr. Rick Leucht, a Professor of Educational Research Methodology at the University of North Carolina at Greensboro. CAMST combines the benefits of adaptive and fixed form testing by essentially lining up a succession of mini fixed form testlets. After each testlet, test takers move up, down or stay at the same level depending on their performance. Here is the link to Avant STAMP’s CAMST algorithm: Computer Adaptive Multi-Stage Test Design. In a purely item-by-item adaptive test, a test taker will drop down if they get a single item wrong. In each testlet, we cluster a mix of topics and item difficulties to make the adaptive algorithm more accurate and efficient over the length of the test.
Fixed form tests are based on older or less sophisticated technology and have a number of shortcomings, some of which I have mentioned already. In order to assess a wide range of language levels it is necessary to have items that cover the entire range of levels, but this means the test becomes very long and lower level students will be faced with items that will overwhelm them and higher level students are often bored or not challenged by the content. As a compromise, in order to prevent these problems, teachers are sometimes asked to select the level of the form their students will take, which places a ceiling on their performance.
However, fixed form tests do have advantages. Because all test takers receive the same items, you can control the mix of topics. It is also easier to conduct the psychometric (statistical) analysis of items on fixed form tests than adaptive tests for some rather technical reasons. Here is a Wikipedia article on Computer Adaptive Testing that addresses this issue if you are really a testing nerd like some of us. https://en.wikipedia.org/wiki/Computerized_adaptive_testing
Avant uses CAMST adaptive test design in all of its proficiency tests. We strongly believe in its power to deliver accurate results and create an enjoyable testing experience for language learners.