The Development and Implementation of Computer-based Test and Diagnosis System, with Research on Item attributes and Examinees’ Performance
|關鍵字:||IDEA 模式;試題反應理論;能力估測引擎;加強式SP模式;網路式二階層診斷測驗;試題參數;IDEA model;Item Response Theory;ability estimation engine;enhanced SP model;web-based two-tier diagnostic test;test item parameters|
合適的能力估測引擎是任何電腦化適性測驗系統的關鍵成份。運用不同的能力估測方法，對於受測者的特殊反應樣式，將產生一些問題如發散或慢速收斂等狀況。能力估測引擎影響適性測驗的結果及效能，針對此問題，本研究實作四種電腦適性系統中常用能力估測程式(OWEN, EAP, MLE和WLE)，且評估及比較其收斂狀態及動態行為。
The purposes of testing in the educational context are to assess the learners’ achievement, to diagnose the test items’ quality and to evaluate the educational goal. Due to the rapid progress and widespread dissemination of information technology, the computer-based test is applied on informal or formal tests. Multimedia materials can facilitate the students in terms of reading, understanding, and interpreting abstract concepts. How multimedia presentation form affects the item attributes (i.e., difficulty level, discriminatory power) and the students’ ability to understand the test items’ semantic meanings in IRT-based computerized test is a valuable topic to survey. An IDEA model and a hybrid assessment framework were constructed and an experimental study was conducted for this purpose. An adequate ability estimation engine is the vital component of any efficient and accurate computerized adaptive test system. When estimated using different ability estimation methods, different test takers’ response patterns, especially extreme ones, would produce some problems such as divergence or slow convergence. The ability estimation engine affects the test with respect to its outcome as well as efficiency. To address these issues, this study developed and implemented four ability estimation programs (OWEN, EAP, MLE, and WLE) widely used as the IRT ability estimation engines for a computerized adaptive test system. Then different response patterns were fed to the IRT ability estimation engines so as to evaluate and compare the convergent state under various response patterns, and investigate the dynamic behavior. How to diagnose the learners’ learning problem is a challenging task in either e-learning or the traditional classroom. S-P chart and its caution indexes can be used for diagnosing the students’ abnormal performance and the test items’ suitability. But S-P chart neglects the response time factor. This study proposed an enhanced S-P model, taking the time factor into consideration, and deduced a nimbleness index and test-item solving index for depicting the test-takers’ responding agility and problem-solving ability in the test. Based on enhanced S-P model, this study developed a web-based test system. Various useful charts and prescriptions can be generated by the system and used by instructors to diagnose their instructional approaches and by students to diagnose their learning performance. In order to further diagnose the learners’ mental misconceptions, this study conducted a two-tier diagnostic test and remedial learning experiment by adopting the electro-magnetic concept as an example. The remedial learning effect of the treatment group was significantly better than that of the control group. The test items’ parameters in Item Response Theory or Classical Test Theory are important indicators for understanding the quality of test items. Scoring and diagnostic information in S-P chart or enhanced SP model are essential indexes for understanding the students’ learning effect and performance. This research adopted an English proficiency test as an example and attempted to conduct an experiment for studying the correlations among different item attributes such as the difficulty index, discrimination index, guess index in IRT model as well as the student caution index, and the correlations among different learning performance indicators such as test score, problem-solving ability, and nimbleness index (or test-taker’s response time) in the enhanced S-P model. Further, regression analysis was conducted to deduce the related index’s power of prediction for learners’ performance.