Weighting Components – Focus: High School Assessments

This topic was prompted by the problem of weighting constructed-response and selected-response items in the high school assessment program. A literature-based paper was written by Lawrence Rudner and forwarded to the Psychometric Council. The paper was subsequently accepted for publication in Educational Measurement: Issues and Practice.

MD Assessment Web Site – Focus: All Assessment Programs

A searchable computer data base for all known literature on Maryland assessment programs is needed. Lawrence Rudner is the lead researcher and is using MARCES-funded ERIC-based personnel for support. They have received copies of all papers MSDE has collected over the years and are organizing the database. See http://marces.org/mdarch/

Evaluating MSPAP – Focus: MSPAP

MSDE has received two evaluative reviews of MSPAP, a content review chaired by Bill Evers and a psychometric review chaired by Ronald Hambleton. Each of these has generated considerable reactions from MSDE and outside individuals. MSDE has a need for an evaluative synthesis of all this material with recommendations about how to proceed to improve assessment in Maryland. After much discussion, MARCES decided to solicit an independent contractor for this purpose. The study will be done by Edys Quellmalz at SRI (Stanford). Bill Schafer is the MARCES lead for the project.

SE(PAC) – Focus: MSPAP

There is interest in expanding the methodology for generating standard errors of percents above cut to the district and state levels. But there is concern that the theory may not support generalizations at these large levels. A Monte-Carlo study has been proposed and agreed to. The MARCES contact is Bill Schafer; Yuan Lee is performing the study as a subcontractor.

Evaluating Service Learning – Focus: High School Graduation Requirements

Maryland has a requirement of a service-learning project for high school graduation. It seems time to evaluate the effectiveness of the requirement for its purpose and the quality of its implementation. The MARCES contact is Bob Lissitz; Melissa Fine, a MARCES affiliate, is performing the study.

Test-Based Methods for Evaluating Teachers – Focus: General

Measurement of teacher quality has obvious potential for policy as well as personnel decisions. It seems reasonable to base assessment of teacher quality on student outcomes. Since these outcomes deemed most important are measured with educational tests, it is desirable to explore ways to quantify teacher quality with data from them. However, there are several methodological approaches to this problem. A study to compare them is being carried out by Terry Alban on a contract basis; Bob Lissitz is the MARCES contact.

Conference for 2001 – Focus: General

The 2001 annual MARCES conference will feature the research interests of Bob Mislevy, who is joining the EDMS department. The dates are August 13 & 14; the topic is Assessments and Cognition: From Theory to Practice. Bob Lissitz is the MARCES contact.

It would be desirable to have a diagnostic, computer-adaptive assessment to use for candidates who are about to take the high school assessments. This would yield a likelihood of passing and areas of particular weakness, should they exist. In order to demonstrate what is feasible, a prototype system is under development. See http://marces.org/mdcat/. Lawrence Rudner is the MARCES lead and Phill Gagne is assisting.

Confounding with Writing – Focus: MSPAP

Since the MSPAP format is constructed-response, the potential exists for writing to be confounded with other achievement domains as an artifact of the assessment, itself. The degree to which the MSPAP scoring process is resistant to such confounding has not been assessed. A study in which writing is manipulated for good and poor content is being planned. Bob Lissitz is the MARCES contact and Phill Gagne is assisting.

Process Control for MSPAP – Focus: MSPAP

Each year the Psychometric Council reviews each phase of the MSPAP analyses. Since eight years of data now exist, it seems reasonable to use process control methods to identify quantitatively areas where the Council should focus particular attention. Bill Schafer is the MARCES contact and Ying Jin is assisting.

Validity of Accommodations – Focus: MSPAP

Tippets, using confirmatory factor analysis for accommodated vs. non-accommodated students, has evaluated the comparative internal structures of MSPAP. However, new insights in the structural equations modeling field have never been applied to her report. Further, whether it may or may not seem logically appropriate, it may not be reasonable to report reading scores for students who receive the reading accommodation if the validity of MSPAP is different for them. Bill Schafer is the MARCES contact and Mara Freeman is assisting.

Web-Based Technical Manual – Focus: MSPAP

The MSPAP technical manual is recreated and redistributed every year, although much of it is redundant. It seems possible to use the web to update the manual. Perhaps only one manual is needed, with tables and text only to augment whatever is no longer current. We are currently considering organizing the manual around the AERA/APA/NCME Standards for Psychological Testing. Bill Schafer is the MARCES contact and Mara Freeman has been assisting.

Combining Data for School Performance Indices – Focus: MSPAP

Currently only MSPAP data and attendance are combined into school performance indices (SPIs) for elementary schools. There is interest in adding a component for CTBS/5 data. A review of approaches in the various states with emphasis on several states that illustrate very different approaches has been developed. A briefing paper on the desirability of adding a new data source to the SPI was developed and a meeting was held with the State Superintendent where the topic was discussed. Bill Schafer is the MARCES contact and Geoff Wiggins is assisting.