Content and Grade Trends in State Assessments and NAEP (2007)
Each state is required by the No Child Left Behind Act to report the percents of its students who have reached a score level called "proficient" or above for certain grades in the content areas of reading (or a similar construct) and math. Using 2005 data from public web sites of states and the National Assessment of Educational Progress (NAEP), state-to-state differences in percents were analyzed, both unconditionally and conditionally on NAEP, for (1) trend across content areas (horizontal moderation), (2) trend across grade levels (vertical moderation), and consistency with NAEP. While there was considerable variation from state to state, especially on an idealistic-realistic dimension, the results generally show that states are relatively consistent in trends across grades and contents.
Universal Design in Educational Assessments (2006)
This study examines the consistency in the decision-making of HSA and MSA with a purpose of identifying students who are at risk of failing the HSA. The data on which the study is based consist of the HSA and MSA scores collected from four counties in Maryland. The HSA scores were obtained from the 2004 administration in the subject area of English, and the MSA scores came from the 2003 administration in reading.
In this study, existing cutoff scores were used to dichotomize the MSA scale and re-categorize students so that the resulting categorizations could be compared with the pass/fail determinations of the HSA. Results show that the cut score for passing the HSA was more demanding than the cut score for the "proficient" category in the MSA. Similarly, it shows that the cut score for the "advanced" category in the MSA was set slightly too low, if the purpose is to identify students who are likely to fail the HSA. With regard to which cut score to use to identify students for remediation, it is recommended that students who are below proficient should be selected initially so that the wasted use of the remediation program is minimized.
This research identifies potential predictors of at risk students before they take the Maryland High School Assessment (HSA) English examination . Research was based on data collected for the years 2002-2004 for students at four different school systems. To the extent possible, this study utilized the same data in each of the four school systems. The systems differ considerably as to the nature of the student population, size of the system and whether the system setting is more urban or rural.
The analysis of these data was done separately for each County. It occurred in a sequential manner with initial calculation of descriptive statistics, followed by ordinary least squares (OLS), and finally multilevel modeling (HLM). The results of the four school systems are compared on three factors: 1) the similarity of variables that are significantly related to HSA performance (their reliability); 2) the modeling (OLS versus HLM) that works best for predicting the HSA score; and 3) the magnitude of the prediction. Several potential indicators for the performance of HSA are identified by the study and discussed in the paper. These include two measures of reading (performance on MSA Reading and the Scholastic Reading Inventory, poverty, special education, and English Language Learner status. There was some evidence that a student's attendance, English scores at midterm, and GPA also seem to be related to his or her performance on the HSA English 1 exam.
Growth Scales as Alternative to Vertical Scales (2006)
Vertical scales are intended to allow longitudinal interpretations of student change over time, but several deficiencies of vertical scales call their use into question. Deficiencies of vertical scales are discussed and growth scales, a criterion-referenced alternative, are described. Some considerations in developing and using growth scales are suggested.
Harford Reading Excellence Program 2001-2002 - Focus: Program Evaluation & Support
The Harford Reading Excellence Act Grant program is a two-year program that strives to improve reading instruction in five target schools in Harford (MD) County Public Schools by implementing research-based instructional approaches through professional development and by providing a literacy-centered summer school program. This report will summarize and highlight the student outcomes based on Reading Excellence Act Grant program data as well as data from the Maryland State Department of Education. Contact: Robert Lissitz; Melissa Fein is performing the evaluation
Weighting Components - Focus: High School Assessments
This topic was prompted by the problem of weighting constructed-response and selected-response items in the high school assessment program. A literature-based paper was written by Lawrence Rudner and forwarded to the Psychometric Council. The paper was subsequently accepted for publication in Educational Measurement: Issues and Practice.
MD Assessment Web Site - Focus: All Assessment Programs
A searchable computer data base for all known literature on Maryland assessment programs is needed. Lawrence Rudner is the lead researcher and is using MARCES-funded ERIC-based personnel for support. They have received copies of all papers MSDE has collected over the years and are organizing the database. See http://marces.org/mdarch/
Evaluating MSPAP - Focus: MSPAP
MSDE has received two evaluative reviews of MSPAP, a content review chaired by Bill Evers and a psychometric review chaired by Ronald Hambleton. Each of these has generated considerable reactions from MSDE and outside individuals. MSDE has a need for an evaluative synthesis of all this material with recommendations about how to proceed to improve assessment in Maryland. After much discussion, MARCES decided to solicit an independent contractor for this purpose. The study will be done by Edys Quellmalz at SRI (Stanford). Bill Schafer is the MARCES lead for the project.
SE(PAC) - Focus: MSPAP
There is interest in expanding the methodology for generating standard errors of percents above cut to the district and state levels. But there is concern that the theory may not support generalizations at these large levels. A Monte-Carlo study has been proposed and agreed to. The MARCES contact is Bill Schafer; Yuan Lee is performing the study as a subcontractor.
Evaluating Service Learning - Focus: High School Graduation Requirements
Maryland has a requirement of a service-learning project for high school graduation. It seems time to evaluate the effectiveness of the requirement for its purpose and the quality of its implementation. The MARCES contact is Bob Lissitz; Melissa Fein, a MARCES affiliate, is performing the study.
Test-Based Methods for Evaluating Teachers - Focus: General
Measurement of teacher quality has obvious potential for policy as well as personnel decisions. It seems reasonable to base assessment of teacher quality on student outcomes. Since these outcomes deemed most important are measured with educational tests, it is desirable to explore ways to quantify teacher quality with data from them. However, there are several methodological approaches to this problem. A study to compare them is being carried out by Terry Alban on a contract basis; Bob Lissitz is the MARCES contact.
Annual Conference - Focus: General
The annual MARCES conference is healf each year in mid August. Bob Lissitz is the MARCES contact.
Computer Adaptive Testing - Focus: High School Assessments
It would be desirable to have a diagnostic, computer-adaptive assessment to use for candidates who are about to take the high school assessments. This would yield a likelihood of passing and areas of particular weakness, should they exist. In order to demonstrate what is feasible, a prototype system is under development. Lawrence Rudner is the MARCES lead and Phill Gagne is assisting.
Confounding with Writing - Focus: MSPAP
Since the MSPAP format is constructed-response, the potential exists for writing to be confounded with other achievement domains as an artifact of the assessment, itself. The degree to which the MSPAP scoring process is resistant to such confounding has not been assessed. A study in which writing is manipulated for good and poor content is being planned. Bob Lissitz is the MARCES contact and Phill Gagne is assisting.
Process Control for MSPAP - Focus: MSPAP
Each year the Psychometric Council reviews each phase of the MSPAP analyses. Since eight years of data now exist, it seems reasonable to use process control methods to identify quantitatively areas where the Council should focus particular attention. Bill Schafer is the MARCES contact and Ying Jin is assisting.
Validity of Accommodations - Focus: MSPAP
Tippets, using confirmatory factor analysis for accommodated vs. non-accommodated students, has evaluated the comparative internal structures of MSPAP. However, new insights in the structural equations modeling field have never been applied to her report. Further, whether it may or may not seem logically appropriate, it may not be reasonable to report reading scores for students who receive the reading accommodation if the validity of MSPAP is different for them. Bill Schafer is the MARCES contact and Mara Freeman is assisting.
Web-Based Technical Manual - Focus: MSPAP
The MSPAP technical manual is recreated and redistributed every year, although much of it is redundant. It seems possible to use the web to update the manual. Perhaps only one manual is needed, with tables and text only to augment whatever is no longer current. We are currently considering organizing the manual around the AERA/APA/NCME Standards for Psychological Testing. Bill Schafer is the MARCES contact and Mara Freeman has been assisting.
Combining Data for School Performance Indices - Focus: MSPAP
Currently only MSPAP data and attendance are combined into school performance indices (SPIs) for elementary schools. There is interest in adding a component for CTBS/5 data. A review of approaches in the various states with emphasis on several states that illustrate very different approaches has been developed. A briefing paper on the desirability of adding a new data source to the SPI was developed and a meeting was held with the State Superintendent where the topic was discussed. Bill Schafer is the MARCES contact and Geoff Wiggins is assisting.
