Guidelines for Ensuring the Technical Quality of Assessments Affecting English Language Learners and Students with Disabilities: Development and Implementation of Regulations
References
Abedi, J. (1999, April). Examining the effectiveness of accommodation on math performance of English language learners. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
Abedi, J. (2001). Validity of accommodations for English language learners. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA.
Abedi, J. (2004). The No Child Left Behind Act and English language learners: Assessment and accountability issues. Educational Researcher, 33(1), 4–14.
Abedi, J., Courtney, M., & Leon, S. (2003). Effectiveness and validity of accommodations for English language learners in large-scale assessments (CSE Report No. 608). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Abedi, J., Courtney, M., Mirocha, J., Leon, S., & Goldberg, J. (2005). Language accommodations for English language learners in large-scale assessments: Bilingual dictionaries and linguistic modification (CSE Report No. 666). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Abedi, J., & Dietel, R. (2004). Challenges in the No Child Left Behind Act for English language learners. Phi Delta Kappan, 85, 782–785.
Abedi, J., & Hejri, F. (2004). Accommodations for students with limited English proficiency in the National Assessment of Educational Progress. Applied Measurement in Education, 17, 371–392.
Abedi, J., Hofstetter, C., Baker, E., & Lord, C. (1998). NAEP math performance and test accommodations: Interactions with student language background (CSE Technical Report No. 536). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74, 1–28.
Abedi, J., Leon, D., & Mirocha, J. (2001). Impact of students’ language background on standardized achievement test results: Analyses of extant data. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14, 219–234.
Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26.
Abedi, J., Lord, C., Kim, C., & Miyoshi, J. (2001). The effects of accommodations on the assessment of LEP students in NAEP (CSE Technical Report No. 537). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Abedi, J., Lord, C., & Plummer, J. R. (1995). Language background as a variable in NAEP mathematics performance: NAEP TRP task 3d: Language background study. Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Abedi, J., Lord, C., & Plummer, J. R. (1997). Final report of language background as a variable in NAEP mathematics performance (CSE Technical Report No. 429). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Albus, D., Bielinski, J., Thurlow, M., & Liu, K. (2001). The effect of a simplified English language dictionary on a reading test (LEP Projects Report No. 1). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved September 26, 2007, from http://education.umn.edu/NCEO/OnlinePubs/LEP1.html
American Diploma Project. (2004, February). Ready or not: Creating a high school diploma that counts. Washington, DC: Achieve.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: AERA.
Anastasi, A. (1988). Psychological testing. New York: Macmillan.
Assessment and Accountability Comprehensive Center. (2007). Evaluation of the technical evidence of assessments for special student populations. San Francisco: WestEd.
Berk, R. A. (1986). A consumer’s guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 56, 137–172.
Berkowitz, D., Wolkowitz, B., Fitch, R., & Kopriva, R. (2000). The use of tests as part of high-stakes decision-making for students: A resource guide for educators and policy-makers. Washington, DC: U.S. Department of Education Office for Civil Rights.
Calhoon, M. B., Fuchs, L. S., & Hamlett, C. L. (2000). Effects of computer-based test accommodations on mathematics performance assessments for secondary students with learning disabilities. Learning Disability Quarterly, 23, 271–282.
Castellon-Wellington, M. (2000). The impact of preference for accommodations: The performance of ELLs on large-scale academic achievement tests (CSE Technical Report No. 524). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Center for Equity and Excellence in Education. (2005). Recommendations for state ELLs accommodation policies. Retrieved May 31, 2006, from http://ceee.gwu.edu/AA/Accommodations_Recos.html
Center on Education Policy. (2005, March). From the capital to the classroom: Year 3 of the No Child Left Behind Act. Washington, DC: Author.
Center on Education Policy. (2006, March). From the capital to the classroom: Year 4 of the No Child Left Behind Act. Washington, DC: Author.
Cizek, G. (2001). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum.
Education Week. (2004). Quality counts. Education Week, 23(17).
Education Week. (2005). Quality counts. Education Week, 24(17).
Elliot, S. N., Kratochwill, T. R., McKevitt, B., Schulte, A. G., Marquart, A., & Mroch, A. (1999, June). Experimental analysis of the effects of testing accommodations on the scores of students with and without disabilities: Mid-project results. Paper presented at the Council of Chief State School Officers Large-Scale Assessment Conference, Snowbird, UT.
Fast, E. F., & Hebbler, S. (2004). A framework for evaluating validity in state accountability systems. Washington, DC: Council of Chief State School Officers.
Fuchs, L. S., Fuchs, D., Eaton, S. B., Hamlett, C. L., Binkley, E., & Crouch, R. (2000). Using objective data sources to enhance teacher judgments about test accommodations. Exceptional Children, 67, 67–81.
Fuchs, L. S., Fuchs, D., Eaton, S. B., & Karns, K. M. (2000). Supplementing teacher judgments of mathematics test accommodations with objective data sources. School Psychology Review, 29(1), 65–85.
Gong, B. (2005). Documenting validity of accountability systems: A progress report. Los Angeles: University of California, National Center for Research on Evaluation, Standards and Student Testing, Center for the Study of Evaluation.
Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89–116). Mahwah, NJ: Lawrence Erlbaum.
Helwig, R., Stieber, S., Tindal, G., Hollenbeck, K., Heath, B., & Almond, P. (2000). A comparison of factor analyses of handwritten and word-processed writing of middle school students. Eugene, OR: RCTP.
Herman, J. L., & Dietel, R. (2005). What school board leaders should know about standards-based accountability. Los Angeles: University of California, National Center for Research on Evaluation, Standards and Student Testing, Center for the Study of Evaluation.
Hollenbeck, K., Tindal, G., & Almond, P. (1998). Teachers’ knowledge of accommodations as a validity issue in high-stakes testing. Journal of Special Education, 32(3), 175–183.
Jaeger, R. M. (1989). Certification of student competence. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 485–514). New York: Macmillan.
Johnson, E., Kimball, K., & Brown, S. (2001). American sign language as an accommodation during standards-based assessments. Assessment for Effective Intervention, 26(2), 39–47.
Johnstone, C. J. (2003). Improving validity of large-scale tests: Universal design and student performance (Technical Report No. 37). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved June 1, 2006, from http://education.umn.edu/NCEO/OnlinePubs/Technical37.htm
Johnstone, C. J., Thompson, S. J., Moen, R. E., Bolt, S., & Kato, K. (2005). Analyzing results of large-scale assessments to ensure universal design (Technical Report No. 41). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved, May 19, 2006, from http://education.umn.edu/NCEO/OnlinePubs/Technical41.htm
Kane, M. T. (2001). So much remains the same: Conception and status of validation in setting standards. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 53–88). Mahwah, NJ: Lawrence Erlbaum.
Kane, M. T. (2002). Validating high-stakes testing programs. Educational Measurement: Issues and Practice, 21(1), 31–41.
Kingston, N., Kahl, S. R., Sweeney, K., & Bay, L. (2001). Setting performance standards using the body of work method. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 219–248). Mahwah, NJ: Lawrence Erlbaum.
Kopriva, R. (2000). Ensuring accuracy in testing for English language learners. Washington, DC: Council of Chief State School Officers.
Kosciolek, S., & Ysseldyke, J. E. (2000). Effects of a reading accommodation on the validity of a reading test (Technical Report No. 28). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996, June). Standard setting: A bookmark approach. Paper presented at the Council of Chief State School Officers Large-Scale Assessment Conference, Colorado Springs, CO.
Linn, R. L. (2003). Performance standards: Utility for different uses of assessments. Education Policy Analysis Archives, 11(31). Retrieved March 8, 2006, from http://epaa.asu.edu/epaa/v11n31
Linn, R. L., Baker, E. L., & Betebenner, D. W. (2002). Accountability systems: Implications of requirements of the No Child Left Behind Act of 2001. Educational Researcher, 31(6), 3–16.
Livingston, S. A., & Zeikey, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.
Mazzeo, J., Carlson, J. E., Voelkl, K. E., & Lutkus, A. D. (2000). Increasing the participation of special needs students in NAEP: A report on 1996 NAEP research activities (NCES Publication No. 2000-473). Washington, DC: National Center for Education Statistics.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: Macmillan.
Mitzel, H. C. (2005). Consistency for state achievement standards under NCLB. Paper presented to CAS SCASS Study Group. Washington, DC: Council of Chief State School Officers.
National Research Council. (2002). In J. Koenig (Ed.), Reporting test results for students with disabilities and English-language learners. Washington, DC: National Academies.
National Research Council. (2004). In J. Koenig & L. Bachman (Eds.), Keeping score for all: The effects of inclusion and accommodation policies on large-scale educational assessments. Washington, DC: National Academies.
No Child Left Behind Act of 2001, 20 U.S.C. 6301 et seq. (2002).
Olson, B., Mead, R., & Payne, D. (2002). A report of a standard setting method for alternate assessments for students with significant disabilities (NCEO Synthesis Report No. 47). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
Rabinowitz, S. (2004). Design considerations for building out NCLB state assessment systems. San Francisco: WestEd.
Rabinowitz, S., & Ananda, S. (2002). Assessment and accountability provisions for the No Child Left Behind Act: Key decision points for states. San Francisco: WestEd.
Rabinowitz, S., & Sato, E. (2005). Evidence-based plan: Technical adequacy of assessments for alternate student populations. A technical review of high-stakes assessments for English language learners. San Francisco: WestEd.
Rabinowitz, S., & Sato, E. (2006). The technical adequacy of assessments for alternate student populations: Guidelines for consumers and developers. San Francisco: WestEd.
Raymond, M. R., & Reid, J. B. (2001). Who made thee judge? Selecting and training participants for standard setting. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 119–157). Mahwah, NJ: Lawrence Erlbaum.
Rivera, C., & Collum, E. (2004, January). An analysis of state assessment policies addressing the accommodation of English language learners. Issue paper commissioned for the National Assessment Governing Board Conference on Increasing the Participation of SD and LEP Students in NAEP. Arlington, VA: George Washington University, Center for Equity and Excellence in Education.
Rivera, C., & Stansfield, C. W. (1998). Leveling the playing field for English language learners: Increasing participation in state and local assessments through accommodations. In R. Brandt (Ed.), Assessing student learning: New rules, new realities (pp. 65–92). Arlington, VA: Educational Research Service.
Rivera, C., & Stansfield, C. W. (2004). The effect of linguistic simplification of science test items on score comparability. Educational Assessment, 9(3&4), 79–105.
Roeber, E. (2002). Setting standards on alternate assessments (NCEO Synthesis Report No. 42). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
Russell, M., & Plati, T. (2000). Mode of administration effects on MCAS composition performance for grades four, eight, and ten. A report of findings submitted to the Massachusetts Department of Education. Chestnut Hill, MA: National Board on Educational Testing and Public Policy.
Schulte, A. G., Elliott, S. N., & Kratochwill, T. R. (2000). Educators’ perceptions and documentation of testing accommodations for students with disabilities. Special Services in the Schools, 16, 35–56.
Shepard, L., Taylor, G., & Beterbenner, D. (1998). Inclusion of limited-English-proficient students in Rhode Island’s grade 4 mathematics performance assessment (CSE Technical Report No. 486). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing, Center for the Study of Evaluation.
Sireci, S. G. (2004). Computerized adaptive testing: An introduction. In J. E. Wall & G. R. Walz (Eds.), Measuring up: Assessment issues for teachers, counselors, and administrators (pp. 685–694). Greensboro, NC: ERIC Counseling and Student Services Clearinghouse.
Thompson, S. J., Johnstone, C. J., & Thurlow, M. L. (2002). Universal design applied to large-scale assessments (Synthesis Report No. 44). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved September 11, 2006, from http://education.umn.edu/NCEO/OnlinePubs/Synthesis44.html
Thurlow, M. L., & Ysseldyke, J. E. (2001). Standard-setting challenges for special populations. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum.
Thurlow, M. L., House, A., Boys, C., Scott, D., & Ysseldyke, J. (2000). State participation and accommodation policies for students with disabilities: 1999 update (Synthesis Report No. 33). Minneapolis: University of Minnesota, National Center on Educational Outcomes. Retrieved June 30, 2006, from http://education.umn.edu/NCEO/OnlinePubs/Synthesis33.html
Tindal, G., Heath, B., Hollenbeck, K., Almond, P., & Harniss, M. (1998). Accommodating students with disabilities on large-scale tests: An experimental study. Exceptional Children, 64, 439–450.
Tindal, G., & Ketterlin-Geller, L. R. (2004). Research on mathematics test accommodations relevant to NAEP Testing. Washington, DC: National Assessment Governing Board. Retrieved March 21, 2006, http://www.nagb.org/pubs/conferences/tindal.pdf
U.S. Department of Education, Office of Elementary and Secondary Education. (2004). Standards and assessments peer review guidance: Information and examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: Author.
U.S. Department of Education, Office of English Language Acquisition, Language Enhancement, and Academic Achievement for Limited English Proficient Students. (2006). Title III OELA Monitoring Reports. Retrieved May 5, 2006 from http://www.ncela.gwu.edu/oela/OELAprograms/MonitoringReports
U.S. Office of Management and Budget and Federal Agencies. (2006). Detailed information on the English language acquisition state grants assessment. Retrieved July 30, 2007, from http://www.whitehouse.gov/omb/expectmore/detail/10003329.2006.html
Uzzell, L. A. (2005). No Child Left Behind—The dangers of centralized education policy (Policy Analysis Report No. 544). Washington, DC: CATO Institute.
Walz, L., Albus, D., Thompson, S., & Thurlow, M. (2000). Effect of a multiple day test accommodation on the performance of special education students (Report No. 34). Minneapolis: University of Minnesota, National Center on Educational Outcomes.
Weston, T. J. (2002, July). The validity of oral accommodation in testing. Paper presented to the NAEP Validity Studies Panel, Washington, DC.
As mentioned at the beginning of this document, the Assessment and Accountability Comprehensive Center (AACC) will update these guidelines as new, relevant research, guidance, and strategies become available. Additionally, these guidelines are designed to evolve with the changing needs of RCCs and states. Future guidelines will include relevant information and research related to: ELL alignment and linkage; linguistic modification and access strategies; and the framework for English language proficiency standards and assessments.
For questions related to these guidelines, please contact:
Edynn Sato, Ph.D.
Director, Special Populations
Assessment and Accountability Comprehensive Center, WestEd
esato@wested.org.
The contents of this guide were developed under a grant from the Department of Education. However, those contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government.
