The purpose of this paper is to assess the validity of multiple-choice questions in measuring fourth grade students’ ability to interpret graphs related to physical science topics such as motion and temperature. We administered a test including 6 multiple-choice questions to 28 fourth grade students. Students were asked to explain their thinking in writing for each question. In addition, we interviewed all 28 students and asked them to justify their answer for each question by thinking out loud. We found that a significant number of students were not able to provide appropriate explanations for their correct answers. Interestingly, however, a significant number of students were able to provide appropriate explanations even though they initially selected an incorrect response. As a result of this study, we suggest caution in using multiple-choice questions as a single data source to assign grades or to make other important decisions about student achievement.

Abrams, L. M., Pedulla, J. J., & Madaus, G. F. (2003). Views from the classroom: Teachers' opinions of statewide testing programs. Theory into Practice, 42(1), 18-29.

Alexander, R. (2010). Children, their world, their education. Final report and recommendations of the Cambridge Primary Review (p.316). London: Routledge.

AERA, APA, & NCME. (1999). Standards for education and psychological testing. Washington, D.C.: American Educational Research Association.

Amrein, A., & Berliner, D. (2002). High-stakes testing & student learning. Education Policy Analysis Archives, 10, Retrieved from http://epaa.asu.edu/ojs/article/view/297.

Aydeniz, M., & Southerland, S. A. (2012). A national survey of middle and high school science teachers' responses to standardized testing: Is science being devalued in schools. Journal of Science Teacher Education, 23, 233–257.

Bailey, K. (1998). Learning About Language Assessment: Dilemmas, Decisions, and Directions. Heinle & Heinle, Pacific Grove, CA.

Beard, R. M., & Senior, I. J. (1980). Motivating students. London, UK: Routledge & Kegan Paul.

Biggs, J. B. (1973). Study behavior and performance in objective and essay formats. Australian Journal of Education, 17, 157–167.

Brickhouse, N. W. (2006). Celebrating 90 years of science education: Reflections on the gold standard and ways of promoting good research. Science Education, 90(1), 1–7.

Chudowsky, N., & Pellegrino, J. W. (2003). Large-scale assessments that support learning: What will it take? Theory into Practice, 42(1), 75-83.

DeBoer, G. E. (1991). A history of ideas in science education: Implications for practice. New York: Teachers College Press.

Entwistle, A., & Entwistle, N. (1992). Experiences of understanding in revising for degree examination. Learning and Instruction, 2, 1–22.

Gilmer, P. J., Sherdan, D. M., Oosterhof, A., Rohani, F. & Rouby, A. (2011). Science competencies that go unassessed. Online Submission. Retrieved from http://www.cala.fsu.edu.

Greene, J. P., & Winters, M. A. (2003). Testing high stakes tests: Can we believe the results of accountability tests? (Report 33). New York: Manhattan Institute Center for Civic Innovation.

Hamilton, L. S. (1994). An investigation of students' affective responses to alternative assessment formats. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.

Hammerman, E. (2005). Linking classroom instruction and assessment to standardized testing. Science Scope, 28(4), 26-32.

Harlen, W. (2013). Assessment & inquiry-based science education: issues in policy and practice. Global Network of Science Academies.

Hillocks, G. (2002). Testing trap: How states writing assessments control learning. New York: Teachers College Press.

Huber, R. A., & Moore, C. J. (2002). High stakes testing and science learning assessment. Science Educator, 11(1), 18-23.

Jones, G., Jones, B., Hardin, B., Chapman, L., Yarbrough, T., & Davis, M. (1999). The impacts of high-stakes testing on teachers and students in North Carolina. Phi Delta Kappan, 81(3), 199-203.

Kaestle, C. (2013). Testing policy in the United States: A historical perspective. Published by the Gordon Commission on the Future of Assessment in Education. Retrieved from http://www.gordoncommission.org/publications_reports/assessment_education.html.

Keeley, P., & Harrington, R. (2010). Forty-Five New Force and Motion Assessment Probes. NSTA Press.

Kersaint, G., Borman, K. M., Lee, R., & Boydston, T. L. (2001). Balancing the contradictions between accountability and systemic reform. Journal of School Leadership, 11(3), 217 – 240.

Kohn, A., Thompson, S., Ohanian, S., & Eisner, E. (2001). Fighting the tests: A practical guide to rescuing our schools. Phi Delta Kappan 82(5). 248-357.

Lee, O., & Luykx, A. (2005). Dilemmas in scaling up innovations in science instruction with nonmainstream elementary students. American Educational Research Journal, 42(3), 411 – 438.

Longo, C. (2010). Fostering creativity or teaching to the test? Implications of state testing on the delivery of science instruction. Clearing House: A Journal of Educational Strategies, Issues and Ideas, 83(2), 54–57.

McMillan, J.H., Myran, S., & Workman, D. (1999, April). The impact of mandated statewide testing on teachers’ classroom assessment and instructional practices. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Quebec, Canada.

Morgenstern, C. F., & Renner, J. W. (1984). Measuring thinking with standardized science tests. Journal of Research in Science Teaching, 21(6), 639-648.

National Research Council. (2014). Developing Assessments for the Next Generation Science Standards. Washington, DC: The National Academies Press.

NGSS Lead States. (2013). Next Generation Science Standards: For states, by states. Washington, DC: National Academies Press.

Nowak, J. (2007). The Problem with using problem-based learning to teach middle school Earth/Space science in a high-stake testing society. Journal of Geoscience Education, 55(1), 62-66.

Osborne, J., Simon, S., & Collins, S. (2003). Attitudes towards science: A review of the literature and its implications. International Journal of Science Education, 25, 1049-1079.

Pellegrino, J. W., Chudowsky, N., & Glaer, R. (Eds.). (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press.

Saturnelli, A. M., & Repa, J. T. (1995, April). Alternative forms of assessment in elementary science: The interactive effects of reading, race, economic level and the elementary science specialist on hands on and multiple-choice assessment of science process skills. Paper presented at the Annual Conference of the American Educational Research Association, San Francisco, CA.

Schafer, W., Wang, J., & Wang, V. (2009). Validity in action: State assessment validity evidence for compliance with NCLB. In R. W. Lissitz (Ed.), The concept of validity: Revisions, new directions, and applications (pp. 173–194). Charlotte, NC: Information Age Publishing.

Scott-Jones, D., & Clark, M. L. (1986). The school experiences of black girls: The interaction of gender, race, and socioeconomic status. Phi Delta Kanpan, 67(7), 520-526.

Shepard, L. A., & Dougherty, K. C. (1991). Effects of high-stakes testing on instruction. Spencer Foundation.

Sireci, S. G., Han, K. T., & Wells, C. S. (2008). Methods for evaluating the validity of test scores for English language learners. Educational Assessment, 13, 108–131.

Tolman, M. N., Sudweeks, R. Baird, H., & Tolman, R. (1991). What research says: Does reading ability affect science test scores? Science and Children, 29(1), 44-47.

United States Congress Office of Technology Assessment. (1992). Testing in American schools: Asking the right questions (Vol. 22). United States Government Printing Office.

Wadsworth, B. J. (1984). Piaget's theory of cognitive and affective development. NY: Longman.