Tips for Writing Items for Quality Data: Lessons Learned from Reviewing Hundreds of Items

Years ago, when I was working as a graphic designer, I designed a menu for a high-end restaurant. Both the client and I proofed this menu, and it went to print. I was mortified when the client came back and pointed out an error where I had spelled crab cakes as crap cakes, making something that should have sounded very appetizing, well...very unappetizing! I have never forgotten this one tiny error and how it changed the entire meaning of an entrée.

Since then, proofreading has become a top priority for any work I do. So much so that I find typos everywhere, all the time—in books, on billboards, on websites, etc. And now that I load district-created assessments into Illuminate, I see typos and other errors in many items, too.

When it comes to assessments, having typos and other errors in items can have bigger implications than being embarrassing. Item errors can impact the quality of the overall assessment. Deficits in items can decrease the quality of the data an assessment provides—data that we use to make decisions for students. 

In today’s world, quality items and assessments have never been more important. Research by Illuminate Education shows that the school disruptions caused by the COVID-19 pandemic have resulted in measurable learning loss. In order to help students catch up, educators must be able to locate students in their learning in order to guide instruction and intervention—and that requires assessment. A comprehensive assessment system of high quality assessment tools and practices is the mechanism by which educators can efficiently and equitably close learning gaps. All assessments and items must be valid and reliable, including the ones that are created in-district.

Writing items and assessments is a complex process. It’s so easy to make little mistakes that can cause bigger issues and undermine the value of your assessment data. If your team is currently creating your own common assessments or assessment items, here are my top tips for reviewing your items before you administer.

Tips for Writing High-Quality Items

Proofreading

So many of the errors I encounter are simple proofreading errors. I have found:

• Typos
• Missing or incorrect punctuation
• Poorly worded items and/or answers
• The wrong “correct” answer is chosen, either when writing the item or when entering the item into the system
• Images supplied are necessary for a student to answer the question, but illegible
• Inconsistent formatting (for example, the numbering is off)

To avoid proofreading errors in your assessments and items, I recommend:

• Using Spelling and Grammar Check: Using Microsoft Word, under the Tools menu, click on Spelling and Grammar every time you make changes.
• Printing Your Document: I find it’s much easier to catch errors on a printed copy than on a screen.
• Reading Backward: Start at the end and read the words backward. You will be surprised at what you find.
• Checking the Formatting: Make sure it is consistent and clear to the reader.
• Most Importantly: Have someone else proofread your work!

Standards-Alignment

Are your items/questions aligned to the correct standard(s)? Confirming this is an important step to help ensure your students are on-track in developing their skills. To help determine which standards your items should align to, consult your district’s scope and sequence as well as their priority standards first. For an additional resource, check out Common Core State Standards. (This site also includes a “Standards in Your State” map providing links to the most up-to-date information on each state/territory’s specific academic standards.)

Rubrics or Scoring Guides

When creating your assessments, the point values of each item should be determined up front and follow your district’s guidelines. Here are a few examples:

• Multiple Response Items with More than 1 Correct Answer:  How many points should it be worth? 1 point total, or perhaps 1 point for each correct answer?
• Constructed Response (Open-Ended) Items: Ensure this item has a rubric/scoring guide, as this is an important tool in grading a student’s constructed response work and should include information that helps a teacher decide how to grade that item/question.

Depth of Knowledge (DOK) and Level of Rigor

If you are using DOK and creating assessments that consider rigor, the website The Hess Cognitive Rigor Matrices (CRMs) by Dr. Karin Hess provides educators with research-based models for effective instruction and assessment, moving students towards greater engagement and deeper learning.

Bias, Sensitivity, and Fairness

Assessment content that is fair is free of unnecessary barriers to success. When test takers encounter unnecessary barriers, they have to process the unnecessary information in addition to the target knowledge or skill. Or students may have such a strong emotional response that they are distracted from thinking about the target knowledge or skill. Additional processing and distraction make the item harder, which is unfair. And when this happens, we create an obstacle for students to demonstrate what they actually know about the target of the assessment. Check out this article to learn more about creating fair assessments.

Multiple-Choice Questions

Does your assessment have too many multiple-choice questions? Although multiple-choice questions have the advantage of taking less time to administer and grade, the disadvantage is the limited type of knowledge that can be assessed using only multiple-choice items. In remote learning settings, opting for other types of items can have a number of other benefits, including the prevention of cheating.

Conclusion

Although there are many things to consider when writing items and assessments, it’s imperative to build in time for applying rigor to your own review process. Perhaps this information will inspire you and your colleagues to review your own assessment review processes in your home school and across the district.

Do you have other questions about writing items or designing assessments? Leave a comment below! To learn more about Illuminate’s Content Loading Services, don’t hesitate to reach out.

*****

Illuminate Education equips educators to take a data-driven approach to serving the whole child. Our solution combines comprehensive assessment, MTSS management and collaboration, and real-time dashboard tools, and puts them in the hands of educators. As a result, educators can monitor learning and growth, identify academic and social-emotional behavioral needs, and align targeted supports in order to accelerate learning for each student. 

Ready to discover your one-stop shop for your district’s educational needs? Let’s talk.