The Workshop: 10 a.m. - 12 noon: The morning colloquium on “Measuring Validity and Reliability in Coding” will demonstrate the ways in which CAT is used in a variety of multi-disciplinary research projects to measure, improve and report reliability and validity in coding. It will also show how these tools can be used to focus attention on difficult to code concepts.

Tools for reviewing, coding, and retrieving text found in qualitative data analysis packages carry with them no particular attributes for ensuring the reliability or validity of the recorded observations. It is preferable to speak of the reliability or validity of the results from a particular set of observations, rather than of the tool itself, the coding system, or its developers. The Qualitative Data Analysis Program (QDAP) uses the Coding Analysis Toolkit (CAT), a suite of custom built, web-based tools, that enables the Project Manager to measure coder reliability on the fly during pioneer pre-tests. CAT’s core functionality allows for the adjudication of coded items by an “expert” user (or users). The system keeps track of which instances of a particular code are scored valid and which coders assigned those codes. This information provides a track record of coders for assessing coder validity over time. It also allows the account holder to see a rank order list of the coders most likely to produce valid observations, report the overall validity scores by code, coder, or entire project, and end up with a ‘clean’ dataset consisting of only valid observations.

1:00-4:00 p.m.: After the lunch break, Dr. Shulman will lead a hands-on, computer lab-based training introducing techniques for using ATLAS.ti and the Coding Analysis Toolkit (CAT) effectively on multi-coder projects. ATLAS.ti is a software program for the qualitative analysis of large bodies of textual, graphical, audio and video data; CAT is a suite of custom-built web-based tools that work with ATLAS.ti to measure coder reliability.

An ongoing ' user group' will be convening to provide follow up to this workshop.

This workshop is offered in collaboration with the Science, Technology and Society Initiative at the University of Massachusetts Amherst.