The Performance Testing Council is supported in part by our Cornerstone Members:
This Summit is sponsored by:
Addressing the Potential for Bias in the Development and Scoring of Performance Examinations
Clarence "Buck" Chaffee
All tests are subject to influence of the authors’ human proclivity to reflect their own experience and background. They are the “subject matter experts” and are charged with deciding what is important and what is true or correct. And theoretically, if you have a diverse enough group of experts their collaborative decision-making will override the influence of individual backgrounds and preferences. Groups however can all share the same biases and it is dependent on the test designer and developer to make the conscious efforts to minimize inappropriate bias. This is especially important in performance-based testing where the number of measurable responses is generally small and especially when human scoring is involved.
This presentation looks at the subtle types of bias that are possible in the design of performance tests and human scoring of those exams and it explores approaches to identify and measure bias and reduce the impact of it in the final testing decision.
Clarence “Buck” Chaffee
President, The Caviart Group
Clarence “Buck” Chaffee has been involved in human and computer scored performance testing for more than 40 years. He served as the Director of Examinations for the National Council of Architectural Registration Boards where he directed the human scoring of more than 15,000 graphic design examination solutions per year. He also led the research and development of converting these examinations to a computer administered and scored model.
Mr. Chaffee also served as the Executive Director of the Council of Landscape Architects where he was responsible for the human scoring of thousands of graphic site design solutions.
For more than 10 years, as the President of The Caviart Group, he has led the design and development of the performance examination for sign language interpreters which includes remote human scoring of recorded videos of interpreting performances.
He is a recognized and respected testing professional and is a regular presenter at annual meetings of The Association of Test Publishers.
The Human Element: Building Structure for Standardized Performance Tests
Vicki Gremelsbacker &
Performance testing offers a unique opportunity to rate practitioners in real-world settings, putting their knowledge and skills to the test as they will actually be used. However, the raters are practitioners as well and can introduce additional variation – positive and negative – to the rating process based on their own experiences. Both candidates and raters deserve a scoring system that levels the playing field to ensure the credibility of the credential.
How can an organization build a performance exam that recognizes and neutralizes rater bias?
In this interactive session, we will explore the real-world experiences of the Certification Council for Professional Dog Trainers® (CCPDT®) as they developed their performance exam, the Certified Professional Dog Trainer – Knowledge and Skills Assessed (CPDT-KSA®). Working closely with Professional Testing Corporation (PTC), CCPDT developed a rating process that minimizes rater bias and balances the scores from each member of the rating panel.
Presented by both CCPDT and PTC, we will provide attendees with practical information and processes that can be implemented in their own performance testing initiatives. The discussion will address the importance of developing a thorough rubric that neutralizes rater bias, including technical specification items, gated items, and scored items. We will also highlight the value of a formal rater selection, training, and debrief process to increase rater accuracy and consistency and to ensure that the exam remains standardized across administrations.
President, Professional Testing Corporation
As President and Chair of the Board of Directors, Vicki Gremelsbacker spearheads the development and implementation of PTC’s strategic planning initiatives and operations. She brings more than 20 years of experience in the testing industry to PTC, developing and administering a variety of testing programs for occupational and professional credentialing and licensure. Vicki is very active in the testing industry and serves on Association of Test Publishers (ATP), Council on Licensure, Enforcement and Regulation (CLEAR), and Institute for Credentialing Excellence (ICE) committees. Vicki earned a B.A. in Psychology with a minor in Educational Psychology from the College at Oneonta and an M.S. in Education in Counseling and Personnel Services from Fordham University, with a focus on psychology and statistics.
Exam Chair, Certification Council for Professional Dog Trainers (CCPDT)
Kate Anders, CPDT-KSA, CBCC-KA, is the owner of Pretty Good Dog in Minnesota. She has worked as a dog professional since 2005 and ran the behavior and training program for the Minnesota Valley Humane Society before starting her own business. Kate specializes in child-dog conflict cases and loves helping families prepare for, and live successfully with, babies/children and dogs. She has taught hundreds of prenatal/pre-adoption education programs to help get families off to a strong start. In addition to her CCPDT certifications, Kate is a professional member of the Association of Professional Dog Trainers (APDT), and a Certified Dog Behavior Consultant (CDBC). Kate is also part of a published research team studying canine heart rate variability and its relationship to behavior and aggression.
How Good Is Good Enough: Standard Setting for Performance Tests
Mark Terry, Greg Applegate, & Ellen Julian
Decisions about how to score performance tests impact the options available for setting passing standards. The two activities should be considered as a unit, even if the final decision about the passing standard must wait until performance data are available. Decisions about whether compensation across cases will be allowed and whether some actions (or lack thereof) are fatal (i.e., trap-door/gated items) will have game- changing implications for standard setting. This presentation will address both the psychometric and practical aspects of these complicated decisions. Data from the National Registry of EMTs psychomotor examination for paramedics will illustrate the relationships among the various approaches and their experience with setting passing standards for performance tests will inform the discussion of the practical implications of different approaches.
Chief Certification Officer, National Registry of EMTs
Mark Terry, MPA, NRP is the Chief Certification Officer for the National Registry of EMTs. He has been a Nationally Registered Paramedic since 1987, serving in several emergency medical services. Mark has contributed to projects with the American Heart Association, National Association of EMS Educators, the National Association of EMTs, and a variety of state and local initiatives.
Chief Science Officer for National Registry of EMTs
Greg Applegate is the Chief Science Officer for National Registry of Emergency Medical Technicians with over 30 years’ experience as an educator, psychometrician, and manager. He is an active member of the education, science, and certification communities serving as a commissioner for the Commission on Accreditation of Allied Health Education Programs (CAAHEP) and as a member of the National Commission on Certifying Agencies (NCCA) guidelines revision committee. He is also a member of the American Association for the Advancement of Science (AAAS) and the National Council on Measurement in Testing (NCME), and the Performance Testing Council (PTC).
President, Julian Consulting
Dr. Ellen Julian has decades of experience running, building, and analyzing high-stakes certification exams and has served as an ANSI/ISO 17024-accreditation psychometric assessor since 2014. After completing her measurement training at Florida State University, she worked with both the physician and nurse licensure examinations as a psychometrician. She moved from crunching numbers to running large-volume, high- stakes testing programs in the health professions, learning about the organizational impact of measurement decisions. Now she provides strategic advice for testing programs. She is a past president of the Performance Testing Council, the author of a chapter in the new ICE Certification Handbook, and a volunteer member of a certification program’s Exam Committee.
Keeping up With the Softwarians – A Look at Developing, Testing, and Validating Exam Performance in the Age of Frequent Software Update Cycles
Jefferson Hansen, Pat Hughes, & Ty Kenworthy
The rapid pace of software and service updates most companies embrace today can cause significant challenges for exam developers who use live or simulated software in their exams to assess candidate skills. In addition to changes to ODs and exam items, PBTs require additional development, testing, and checks for relevance. Certiport and ITS have confronted this challenge for all of the live-in-the-app exams we deliver. We will first present the challenges and some solutions we’ve employed and then open up to a group discussion and Q&A.
Director of Exam Development, Certiport
Mr. Hansen is the director of exam development at Certiport. As a product manager and a people manager, he has several years of experience managing the vision, strategy, design, and development of several SaaS, software, and systems products. He is passionate about improving how businesses and people get things done, whether through new processes, users experience design, or software.
Client Portfolio Manager, ITS
Mr. Hughes is a client portfolio manager at ITS. He has several years of experience in the testing industry, in particular overseeing performance-based exams.
Business Development, Certiport
In his business development role at Certiport, Mr. Kenworthy focuses on product and market strategies at Certiport. He has several years of experience in the testing industry, in particular overseeing performance-based exams.