2024 Spring Summit Programming

March 6 – 7, 2024

The Performance Testing Council is supported in part by our Cornerstone Members:

 

Check-in and Networking

All

Day 1 / 3:00pm – 3:15pm PT

Check-in and Networking

Welcome and Summit Kick-off

Maurilio Gorito (Performance Testing Council Board President)
Marcus Bolton, Ph.D. (Performance Testing Council Executive Director)

Day 1 / 3:15pm – 3:45pm PT

Attendee Welcome, Summit Kick-off, and Intros

Small Group Activity

All

Day 1 / 3:45pm – 4:30pm PT

Small Group Activity: Speed Networking

Applications of AI in Performance Testing

Alan Mead (Certiverse)
Wallace Judd (Authentic Testing)
Maurilio Gorito (Performance Testing Council Board President)
Bob Mahlman (CCO)

Day 1 / 4:30pm – 5:45pm PT

This panel session will explore topics surrounding the use of AI in performance testing.  Areas of discussion may include:

  • What you should know about using OpenAI AI Assistants in an exam program.
  • How AI free response items relate to Innovative Item types
  • How AI items relate to IRT, MC, and Performance Test items.
  • Using AI to assist in job analysis;
  • Comparison of the use of AI for standard multiple-choice tests versus its utilization in performance testing;
  • Using AI for generating prompts, alternative correct responses, and in the scoring of performance exams; and alternatives to OpenAI’s models.

The panelists will delve into prompt engineering tips for use in performance testing contexts, and the impact of AI on development time.

Wrap-up

Marcus Bolton, Ph.D. (Performance Testing Council Executive Director)

Day 1 / 5:45pm – 6:00pm PT

Day 1 wrap-up

Reception

All

Day 1 / 6:00pm – 7:00pm PT

Reception

Check-in and Networking

All

Day 2 / 9:00am – 9:15am PT

Networking and Day 2 Kick-off

Conversational Testing and Performance

Wallace Judd, Ph.D. (Authentic Testing)

Day 2 / 9:15am – 10:00am PT

Conversational testing is defined as having the computer speak to the candidate, and having the candidate respond by speaking back to the computer.

Dr. Judd will present a number of illustrative uses of conversational testing. How conversational testing complements performance testing will be addressed. Comparisons with multiple-choice testing will be made.

Learning Objectives:

  • Learn options for short-phrase audio response
  • Learn about response latency as critical component of analysis
  • Highlight ramifications of conversational testing for DE&I concerns
  • Learn how conversational testing can complement performance testing

 

Break

All

Day 2 / 10:00am – 10:15am PT

Break

Designing a Skills-Based VR Assessment

Elizabeth Santana (Inteleos)
Cynthia Parshall (Touchstone Consulting)
Jagrup (Jay) Kahlon (VantariVR)

Day 2 / 10:15am – 11:15am PT

This session will be an interactive discussion between designers and developers of a healthcare virtual reality program incorporating assessment. This case study will address the planning and development of a Simulation-Based Test (SBT) — a practical skills assessment within the field of sonography. This VR assessment includes an extensive training component in which an individual learner is guided through a series of ultrasound protocols, and is provided with “coaching” to improve their responses in real time. Specific challenges for the ultrasound SBT case study will be discussed.

This is planned to be a highly interactive session. We will walk the audience through the decisions that were made along the way and entertain questions and discussions from our peers. A short video demonstration will illustrate what has been developed to date and will lead to a discussion of the next steps.

Learning Objectives:

  • Attendees will understand how the proposed SBT framework could guide the design and development of other SBTs
  • Attendees will understand how different types of simulations meet different assessment needs for content and construct
  • Attendees will realize, from the discussions of decisions made during the project covered, how SBTs can expand and enhance assessment

Break

All

Day 2 / 11:15am – 11:45am PT

Break

Lunch

All

Day 2 / 11:45am – 1:00pm PT

Lunch and Sponsor Updates

History and the Future of Testing

Wallace Judd, Ph.D. (Authentic Testing)

Day 2 / 1:00pm – 1:30pm PT

A review of historical foundations and the future directions of testing.

Tablet-Based Performance Testing for Crane Operators

Erin Jones (CCO)
Bob Mahlman (CCO)

Day 2 / 1:30pm – 2:45pm PT

For many years, the National Commission for the Certification of Crane Operators (CCO) had been administering performance exams using a paper-based recording system. These score sheets were scanned and sent to CCO and then hand-entered into a scoring algorithm. In 2024 CCO introduced a tablet-based test administration and reporting system. Performance-based proctors record the observations in the field, which are immediately uploaded and scored. The system provides more robust test data, ease of scoring and reporting, and immediate score results for the candidate. The presenters will describe the elements and benefits of the new system, issues encountered during its design and development, and potential improvements envisioned. Feedback from industry with respect to ease of use, administration issues, and results turnaround has been very positive.

Learning Objectives:

  • Show how tablet-based testing has solved issues with respect to test security and fairness in testing
  • Highlight potential advantages in data analytics possibilities
  • Highlight issues and solutions encountered in system design and build
  • Discuss potential for improvements in performance -based testing that this technology may bring to fruition

Performance Based Testing Is High on All Forms of Validity, Why Aren’t We Using It?

Kyana Beckles (Leverage Assessments)
William Luse, Ph.D. (University of Laverne)

Day 2 / 2:45pm – 3:45pm PT

Given the recent focus on diversity, equity, and inclusion (DE&), this has numerous implications for performance-based testing (e.g., does testing help or hinder diversity goals, is testing equitable across all, what can be done to make testing more equitable, etc.).

We will begin with a review of the performance testing literature, then provide an overview of a case study from the U.S Census Bureau, and then provide a list of guidelines that organizations can use when transitioning to Performance Testing.

Considering the scale of the job applicants for the U.S Census Bureau the agency seeks to prevent adverse impact for applicants after a lawsuit in 2010 after it was discovered that the background check process during the Decennial Census resulted in poor outcomes for African American and Hispanic candidates.

We will provide the specifications and scope of the project, and discuss some high level guidelines which will create more diverse candidate pools. 

Learning Objectives:

  • Share challenges to diversity with traditional assessments:
  • Highlight research on ethnic groups and test performance
  • Review performance testing and validity across cultures
  • Share written/Spoken language and assessments
  • Highlight underrepresented groups in sample sizes

Open Dialogue

Maurilio Gorito (Performance Testing Council Board President)

Day 2 / 3:45pm – 4:15pm PT

Reflection of the Summit

Closing

Maurilio Gorito (Performance Testing Council Board President)
Marcus Bolton (Performance Testing Council Executive Director)

Day 2 / 4:15pm – 4:30pm PT

Day 2 Summit Closing

Optional VR Demo

Elizabeth Santana (Inteleos)
Cynthia Parshall (Touchstone Consulting)
Jagrup (Jay) Kahlon (VantariVR)

Day 2 / 4:30pm – 5:30pm PT

An optional demo of the interactive VR technology highlighted within the earlier session.

Get Involved

We want to hear your ideas. There are many ways you can get involved today.

Share Your Accomplishments.

One of the delights of doing a tough job is sharing with others how you did it. There are plenty of people who would love to hear how you overcame obstacles to create your performance test. Tell us how you did it by presenting in one of our short webinars.

Send us a note

Come to an Event.

The PTC hosts numerous events throughout the year, from webinars to bi-annual Summits. Someone likely has already tackled the very issue you’re experiencing now. Come learn, share, and network with some of the best minds working in the field

See all events

Join Us Today.

Becoming a member of the Performance Testing Council provides valuable benefits that can elevate your testing strategies. Share performance testing resources, network with industry experts, and receive exclusive member discounts on upcoming Performance Testing Council events.

Go to application