The Performance Testing Council

Performance Testing Council
Alexandria Summit Highlights

Hello Everyone, earlier this month many of you joined us at the PTC Summit in historic Alexandria, Virginia. If you were there you know that it was an exciting meeting chock full of interaction, exploration and creativity. If you were unable to attend here is a little bit of what you missed.

The PTC once again hosted industry thought leaders in two PTC pre-summit workshops. Robert Mislevy, Professor, Measurement, Statistics, & Evaluation, University of Maryland; Cathleen Kennedy, Director of Technology for the Berkeley Evaluation and Assessment Research Center at the University of California and Larry Hamel, President of Codeguild Inc. presented on Principled Assessment Design for Inquiry and Lawrence Rudner, VP Research and Development, Graduate Management Admission Council presented on Measurement Theory. These workshops were highly informative, fast paced, interactive and focused on original and pioneering concepts. Based on the positive response we received following the workshops we will be working to offer additional educational opportunities in both models in the future. Many thanks to our awesome presenters for putting these together for the PTC!

In addition to our pre-summit workshops we had a unique panel perspective on ANSI Accredited Personnel Certification. The panel consisted of an overview of the personnel certification presented by Roy Swift, ANSI; a rationale for requiring the certification from George Bieber, Department of Defense; the elements of the certification audit and advancements in the clarifications made for performance exams presented by Jim Olsen, Alpine Testing Solutions and, last but not least, the applicant’s perspective presented by Jeff Pike, GIAC. We then had a chance to really deep drill into the panel and the questions were flying! This is a weighty topic for our members and this format managed to break it down into meaningful segments based on each presenter’s specific focus. The follow-up panel Q&A time was an opportunity to get answers to questions from each perspective and there were many "Aha! Moments" going on in the room!

After lunch we had a fast paced round of "Speed Dating the PTC Way" fueled by the romantic sounds of Barry White. I am sure that Barry's velvet voice had something to do with the fun but the creative thinking and idea sharing was all about the "daters"! Many kudos to Randy Russell, Red Hat, for putting together such a fun way to share our experiences and ideas on four important performance testing topics. We will definitely be seeing some "second dates" come out of this segment!

If that wasn't enough for one day, we went right into a breakaway! No, not a break, a BreakAway! Jennifer McNamara, BreakAway LTD presented on “Simulation and the Future of Testing”. Jennifer's presentation was extremely engaging as she demonstrated state of the art simulations used in high level gaming and talked to us about how such technology is applicable to the testing industry. Being a late afternoon speaker is not always the best time to be presenting but Jennifer's personality and BreakAway's technology had us riveted!

Day one wasn’t done yet! I didn't even mention all the committee work that went on between sessions! Committees are the heart of the PTC. As a working group, Summits are a critical time for the PTC to advance performance testing. By the end of this summit we had several new work products underway and the completion of a project on the “Classification of Performance Items”. Many thanks to the Design and Delivery committee and to Elaine Lopes, Cisco for keeping that project on track and creating this extremely useful classification tool. The Psychos, led by Johnny Weissmuller, Air Force were also busy reviewing three new standards and tackling innovation in performance testing analytics.  And that fun Marketing and Communications committee would like to thank everyone who participated in our website feedback session to help us finalize our new website design. We certainly got a lot of interesting ideas! Finally, Wallace Judd, President of the Board of Directors for the PTC presented the first findings from the PTC sponsored Word Validation Study, a project researching Predictive Validity, Standard Setting and Reliability Measures in a performance exam versus a simulation exam and a multiple choice exam. Stay tuned as more of these research findings are revealed later this year!

Day two was just as busy as day one and started out with a unique opportunity to partner with our friends from NOCA. Jim Kendzel, Executive Director for NOCA told us about the efforts NOCA has been making towards becoming a standards body and we discussed collaboration possibilities for setting national testing standards.  Collaborations such as this are key to moving forward and we will continue to pursue such opportunities to advocate for the inclusion of performance testing in the industry.

Is there a doctor in the house, or is House the doctor? Well, actually, it wasn’t Hugh Laurie, the doctor in this case was Melissa Tuite, VMware and her perplexing patient was Robert Cromartie, also from VMware. Robert's puzzling symptoms were presented to the "Summit Physicians" in the Grand Rounds segment of our summit.   All summit participants got to play the role of doctor, resident, or just a medical student in trying to save poor Robert from virtualization!

And if saving Robert wasn't enough for us, we next learned how to be sure an Athletic Trainer knows what to do to save someone, or at least fix them up when they are injured. The National Athletic Trainers’ Association Board of Certification has a unique exam with “hybrid items”. This is a high stakes exam that previously was done in three parts including a performance test. The conversion to all computer based was a whirlwind for Denise Fandel, Executive Director at the BOC and Jim Henderson, CASTLE Worldwide, Inc. The unique item types and scoring methods were of keen interest to PTC Summit participants. You can check out these items yourself at http://www.castleworldwide.com/tds_v5/hybrid/asp/hybrid_exam_driver.asp.

Our final presentation of the Alexandria Summit focused on implementing performance based scenarios using Delivery Manager and simulation driver technology developed by John Sotiros and Summit Consulting Group and John’s example of a simulation used in the health care profession kept us in the patient mode!

It was great seeing all of you at the Summit. Not only was it a fantastic opportunity to work, learn, share and care, it was a chance to explore, consider and tackle tough questions! It was great to spend time together and to share our interest in performance testing and it was great fun to party at Bilbo Baggins! Mark your calendars now for October 22, 23 and 24 and we will see you in Boulder for the next Performance Testing Council Summit!





		Performance Testing Council Alexandria Summit Highlights Hello Everyone, earlier this month many of you joined us at the PTC Summit in historic Alexandria, Virginia. If you were there you know that it was an exciting meeting chock full of interaction, exploration and creativity. If you were unable to attend here is a little bit of what you missed. The PTC once again hosted industry thought leaders in two PTC pre-summit workshops. Robert Mislevy, Professor, Measurement, Statistics, & Evaluation, University of Maryland; Cathleen Kennedy, Director of Technology for the Berkeley Evaluation and Assessment Research Center at the University of California and Larry Hamel, President of Codeguild Inc. presented on Principled Assessment Design for Inquiry and Lawrence Rudner, VP Research and Development, Graduate Management Admission Council presented on Measurement Theory. These workshops were highly informative, fast paced, interactive and focused on original and pioneering concepts. Based on the positive response we received following the workshops we will be working to offer additional educational opportunities in both models in the future. Many thanks to our awesome presenters for putting these together for the PTC! In addition to our pre-summit workshops we had a unique panel perspective on ANSI Accredited Personnel Certification. The panel consisted of an overview of the personnel certification presented by Roy Swift, ANSI; a rationale for requiring the certification from George Bieber, Department of Defense; the elements of the certification audit and advancements in the clarifications made for performance exams presented by Jim Olsen, Alpine Testing Solutions and, last but not least, the applicant’s perspective presented by Jeff Pike, GIAC. We then had a chance to really deep drill into the panel and the questions were flying! This is a weighty topic for our members and this format managed to break it down into meaningful segments based on each presenter’s specific focus. The follow-up panel Q&A time was an opportunity to get answers to questions from each perspective and there were many "Aha! Moments" going on in the room! After lunch we had a fast paced round of "Speed Dating the PTC Way" fueled by the romantic sounds of Barry White. I am sure that Barry's velvet voice had something to do with the fun but the creative thinking and idea sharing was all about the "daters"! Many kudos to Randy Russell, Red Hat, for putting together such a fun way to share our experiences and ideas on four important performance testing topics. We will definitely be seeing some "second dates" come out of this segment! If that wasn't enough for one day, we went right into a breakaway! No, not a break, a BreakAway! Jennifer McNamara, BreakAway LTD presented on “Simulation and the Future of Testing”. Jennifer's presentation was extremely engaging as she demonstrated state of the art simulations used in high level gaming and talked to us about how such technology is applicable to the testing industry. Being a late afternoon speaker is not always the best time to be presenting but Jennifer's personality and BreakAway's technology had us riveted! Day one wasn’t done yet! I didn't even mention all the committee work that went on between sessions! Committees are the heart of the PTC. As a working group, Summits are a critical time for the PTC to advance performance testing. By the end of this summit we had several new work products underway and the completion of a project on the “Classification of Performance Items”. Many thanks to the Design and Delivery committee and to Elaine Lopes, Cisco for keeping that project on track and creating this extremely useful classification tool. The Psychos, led by Johnny Weissmuller, Air Force were also busy reviewing three new standards and tackling innovation in performance testing analytics. And that fun Marketing and Communications committee would like to thank everyone who participated in our website feedback session to help us finalize our new website design. We certainly got a lot of interesting ideas! Finally, Wallace Judd, President of the Board of Directors for the PTC presented the first findings from the PTC sponsored Word Validation Study, a project researching Predictive Validity, Standard Setting and Reliability Measures in a performance exam versus a simulation exam and a multiple choice exam. Stay tuned as more of these research findings are revealed later this year! Day two was just as busy as day one and started out with a unique opportunity to partner with our friends from NOCA. Jim Kendzel, Executive Director for NOCA told us about the efforts NOCA has been making towards becoming a standards body and we discussed collaboration possibilities for setting national testing standards. Collaborations such as this are key to moving forward and we will continue to pursue such opportunities to advocate for the inclusion of performance testing in the industry. Is there a doctor in the house, or is House the doctor? Well, actually, it wasn’t Hugh Laurie, the doctor in this case was Melissa Tuite, VMware and her perplexing patient was Robert Cromartie, also from VMware. Robert's puzzling symptoms were presented to the "Summit Physicians" in the Grand Rounds segment of our summit. All summit participants got to play the role of doctor, resident, or just a medical student in trying to save poor Robert from virtualization! And if saving Robert wasn't enough for us, we next learned how to be sure an Athletic Trainer knows what to do to save someone, or at least fix them up when they are injured. The National Athletic Trainers’ Association Board of Certification has a unique exam with “hybrid items”. This is a high stakes exam that previously was done in three parts including a performance test. The conversion to all computer based was a whirlwind for Denise Fandel, Executive Director at the BOC and Jim Henderson, CASTLE Worldwide, Inc. The unique item types and scoring methods were of keen interest to PTC Summit participants. You can check out these items yourself at http://www.castleworldwide.com/tds_v5/hybrid/asp/hybrid_exam_driver.asp. Our final presentation of the Alexandria Summit focused on implementing performance based scenarios using Delivery Manager and simulation driver technology developed by John Sotiros and Summit Consulting Group and John’s example of a simulation used in the health care profession kept us in the patient mode! It was great seeing all of you at the Summit. Not only was it a fantastic opportunity to work, learn, share and care, it was a chance to explore, consider and tackle tough questions! It was great to spend time together and to share our interest in performance testing and it was great fun to party at Bilbo Baggins! Mark your calendars now for October 22, 23 and 24 and we will see you in Boulder for the next Performance Testing Council Summit!