Today we are starting a new column at Air Facts called Washington Report. Our aim is to alert you to new or proposed changes to regulations that affect pilots. This includes Notices of Proposed Rulemaking (NPRMs), Aviation Rulemaking Committee (ARC) reports and more. No boring legislative discussions here–just the facts and a chance for you to weigh in on important issues. –Ed.
In 2011, a rash of Knowledge Test (“the written” to long-time pilots) failures at numerous flight schools caused a bit of a stir, and the FAA admitted that they had added a number of new questions to the test question database without notifying test prep providers or flight instructors. Some of these questions were downright silly, with no relationship to real world flying or even textbook answers. The written test has often been derided for being out of date and out of touch (a question on celestial navigation is still in the database), but the most recent controversy exposed the flawed system that creates test questions in the first place.
In response, an Aviation Rulemaking Committee (ARC) was formed by the FAA to study the issue of Knowledge Tests in general: how the questions are developed, what the goal of these tests is and how to keep them up to date. The ARC recently released its full report (read it here), with dozens of criticisms. A summary of the group’s recommendations is below:
- Establish a stakeholder body of subject matter experts and relevant FAA policy offices to assist with the development and boarding of knowledge questions, review standards and handbook content, and assist with the review of the current bank of FAA test questions for validity and quality.
- The ARC recommends the FAA revise the quality management system (QMS) process through which key policy offices make recommendations to the FAA Airman Testing Standards Branch about needed changes to training and testing documents.
- The ARC recommends the FAA transition to a single testing standards document, the airman certification standards (an update to the PTS), to include knowledge, skills, and risk management.
- Concerning the philosophy of question development, the ARC recommends the FAA should write questions that are pertinent to safe operations and necessary for sound airmanship and maintain discretion to write questions that reference multiple documents such that applicants must correlate data. Further, FAA should adopt a continuous review process to ensure test questions are relevant to the current technology used in aviation with priority given to removing obsolete information from the tests.
- The ARC recommends the FAA improve the feedback mechanism subsequent to knowledge testing by providing the applicant and instructor the specific missed question(s) to identify the deficient knowledge by review of the Airman Knowledge Test Report. Aggregate results of knowledge testing failure areas should be published to provide a mechanism through which training organizations, providers, and publishers can improve and better target their instruction.
What do you think about the state of written testing? Does it focus too much on obscure regulations and not enough on real world piloting skills? Should we cut down on the questions about NDBs and finally add some about GPS? And should all of these questions be published for study or kept secret? Add your opinion below.