The Creation of Advanced Level ASE Tests
Thank You for the Insight, I for one appreciate the time you took to show us the inner workings.
Hi Jeff. Great job and nothing less than I'd expect from you my friend! I fully concur with your description and experiences, as being accurate and fair representations. Having been one of "those" people myself, circa … for the ASE A3 Manual Drive train, A7 HVAC and AYES workshops in the presence of David Milne and John Tisdale and others, my experiences very closely mirror yours. Every person in the teams totaling about 15 members, brought significant expertise and a wide range of knowledge and experience to the table. There were no "imposters" in the room. Everyone was well-vetted for what they could contribute to the question-writing sessions, accuracy of content and discussions. So, the groups typically were comprised of engineers, technicians and experts from other realms of the trade. While engineers may not be on the front lines of repair, their input to technical accuracy was appreciated.
One of the biggest hurdles is writing technically challenging questions, given that vehicle manufacturers do not design and engineer systems identically, or may not have standardized diagnostic procedures for every fault. Even criteria for generating DTCs has enough variance, that what works for one manufacturer has some different aspects when compared to others. There is simply no room for conflicting and manufacturer unique diagnostics. So, everything has to be created so that any technician can successfully respond to the questions, based on knowledge and the information provided in the body of the question.
For those who have not participated in such events, the mental image of staying in a hotel in a region that you might never travel to on your own dime, at first might sound like a mini-vacation. However, being a workshop participant is anything but a vacation, as all "newbie's" to the ASE workshop environment will soon discover. The reality is, that the workshops that I attended were three days in length in Florida and Virginia, with long working days in small group conference rooms at the hotels, that began with breakfast in the conference room each morning. I also participated in an AYES workshop with ASE in the same format.
Day 1, beyond introductions began with expectations, confidentiality agreements in regard to the content of the questions, preliminaries that included the ASE question writing style and structure, statistical data such as information about how well or poorly questions perform at the time. Each workshop provided an opportunity to salvage (at best), or scrap some questions from previous workshops that had under performed. To spend a little time re-writing a question so that it might better perform, saves money and time.
As far as the question-writing sessions, the format was individual authoring of questions for about one hour at a time, with each question being passed to our hosts for some adjustment to best meet the familiar ASE styling, while a secretary sat off to the side typing out each question into a booklet. When time came for a break in writing, we were handed booklets for team review, which often led to active discussion and "put your money where your mouth is" to prove whether a question as written could be qualified or disqualified. Various information resources were utilized if needed, during debate about the validity of every aspect of a question. Whether it was accuracy of the body of the question or the plausibility of the distractors and the validity of the selected best response, it was all subject to detailed scrutiny. If not accepted as formatted, a question was either taken back by the author to be re-worked, or set aside.
If anyone thinks that some "schmuck" who has too much free time on his or her hands just signs up for and writes questions and ASE or the other participants accept them blindly, then they are sadly mistaken. To describe the review process, it is a rigorous and energy consuming process. A question that may be excellent in format and content, may just fall outside the boundaries of being considered to be a useful performer. All questions are worthy of discussion, but if the group cannot arrive at a consensus within a reasonable time frame, such questions are usually tabled or taken by the original author or others interested in the challenge. This sort of question if discussed for too long, comes at the expense discussing or final approval of other questions.
Sometimes, when questions were restructured or polished a bit more into the approved ASE style, the context and intent of the question could be altered significantly or lost. Their editing methods could possibly result in a sound question not being acceptable to the author's original intent. So, taking a question back and re-working it to better define the purpose was also an option. Regardless, every participant put forth their best efforts to author and challenge every question in an effort to make them accurate, fair and provide a bank of questions that would perform well over a wide range.
After long days in a room behind closed doors and a constant supply of food and beverages, ASE took us out to dinner the first two evenings. At the end of day 3, we said our "goodbyes" and headed to the airport. The whole experience for me was enjoyable, along with the opportunities to network and make new friends. In my work, writing questions is usually in the Canadian Federal Style or other rather than Technician A and Technician B, never the less, spending time all those years ago contributing to ASE workshops was a valuable, interesting and enjoyable experience in a professional environment. Of course, there are always confidentiality disclosure agreements signatures required before participation, so that participants will not discuss actual questions or truly confidential information outside the closed doors of the workshop.
Nothing confidential has been disclosed above. It is simply a descriptive overview, from recall of my experiences of the process to develop ASE questions about 20 years ago. It is very similar to the way that we write and validate questions elsewhere in various formats. The intimate details remain behind closed doors, but I am sure that while the process was rigorous in those days, that ASE has improved certain aspects to reduce inaccuracies.
Like anything else, nothing is perfect and ASE questions are generated to represent the majority of vehicles, so that one manufacturer that used mineral oil in brake hydraulic systems once upon a time or other similarly uniquely engineered systems used in relative obscurity, are not the focus of questions developed to meet the general standards and acceptance.
There are many from various manufacturers and independent sector repair facilities within this industry, who come together periodically create fair questions that are general and are not manufacturer-specific. That is quite a challenge to face and takes some energy to make the workshops successful for the general technician population that takes the tests. I am thankful for my experiences in and learning about the process all those years ago and encourage others to seize the opportunity if an offer comes their way. I can almost guarantee that all but the most cynical and argumentative "the sky is falling" types, would come away with a rather different perspective of the process once they have been an active participant in an ASE workshop on the "other side of the desk".
I look forward to your next instalment.
Martin,
I think you just provided the next installment.... :)
but as always, well said!
Hi Tim. Lengthy for sure, but not a single PowerPoint slides or bullet point was harmed in the process! VBG (an inside Jeffie joke)
There is a reason why I related my own experiences in detail as I recall them, before Jeff posts his next segment.
Even though it is a heavy overhaul of an advanced test, I expect to see a report that is quite similar in overall format. However, in the past 20 years I do also expect that some changes have taken place in the process.
Thank you Martin. I have always appreciated the depth that you bring to the discussion.
I still recall on iATN where Carl Rader (may he rest in peace) commented on my brevity in writing style by saying, “Jeff’s no Hemingway.” I loved that. (;
Happy New Year friend