The New Statistics seems OK to use with the Psychology Major Field Test

“If my department switches to the new statistics, will this tank our majors’ scores on the Major Field Test, which our administration uses as an important assessment tool?”

This was one of the first questions asked during a workshop Craig Wendorf and I gave on the New Statistics at the 2016 Society for Teaching Psychology conference.   Having never taken or administered the Psychology Major Field Test (P-MFT), I was left very unsure…and very troubled.  There are already enough roadblocks towards adopting the new statistics…worry about misalignment with a trusted assessment tools hopefully shouldn’t be one of them.

To start investigating this issue, I requested to take a sample P-MFT from ETS, which owns and administers this assessment tool (https://www.ets.org/mft/about).  I’m happy to report that, at least on the test form I took, I could see no issues with using the P-MFT to assess students who have been taught the new statistics.

Confidentiality issues related to the P-MFT preclude me from describing any of the sample questions in detail.  What I can say is that, consistent with the content description of the P-MFT, there is relatively little coverage of statistics (though a decent bit on design).  The coverage that I did encounter was almost exclusively not from a NHST approach, but rather focused on basic interpretation of statistics.  Of the 140 questions I answered in the whole exam, only 2 were specific to the NHST approach, and both of these probed issues well covered in the new book.  So, at least for that test form, I can’t see any substantive mis-alignment between teaching students the New Statistics and using the P-MFT as an assessment tool.

There’s a lot more work to be done to investigate this issue.  First, I’d like to make some contact with ETS to encourage awareness of this issue when developing new test items.  Moreover, perhaps making an effect size or CI judgement would be a useful item, given the way the field is going.

In addition, there are some other standardized assessment tools out there which I need to explore: the ACATs and the Psych GRE come to mind.

Also, while I am relieved to see that the P-MFT does not seem to stress fine-grained details of the NHST approach, I came away not very impressed.  Many of the items seemed like trivia, others could easily be guessed based on context and vocabulary, some probed theories which are no longer particularly well supported by the evidence… in general it didn’t really feel like it measured much of what I wanted my majors to know.  It’s not sour grapes, either.  I managed to score a 200, the top score–but that was concerning because I completely bumbled my way through sections on clinical psych and personality.  I’ve never taken course work in these areas!

Anyway, to reiterate–it seems like the P-MFT is not currently a major roadblock to a department considering adopting the new statistics.