Rec reading for assessment measure development?

[Therapist4Chnge]

Advertisement - Members don't see this ad

I know I asked about this in a thread awhile back, but after a few fruitless searches...I decided to post a new thread. I have always had an interest in psychometrics and in the past couple of years I've become more interested in the actual development of new assessment measures. ​

As luck would have it, I am now a Co-PI on a really cool pilot study that involves the development of a new assessment measure, and I'd like to lessen the learning curve on the stats side of the project. The heavy stats lifting is being done by other team members, though I would like to use this as an opportunity to really dig into the work and maximize my opporunities to learn. 😳​

I would like to get some recommendations for journal articles, chapters, and/or books on test development. So....recommendations?​

 

[Ollie123]

Brief and somewhat introductory, but Devellis's "Scale Development" is probably the best overview text I've read, and is well-referenced so you'll get pointed to many other great references. Might be slightly outdated if they haven't released a new edition (mine is almost 10 years old), but as with most things the core of the field remains the same.

 

[Therapist4Chnge]

Brief and somewhat introductory, but Devellis's "Scale Development" is probably the best overview text I've read, and is well-referenced so you'll get pointed to many other great references. Might be slightly outdated if they haven't released a new edition (mine is almost 10 years old), but as with most things the core of the field remains the same.

Thanks, I'll definitely check that out.

Do you know much about the use of Blant-Altman plots for validation of new measures? It seems to have a lot of support, though I recently read a critique where the authors favored a regression analysis, even though B-A plot validation seems to be all over the literature. I believe we are going to use multiple validation approaches, though I'm wondering what a more stats savvy person like yourself thinks about the differences between the two approaches. I'm a stats newb (to these things), so be gentle.

 

[Ollie123]

Probably not the best one to ask as I had to google it to even jog my memory what it was!

I'm not totally ignorant of the area, but my stats knowledge is more limited to experimental research. I have yet to do any formal measurement development. I'd love to get that in a post-doc, though the more I learn the more I realize I could spend the rest of my life hopping from post-doc to post-doc and still not learn everything I want to (right now - I want to learn clinical trial methodology, chaos theory/non-linear analysis, pure quant, and computational neuroscience...in addition to that whole addiction/health psych stuff I'm supposed to actually ya know...be focusing on). As fun as that sounds, I should probably get a grown-up job at some point.

I glanced at the wiki (terrible I know, but the technique is basic enough to get the gist just from the formula). I can see some utility in it, though I imagine it would have far more utility in bench sciences, where the error variance is generally much smaller and not expected the way it is for our work. I'm not sure how much it would get you that a careful glance at a scatterplot wouldn't, since unless I missed something in the math, the information you'd get from it seems like it would roughly equate to homogeneity of variance - just "centered" so perhaps a bit easier to see visually.

I think multi-method approaches are the way to go, and something psychology could stand to use a lot more of. Its semi-popular in measurement but nowhere else. My view is that if its a "real" effect you should be able to analyze the data 10 different ways and get the same answer. My experience suggests that this is almost never the case, which of course opens the door for people to just analyze things 10 different (perfectly justifiable) ways and pick their favorite outcome.

 

[nononora]

I'm attempting to develop a personality measure for my dissertation and I'm using mostly EFA/CFA techniques (through Mplus) as well as IRT (through MULTILOG). In my limited experience, that combination seems to be the new go-to method in firstly identifying factors/facets/traits and secondly, identifying the items that contribute the most information to the latent factor/facet/trait.

Some books that may be helpful to you are "Confirmatory Factor Analysis for Applied Research" - Timothy Brown and "The Theory and Practice of Item Response Theory" - R. J. de Ayala.

 

[Therapist4Chnge]

As fun as that sounds, I should probably get a grown-up job at some point.

There was a really cool fellowship that was posted a handful of months ago for an fMRI research study, and my first reaction was..."That would be SO COOL to do!" The stipend wasn't bad for a fellowship, but I've already invested 2 years into mine, so another 1-2 years wasn't feasible. 🙁

I'm not sure how much it would get you that a careful glance at a scatterplot wouldn't, since unless I missed something in the math, the information you'd get from it seems like it would roughly equate to homogeneity of variance - just "centered" so perhaps a bit easier to see visually.

As a visual person I enjoy all kinds of scatterplots...but I see your point. 😀

My view is that if its a "real" effect you should be able to analyze the data 10 different ways and get the same answer. My experience suggests that this is almost never the case, which of course opens the door for people to just analyze things 10 different (perfectly justifiable) ways and pick their favorite outcome.

The wide variety of analysis has always been a huge challenege/limitation to most of the research out there, though I'm hoping we can at least try and compare Apples to Apples within topic areas. One of my biggest pet peeves about pharma research (besides them burying poor/non-sig. outcome studies) is their use of the HAM-D, which is not a great choice for the measurement of depression....but at least most of the studies use it and you can have a relative comparison across studies.

I'm attempting to develop a personality measure for my dissertation and I'm using mostly EFA/CFA techniques (through Mplus) as well as IRT (through MULTILOG). In my limited experience, that combination seems to be the new go-to method in firstly identifying factors/facets/traits and secondly, identifying the items that contribute the most information to the latent factor/facet/trait.

Thanks! I dont' know much about either approach, though I'll add them to my reading list.

Some books that may be helpful to you are "Confirmatory Factor Analysis for Applied Research" - Timothy Brown and "The Theory and Practice of Item Response Theory" - R. J. de Ayala.

I've heard the Brown book is quite popular, and I'm glad to see it get recommended here. I'll check out the other one too.

It should be no surprise that my "saved list" in amazon is 12+ pages of books. Maybe if/when I get married I'll add the list to the wedding registry.

 

[Ollie123]

It should be no surprise that my "saved list" in amazon is 12+ pages of books. Maybe if/when I get married I'll add the list to the wedding registry.

Do what I'm doing and marry a biostatistician😉

I could TOTALLY talk her into including something like that in our registry...