erratic cronbach's alpha

[bmedclinic]

---

Members do not see this ad.

Hey, all
I hope I dont get in trouble for "homework help". Truth be told, this is one of the last small parts of my dissertation.

So I made a measure as a very small part of my dissertation, tested it out on undergrads before administering it to my target population (physicians). Decent alpha .659. The measure is designed to assess physician's knowledge on psychological interventions via 20 MC questions.

Anyways, I had two intervention phases where I administered the measure pre-post both of those (so 4 administrations). Now I have 5 alpha scores total - one for the development and 4 during the intervention phase (pre intervention 1 post intervention 1 pre intervention 2 post intervention 2).

Anyways, the cronbach's alpha scores are erratic and despite my efforts, I cant find a good reason for why. I'm not asking for help as much as maybe a suggestion of where to look or what to do to explain the scores.

Chronbach's Alpha Development .659 (reported above)
Chronbach's Alpha Pre- 1 .078
Chronbach's Alpha Post- 1 .451
Chronbach's Alpha Pre- 2 -.123
Chronbach's Alpha Post- 2 .290

In case you're wondering, it's the same 20 items and pretty much the same people (a few different for each).

I feel so stupid putting this out here, but I'm at my end trying to find a reason for this.
Could any of this have to do with them doing MUCH better than I expected on the measure of their psych knowledge?

Does it have to do with omitted items? Each time there were omitted items by SPSS.

 

[Ollie123]

"Duh" check. Since you used the phrase MC questions, I assume these were right vs. wrong instead of rating scales (i.e. 1 to 7 likert-type). Did you recode responses to be zero (incorrect) or one (correct) for the analysis? If not, the values you are plugging in are meaningless anyways, so that could certainly cause them to jump around everywhere. Not meaning that to be insulting if it is obvious...just seemed an easy thing to forget and was the first thing that came to mind.

Omitted items shouldn't cause that directly if they are missing - if they were coded as -99 or something similar it could absolutely affect it. Might also want to check pairwise versus listwise deletion - if you are missing lots of data do to listwise deletion weird things might happen. Don't know what your n is but that is a factor as well. Cronbach's alpha isn't an inferential technique so doesn't require large n's to be meaningful but as with factor analysis the stability becomes highly questionable with small n's.

That's what I'd look to first. If none of those, let me know and I'll think on it some more.

 

[thewesternsky]

I agree with Ollie's comments. SPSS's default is listwise deletion (e.g., to omit the participant from the scale analysis if they are missing one item), so if there are a lot of participants missing 1-2 items that will make a difference.

It may also be informative to check the item-total correlations and the "alpha if item deleted" values to see if you have a couple really problematic items, though with alphas that low this might not be hugely helpful.

 

[bmedclinic]

"Duh" check. Since you used the phrase MC questions, I assume these were right vs. wrong instead of rating scales (i.e. 1 to 7 likert-type). Did you recode responses to be zero (incorrect) or one (correct) for the analysis? If not, the values you are plugging in are meaningless anyways, so that could certainly cause them to jump around everywhere. Not meaning that to be insulting if it is obvious...just seemed an easy thing to forget and was the first thing that came to mind.

Disclaimer: it's been a LONG time since I took all my stats coursework and at this point, I'm just rusty.

"Duh" indeed, but it was at least the majority of my error.
Now I've done that (I already had it recoded but wasnt using it for the analysis) and 3 of my 4 alphas are above .96.

Fantastic. However, the first administration's output says chronbach's alpha is -.781. Now that the other three are fixed though, I'm thinking this is probably a coding error, or something, right?

 

[Ollie123]

Welp, that's a good start then.

Could be a coding error or related to one of the other issues I raised (i.e. missing values as numbers). Depending on how much your intervention "taught to the test", so to speak, I'd expect a lower alpha at the first administration, particularly if overall performance is good. i.e. People get a few wrong in the beginning (driving down Cronbachs), you intervene and now they get them all right so Cronbach's is > .999 (oversimplification, but illustrates the idea). Test development and scale development are certainly closely related, but the assumptions and goals underlying each are not entirely overlapping - at least in my eyes (not exactly my field).

I'd re-check the coding as the next step, but if there aren't any issues I think it is entirely possible .781 is legit. Hard to say for sure without more in-depth knowledge of the study/intervention.

 

[bmedclinic]

I'd re-check the coding as the next step, but if there aren't any issues I think it is entirely possible .781 is legit.

I'll definitely play around/look more closely for a coding error.. But I already feel much better given the other three alphas and my "duh".

One question: did you mean the .781 is legit? My chronbach's output for the first one was negative (-.781). Can that be legit, let's say, if before the first intervention, they were all over the place about their thoughts about mental health? Hypothetically, of course, because they're smart M.D's and my intervention is tops, afterwards, they're all on the same page and have an alpha > .96?

Seems too good to be true.. but I'll keep looking for my mistake.

 

[Ollie123]

Oh, missed the negative sign. That seems extraordinarily unlikely. In theory it only goes from zero to 1 and negative values are usually a sign something really whacky is happening. I'd check alpha if item deleted, look to make sure you didn't miscode the scoring as correct/incorrect and double check your data. If there is any possibility you used different versions of the scales across time points I'd also look at your physical questionnaires themselves - be they paper or electronic. If the first one had some items swapped or something weird like that, but was scored using syntax for a different version I "think" it could manifest as a problem like that. I'm operating in theory though - I've certainly never seen it.

 

[bmedclinic]

That's kinda what I was figuring. Thanks Ollie.