“It’s time to abolish the MCAT”

[BorderlineQueen]

Data from multiple schools and AAMC suggest otherwise.

Those of you who have never taught medicals students, please, just stop.

@Goro I heard either a 497 or 500 predicts

I mean that's just people being type A dinguses, entitled brats, immature, and gunners. While its terribly obnoxious, I don't think it qualifies as a mental illness

Plenty of people with cluster B personalities in medicine. Borderline PD, Antisocial PD and Histrionic PD.

---

Members don't see this ad.

 

[Select All That Apply]

This tangential conversation is the reason why there should be hard metrics. Weeding people off on subjective factors like resilience, perceived personality disorders, and a sense of entitlement are all factors that are open to interpretation based on whoever is on the panel.

Stanley Kaplan in an interview with the New Yorker couldn't get into medical school in the 1930s due to ethnic quotas being filled for Jewish applicants in the five boroughs. This was brought up by Conrad Fischer (medical professor at Touro) in a discussion as he was explaining why performing on Step 1 is so important. It's because standardized exams are the greatest of equalizers, they do not discriminate based on race or socioeconomic background. Kaplan dedicated himself to tutoring and the SAT because he believed that what was presumed to be a test that couldn't be studied for, could be studied for if someone were to thoroughly review the concepts and present it in a teachable manner. That's the premise of standardized testing material.

Inginia Genao is completely insincere in her article to KevinMD as she statistically brings up the AAMC data to point out the "racial gap" in standardized test data. However, she does not statistically discuss the next metric that would be considered if, "Nobody needs the MCAT..." This would be GPA. According to AAMC in 2018 Black or African American applicants had a mean GPA of 3.31 while matriculants had a mean GPA of 3.51. This is a significant variance from total GPA of all applicants at 3.57 and matriculants at 3.72. This makes them categorically the lowest scoring among applicants and among matriculants when it comes to GPA. The MCAT is not a divorced metric, but trends fairly consistently on average with GPA when it comes to applicants and matriculants.

In the Annals of Internal Medicine, Inginia Genao argued that black applicants are targeted through the MCAT and suffer from disparate impact because they do not have the socioeconomic ability to afford testing material. However, this argument is ridiculous considering the cost of a 4 year college is disproportionately more expensive than the cost of supplemental testing materials to study for the MCAT. It's also ridiculous to attempt to legally utilize disparate impact as it exists in Griggs v Duke Power on the stipulation that an inferior education is specifically being provided to blacks over other races in 4 year colleges. In order to Genao to utilize this statute, she has to prove that MCAT materials are too expensive for African Americans to afford or that 4 year colleges in the United States are intentionally providing biased education to limit blacks and only blacks from succeeding on the MCAT.

Advocates like Genao fundamentally do not believe that URMs can succeed on standardized tests, so instead of trying to push for education reform they are looking to abolish metrics altogether. This should come across as being incredibly insulting to URMs. This discourse is also disrespectful to minorities who are considered to be overrepresened in medicine and are still able to perform on standardized tests. Despite the Supreme Court ruling in Bakke that there ought to be no more ethnic quotas, the quota system is still used to limit seats from unwanted minorities that the college administration feels would improperly disrupt the color palette of their student body. If you think that Asians systematically and systemically scoring low on personality traits at Harvard which resulted in many of them being denied admission is unrelated to a pseudo-quota system being implemented in higher education then you need to hang out with other Asian people.

It's just a sad turn that what was seen as being the great equalizer, a standardized exam is now being portrayed as a racially biased examination. And the reason has nothing to do with the actual exam, but is a blatant attempt at pushing for black racial representation at all costs. Even if the cost is having a fair and meritocratic system where people earn their acceptance into medical school.

 

[BorderlineQueen]

I got all of em

I'm highly impressed lol.

 

[BorderlineQueen]

This tangential conversation is the reason why there should be hard metrics. Weeding people off on subjective factors like resilience, perceived personality disorders, and a sense of entitlement are all factors that are open to interpretation based on whoever is on the panel.

Stanley Kaplan in an interview with the New Yorker couldn't get into medical school in the 1930s due to ethnic quotas being filled for Jewish applicants in the five boroughs. This was brought up by Conrad Fischer (medical professor at Touro) in a discussion as he was explaining why performing on Step 1 is so important. It's because standardized exams are the greatest of equalizers, they do not discriminate based on race or socioeconomic background. Kaplan dedicated himself to tutoring and the SAT because he believed that what was presumed to be a test that couldn't be studied for, could be studied for if someone were to thoroughly review the concepts and present it in a teachable manner. That's the premise of standardized testing material.

Inginia Genao is completely insincere in her article to KevinMD as she statistically brings up the AAMC data to point out the "racial gap" in standardized test data. However, she does not statistically discuss the next metric that would be considered if, "Nobody needs the MCAT..." This would be GPA. According to AAMC in 2018 Black or African American applicants had a mean GPA of 3.31 while matriculants had a mean GPA of 3.51. This is a significant variance from total GPA of all applicants at 3.57 and matriculants at 3.72. This makes them categorically the lowest scoring among applicants and among matriculants when it comes to GPA. The MCAT is not a divorced metric, but trends fairly consistently on average with GPA when it comes to applicants and matriculants.

In the Annals of Internal Medicine, Inginia Genao argued that black applicants are targeted through the MCAT and suffer from disparate impact because they do not have the socioeconomic ability to afford testing material. However, this argument is ridiculous considering the cost of a 4 year college is disproportionately more expensive than the cost of supplemental testing materials to study for the MCAT. It's also ridiculous to attempt to legally utilize disparate impact as it exists in Griggs v Duke Power on the stipulation that an inferior education is specifically being provided to blacks over other races in 4 year colleges. In order to Genao to utilize this statute, she has to prove that MCAT materials are too expensive for African Americans to afford or that 4 year colleges in the United States are intentionally providing biased education to limit blacks and only blacks from succeeding on the MCAT.

Advocates like Genao fundamentally do not believe that URMs can succeed on standardized tests, so instead of trying to push for education reform they are looking to abolish metrics altogether. This should come across as being incredibly insulting to URMs. This discourse is also disrespectful to minorities who are considered to be overrepresened in medicine and are still able to perform on standardized tests. Despite the Supreme Court ruling in Bakke that there ought to be no more ethnic quotas, the quota system is still used to limit seats from unwanted minorities that the college administration feels would improperly disrupt the color palette of their student body. If you think that Asians systematically and systemically scoring low on personality traits at Harvard which resulted in many of them being denied admission is unrelated to a pseudo-quota system being implemented in higher education then you need to hang out with other Asian people.

It's just a sad turn that what was seen as being the great equalizer, a standardized exam is now being portrayed as a racially biased examination. And the reason has nothing to do with the actual exam, but is a blatant attempt at pushing for black racial representation at all costs. Even if the cost is having a fair and meritocratic system where people earn their acceptance into medical school.

While I'm not going to respond to your post I will comment on the term "racially biased examination". In the past the MCAT was designed to weed out recent immigrants and African Americans. Some of the questions on the MCAT in the 1940s were about the civil war. Standardized exams have a long history of being "racially biased". In the 1900s Psychologists developed the IQ test. Psychologists claimed that no amount of education would ever make a white and African American person have the same IQ. The development of these tests and claims from a few prominent psychologists lead to the eugenics moment. So while the MCAT is a helpful screening tool and while IQ tests are now pretty unbiased. You can't deny that these tests were racially bias in the past. Are they still now? Who knows.

 

[Ho0v-man]

While I'm not going to respond to your post I will comment on the term "racially biased examination". In the past the MCAT was designed to weed out recent immigrants and African Americans. Some of the questions on the MCAT in the 1940s were about the civil war. Standardized exams have a long history of being "racially biased". In the 1900s Psychologists developed the IQ test. Psychologists claimed that no amount of education would ever make a white and African American person have the same IQ. The development of these tests and claims from a few prominent psychologists lead to the eugenics moment. So while the MCAT is a helpful screening tool and while IQ tests are now pretty unbiased. You can't deny that these tests were racially bias in the past. Are they still now? Who knows.

No. They are not. Not at all.

 

[Select All That Apply]

@BorderlineQueen The "understanding modern society/general information" section that you are referring to as per my peruse on Wikipedia was present from 1962-1991 and had been removed after 1991 containing primarily science/medical sections. More notable than the MCAT having an "understanding modern society" section it is more likely that the Civil Rights movement in the 1960s, establishment of EEOP in 1965, and acute push back to the changes in the 1970s had more influence on racial considerations for black applicants at this time period than the MCAT section. For Asian Americans at this time period, strict immigration quotas were finally lifted in 1965 since America christened its first federal immigration ban on Chinese women in 1875 and the Chinese Exclusion Act in 1882. America continually renewed these immigration bans on Asian Americans for several decades and treated current Asian Americans as second class citizens who were exploited for manual labor.

You asked if the current MCAT could still be considered to be a racist examination according to the parameters outlined before? Genao thinks so, but only against minorities that are underrepresented within medicine.

 

[deleted1001426]

Tbh I’ve always viewed the mcat as something which is fairly irrelevant to the quality of an applicant, rather it seems more like a means of medical schools to separate an otherwise fairly homogenous applicant pool. We all for the most part have pretty extensive leadership/volunteering/research /clinical experiences and fairly high gpas so if it weren’t for things like the mcat which provide a quantitative means of comparing 2 students it’d be nearly impossible for them to decide who to interview.

 

[theKingLT]

The author states that the MCAT is a poor predictor of overall med school performance and provides a link to a study to back it up, but then proceeds to state this with no link:

"The combination of grade point average, personal statement, conversations with writers of letters of recommendation, and time spent in clinical settings are much more predictive of future performance as a physician"

Anyone know to which study/studies this might be referring or is the author just assuming this?

Also, the author's assumption that MCAT is the most important part of the application simply by comparing general admissions rates of 501 vs "higher" is laughable. I think an 11% admissions rate for having a 501 would beat out those with no clinical experience any day.

 

[AnatomyGrey12]

The author states that the MCAT is a poor predictor of overall med school performance and provides a link to a study to back it up, but then proceeds to state this with no link:

"The combination of grade point average, personal statement, conversations with writers of letters of recommendation, and time spent in clinical settings are much more predictive of future performance as a physician"

Anyone know to which study/studies this might be referring or is the author just assuming this?

I’m pretty confident that statement is completely fabricated and not based on any actual study.

 

[the Mannis]

The author states that the MCAT is a poor predictor of overall med school performance and provides a link to a study to back it up, but then proceeds to state this with no link:

"The combination of grade point average, personal statement, conversations with writers of letters of recommendation, and time spent in clinical settings are much more predictive of future performance as a physician"

Anyone know to which study/studies this might be referring or is the author just assuming this?

The MCAT is far and away the best predictor of Step 1 and Step 2CK performance. I just saw a recent 2019 paper report that it was the best predictor of step 1 after doing a multivariable regression, and it had an r=.72 with step 1, and >r=.6 for 2CK. There’s at least a half-dozen other papers reporting similar predictive validity for the new MCAT. The author is either lying or doesn’t have a clue what she’s talking about.

 

[907914]

The MCAT is far and away the best predictor of Step 1 and Step 2CK performance. I just saw a recent 2019 paper report that it was the only predictor of step 1 after doing a multiple linear regression, and it had an r=.72 with step 1, and >r=.6 for 2CK. GPA didn’t account for any variance on step 1 or 2CK after controlling for the MCAT. Other studies have shown GPA does add some additional predictive power. There’s at least a half-dozen other papers reporting similar predictive validity for the new MCAT. The author is either lying or doesn’t have a clue what she’s talking about.

I am not too big of a statistics guy, but is 0.72 considered high? Like, not a perfect predictor but much better than random?

Is that value to say that “if you score this range on the MCAT, you have a 72% correlated chance at scoring within this Step1 range” basically? Or am I completely wrong (biostatistics was 5 years ago, sorry lol)

 

[the Mannis]

I am not too big of a statistics guy, but is 0.72 considered high? Like, not a perfect predictor but much better than random?

Is that value to say that “if you score this range on the MCAT, you have a 72% correlated chance at scoring within this Step1 range” basically? Or am I completely wrong (biostatistics was 5 years ago, sorry lol)

So psychologists use Cohen’s conventions for the strength of effect sizes. r=.1 is small, .3 is medium, .5 is large. At least 19/20 psych papers don’t achieve an r=.7. So a psych researcher would be popping champagne if they found something this strongly correlated

The r=.72 means that ~50% (.72 x .72) of the variance in Step 1 scores can be explained by variance in MCAT scores. So half of the differences in Step 1 scores are due to whatever factors lead to differences in MCAT scores. A large part, about half, of what's leading to students doing better or worse on Step 1 also drives MCAT score differences. The relationship holds after you control for GPA, top-funded NIH schools vs not top-funded, public vs private, and whatever other variables the researchers included in the paper (it’s cited below). When they controlled for the MCAT, none of the other variables had a significant p-value. It looks like the MCAT is a really good tool for predicting medical school academic performance.

You would make a killing if you could bet on Step 1 scores if you had access to MCAT data, if you were betting with someone who had no information to go on. So for your question, the r value indicates how strongly MCAT scores can predict Step 1 across the entire student population, not a single person. But you can see the likelihood of reaching a specific Step score based on your MCAT.

Here’s a graph provided by the AAMC, it gives you a good intuition of the relationship. The blue bars comprise the 25th-75th percentiles for scores on Step 1, and the black bars span the 10th-90th range. You can see the variability for any single score. For example, let's say you had a 510 MCAT; 50% of people, 25th-75th percentile with a 510 scored between 215 and 235. We don’t want to overlook the fact that there are very high Step 1 scores for the entire range of MCAT scores that medical schools admit.

The plot below shows an r=.62, whereas the paper I just saw a couple weeks ago was .72. The AAMC's data comes from a study using the current MCAT, and the study only included data from 16 schools. The .72 paper included data from 100 schools but it used data from the pre-2015 MCAT. You can see there’s a lot of variation in Step 1 performance for any single MCAT score, so plenty of people who didn’t score at the right tail on the MCAT end up doing very well on Step 1. Nevertheless, you can glean from the plot that the variability is uniform and the relationship is linear throughout the entire MCAT distribution , indicating that it remains predictive even out at the right tail. The MCAT appears to not have any ceiling effect.

An important takeaway from the paper I cited below is that beyond the MCAT, nothing else is predicting Step 1 scores, but 50% of the variance remains unexplained. This could be due to differences in the quality and quantity of prep for Step 1, motivation, stress and depression , small deviations in scores on test day due to randomness, etc.

Here’s the paper: Multivariable analysis of factors associated with USMLE scores across U.S. medical schools. - PubMed - NCBI
The paper the AAMC's data comes from: DOI: 10.1097/ACM.0000000000002942
If anyone's interested, heres an article published in Science; it's a huge meta-analysis of the predictive validity of standardized exams across almost all types of graduate schools: DOI: 10.1126/science.1136618

 

[Deecee2DO]

So psychologists use Cohen’s conventions for the strength of effect sizes. r=.1 is small, .3 is medium, .5 is large. At least 19/20 psych papers don’t achieve an r=.7. So a psych researcher would be popping champagne if they found something this strongly correlated

The r=.72 means that ~50% (.72 x .72) of the variance in Step 1 scores can be explained by variance in MCAT scores. So half of the differences in Step 1 scores are due to whatever factors lead to differences in MCAT scores. A large part, about half, of what's leading to students doing better or worse on Step 1 also drives MCAT score differences. The relationship holds after you control for GPA, top-funded NIH schools vs not top-funded, public vs private, and whatever other variables the researchers included in the paper (it’s cited below). When they controlled for the MCAT, none of the other variables had a significant p-value. It looks like the MCAT is a really good tool for predicting medical school academic performance.

You would make a killing if you could bet on Step 1 scores if you had access to MCAT data, if you were betting with someone who had no information to go on. So for your question, the r value indicates how strongly MCAT scores can predict Step 1 across the entire student population, not a single person. But you can see the likelihood of reaching a specific Step score based on your MCAT.

Here’s a graph provided by the AAMC, it gives you a good intuition of the relationship. The blue bars comprise the 25th-75th percentiles for scores on Step 1, and the black bars span the 10th-90th range. You can see the variability for any single score. For example, let's say you had a 510 MCAT; 50% of people, 25th-75th percentile with a 510 scored between 215 and 235. We don’t want to overlook the fact that there are very high Step 1 scores for the entire range of MCAT scores that medical schools admit.

The plot below shows an r=.62, whereas the paper I just saw a couple weeks ago was .72. The AAMC's data comes from a study using the current MCAT, and the study only included data from 16 schools. The .72 paper included data from 100 schools but it used data from the pre-2015 MCAT. You can see there’s a lot of variation in Step 1 performance for any single MCAT score, so plenty of people who didn’t score at the right tail on the MCAT end up doing very well on Step 1. Nevertheless, you can glean from the plot that the variability is uniform and the relationship is linear throughout the entire MCAT distribution , indicating that it remains predictive even out at the right tail. The MCAT appears to not have any ceiling effect.

View attachment 282973
An important takeaway from the paper I cited below is that beyond the MCAT, nothing else is predicting Step 1 scores, but 50% of the variance remains unexplained. This could be due to differences in the quality and quantity of prep for Step 1, motivation, stress and depression , small deviations in scores on test day due to randomness, etc.

Here’s the paper: Multivariable analysis of factors associated with USMLE scores across U.S. medical schools. - PubMed - NCBI
The paper the AAMC's data comes from: DOI: 10.1097/ACM.0000000000002942
If anyone's interested, heres an article published in Science; it's a huge meta-analysis of the predictive validity of standardized exams across almost all types of graduate schools: DOI: 10.1126/science.1136618

Preclinical grades in M1 and M2 are the best predictor of Step 1 performance-yes, better than MCAT performance. The MCAT and Step 1 are grossly different exams-the correlation isnt as strong. By doing well in the first two years of medical school it demonstrates 2 things: a strong foundation/grasp of the material tested on Step 1 and a strong work ethic-both of which are required to succeed on that exam

 

[theKingLT]

Preclinical grades in M1 and M2 are the best predictor of Step 1 performance-yes, better than MCAT performance. The MCAT and Step 1 are grossly different exams-the correlation isnt as strong. By doing well in the first two years of medical school it demonstrates 2 things: a strong foundation/grasp of the material tested on Step 1 and a strong work ethic-both of which are required to succeed on that exam

Maybe so, but I dont think this is really relevant in the context of the thread given that you can't use preclinical grades as an admissions indicator

Sent from my LG-H918 using SDN mobile

 

[Mass Effect]

What kind of mental illness are you talking about? The interview screens out the sociopaths and outward illnesses

Meh, sociopaths get admitted. They usually end up on the news years later. Part of being a sociopath is hiding that you're a sociopath when being screened for it.

Disagree. I see plenty of anxiety disorders. Depression.

How about instead of blocking those with anxiety and depressive disorders from gaining admission, we block those who discriminate against those with anxiety and depressive disorders? I like that idea much better.

 

[Deecee2DO]

Maybe so, but I dont think this is really relevant in the context of the thread given that you can't use preclinical grades as an admissions indicator

Sent from my LG-H918 using SDN mobile

fair enough

 

[751866]

So I just thought this was interesting.

That’s the headline from doximity today.
Here’s the article: It's time to abolish the MCAT

While I do understand the point that it screens people I wonder what the alternative would be?
“To be sure, the MCAT simplifies the admissions screening process and is good at predicting student performance on other standardized tests. But the MCAT score correlates poorly with overall performance in medical school and beyond.”

I think the point against them would be “good at predicting student performance on other standardized tests”. I mean becoming a doctor is just a series of standardized tests no?

Anyway, thought y’all be interested and have an entertaining discussion.

So basically the person who has a PhD thinks they’re entitled to admission at this school of their choosing because of the PhD, despite the school deciding (rightly or wrongly) that an MCAT cutoff is valuable to selecting students. The student still couldn’t hack the score after going full throttle at the exam. I don’t think that scores are all there is, but mentioning this in the article is a poorly constructed argument based on a logical fallacy (appeal to authority, the PhD).

It’s never the Yankees or the Patriots that complain about the rules that everyone plays by and comes to terms with when they play the game. Knowing the process for getting into and through medical school means you don’t get to stomp your feet and claim life isn’t fair when it doesn’t work out. You know this person wouldn’t have made any sort of comment on the process if they had been interviewed or admitted with a sub-500 mcat score (which isn’t that unreasonable of a hurdle).

 

[Mass Effect]

What is 500 under the old scoring system? Anyone familiar with the old system versus new system?

 

[Deecee2DO]

What is 500 under the old scoring system? Anyone familiar with the old system versus new system?

~25

 

[Ho0v-man]

What is 500 under the old scoring system? Anyone familiar with the old system versus new system?

25-26

 

[751866]

So psychologists use Cohen’s conventions for the strength of effect sizes. r=.1 is small, .3 is medium, .5 is large. At least 19/20 psych papers don’t achieve an r=.7. So a psych researcher would be popping champagne if they found something this strongly correlated

There are numerous problems with 'standardizing' whats "large" as a measure of association as this usually is field specific. A psych researcher might be happy but a physicist would not.

The r=.72 means that ~50% (.72 x .72) of the variance in Step 1 scores can be explained by variance in MCAT scores. So half of the differences in Step 1 scores are due to whatever factors lead to differences in MCAT scores. A large part, about half, of what's leading to students doing better or worse on Step 1 also drives MCAT score differences.

This is just incorrect. You're grossly conflating correlation and causation. The bivariate Pearson coefficient is only the R-squared for a univariable regression (i.e. only the outcome and that single predictor) and is totally incorrect if important variables are omitted (which you're doing by looking at it in isolation). Correlation and causation are not the same; r-squared and correlation are about linear covariation. Nothing more.

The relationship holds after you control for GPA, top-funded NIH schools vs not top-funded, public vs private, and whatever other variables the researchers included in the paper (it’s cited below). When they controlled for the MCAT, none of the other variables had a significant p-value. It looks like the MCAT is a really good tool for predicting medical school academic performance.

This is another misconception. Nonsignificant p-values do not mean no association/effect/relationship nor do "small" p-values mean "good predictor".

You would make a killing if you could bet on Step 1 scores if you had access to MCAT data, if you were betting with someone who had no information to go on. So for your question, the r value indicates how strongly MCAT scores can predict Step 1 across the entire student population, not a single person.

Again, not really. you can have a Pearson correlation of zero and have a deterministic (perfect prediction) non linear relationship. Don't believe it? Calculate r for a deterministic (i.e. don't add noise) quadratic function. In a perfect parabola, r is zero but there is one heck of an association (again, without noise, we have perfect prediction). Pearson correlation only indicates the strength of linear relationship between the variables, how tightly the cluster around the identity line and how they tend to move together (either on same or opposite sides of their respective means, on average).

An important takeaway from the paper I cited below is that beyond the MCAT, nothing else is predicting Step 1 scores, but 50% of the variance remains unexplained. This could be due to differences in the quality and quantity of prep for Step 1, motivation, stress and depression , small deviations in scores on test day due to randomness, etc.

Aside from an in sample r-squared there's nothing in your post about actual predictive ability (and r-squared isn't as much about prediction as it is explained variance): no mention of calibration, prediction errors, out of sample performance.

The paper also mentioned use of variable screening with p-value to then enter variables in a multivariable model; there is good literature nowadays showing this is incredibly bad at picking the "true variables" which further complicates the attempt at interpreting the model practically (especially causally as you did). It's bizarre when papers like this miss out on elementary statistical practice and then somehow know to look at model diagnostics like leverage and influence. Imagine the physician who doesn't know how to take a blood pressure but can manage complex disease.

 

[Mass Effect]

~25

25-26

Thanks!!

 

[Deecee2DO]

There are numerous problems with 'standardizing' whats "large" as a measure of association as this usually is field specific. A psych researcher might be happy but a physicist would not.

This is just incorrect. You're grossly conflating correlation and causation. The bivariate Pearson coefficient is only the R-squared for a univariable regression (i.e. only the outcome and that single predictor) and is totally incorrect if important variables are omitted (which you're doing by looking at it in isolation). Correlation and causation are not the same; r-squared and correlation are about linear covariation. Nothing more.

This is another misconception. Nonsignificant p-values do not mean no association/effect/relationship nor do "small" p-values mean "good predictor".

Again, not really. you can have a Pearson correlation of zero and have a deterministic (perfect prediction) non linear relationship. Don't believe it? Calculate r for a deterministic (i.e. don't add noise) quadratic function. In a perfect parabola, r is zero but there is one heck of an association (again, without noise, we have perfect prediction). Pearson correlation only indicates the strength of linear relationship between the variables, how tightly the cluster around the identity line and how they tend to move together (either on same or opposite sides of their respective means, on average).


Aside from an in sample r-squared there's nothing in your post about actual predictive ability (and r-squared isn't as much about prediction as it is explained variance): no mention of calibration, prediction errors, out of sample performance.

The paper also mentioned use of variable screening with p-value to then enter variables in a multivariable model; there is good literature nowadays showing this is incredibly bad at picking the "true variables" which further complicates the attempt at interpreting the model practically (especially causally as you did). It's bizarre when papers like this miss out on elementary statistical practice and then somehow know to look at model diagnostics like leverage and influence. Imagine the physician who doesn't know how to take a blood pressure but can manage complex disease.

how do you guys have this much time on your hands haha

 

[the Mannis]

There are numerous problems with 'standardizing' whats "large" as a measure of association as this usually is field specific. A psych researcher might be happy but a physicist would not.

This is just incorrect. You're grossly conflating correlation and causation. The bivariate Pearson coefficient is only the R-squared for a univariable regression (i.e. only the outcome and that single predictor) and is totally incorrect if important variables are omitted (which you're doing by looking at it in isolation). Correlation and causation are not the same; r-squared and correlation are about linear covariation. Nothing more.

This is another misconception. Nonsignificant p-values do not mean no association/effect/relationship nor do "small" p-values mean "good predictor".

Again, not really. you can have a Pearson correlation of zero and have a deterministic (perfect prediction) non linear relationship. Don't believe it? Calculate r for a deterministic (i.e. don't add noise) quadratic function. In a perfect parabola, r is zero but there is one heck of an association (again, without noise, we have perfect prediction). Pearson correlation only indicates the strength of linear relationship between the variables, how tightly the cluster around the identity line and how they tend to move together (either on same or opposite sides of their respective means, on average).


Aside from an in sample r-squared there's nothing in your post about actual predictive ability (and r-squared isn't as much about prediction as it is explained variance): no mention of calibration, prediction errors, out of sample performance.

1. Yes, physicists are more accurate at predicting the motion of Venus than psychologists are at predicting human behavior. The AAMC uses these conventions for their validity studies.

2. The R-squared from a regression tells you how much of the variance in the dep. variable is predictable from the ind. variable/variables. You can predict Step-1 scores more accurately with MCAT scores than any other variable on the pre-med application. 36-50% of the variance in Step-1 scores can be accounted for by differences in MCAT scores. Some of the factors leading to differences in Step-1 scores are also relevant in driving differences in MCAT scores. That was my point. Do you really dispute that? Some combination of factors leads to differences in Step-1 scores. The stronger the correlation, the more confidence we have that there's a causal relationship. I emphasized the fact that other factors matter too since it's not accounting for all or even most of the variance.

The paper showed the MCAT had by far the highest R-squared among the independent variables. The multivariable linear regression returned the following p-values: GPA ~ .15, School type private vs public ~ .19, Full-time faculty-student ratio ~ .57, MCAT ~ .0002. The difference is staggering.

3. I never said that a non-significant p-value indicates the absence of an effect/relationship. The drastic differences of the p-values in the multivariable linear regression suggests the MCAT is the variable we can be most confident in saying that there's a relationship with Step-1 scores.

4. "the r value indicates how strongly MCAT scores can predict Step 1 across the entire student population" This is true. The coefficient of determination informs how much variance is explained, and thus how much of y you can predict with x. The parabola thing is impertinent - the relationship between step-1 scores and MCAT scores is linear, not quadratic.

Here's a quote from the authors of the paper: "Univariate analysis (Table 2) suggests that all measured variables except total medical student enrollment are significant predictors of Step 1 and Step 2 scores, with MCAT having the highest correlation. Such corresponds with other studies utilizing different data sets, which indicate that MCAT is a strong predictor of medical school success, and thus positively correlates with Step scores".

The AAMC: "Using MCAT total scores and undergraduate GPAs provides better prediction of Step 1 scores than using either one alone." So the AAMC thinks the MCAT is a predictor of Step-1, and their data indicates that, from the variables on the pre-med application, it's the best predictor of Step-1 scores and preclerkship course performance.

Do you disagree with me about the predictive validity of the MCAT? That's the crux of this thread and my posts. If you don't like that paper, fair enough. There are plenty of papers showing a strong relationship between preclerkship course performance, step-1 scores and the MCAT.

 

[Chibucks15]

1. Yes, physicists are more accurate at predicting the motion of Venus than psychologists are at predicting human behavior. The AAMC uses these conventions for their validity studies.

2. The R-squared from a regression tells you how much of the variance in the dep. variable is predictable from the ind. variable/variables. You can predict Step-1 scores more accurately with MCAT scores than any other variable on the pre-med application. 36-50% of the variance in Step-1 scores can be accounted for by differences in MCAT scores. Some of the factors leading to differences in Step-1 scores are also relevant in driving differences in MCAT scores. That was my point. Do you really dispute that? Some combination of factors leads to differences in Step-1 scores. The stronger the correlation, the more confidence we have that there's a causal relationship. I emphasized the fact that other factors matter too since it's not accounting for all or even most of the variance.

The paper showed the MCAT had by far the highest R-squared among the independent variables. The multivariable linear regression returned the following p-values: GPA ~ .15, School type private vs public ~ .19, Full-time faculty-student ratio ~ .57, MCAT ~ .0002. The difference is staggering.

3. I never said that a non-significant p-value indicates the absence of an effect/relationship. The drastic differences of the p-values in the multivariable linear regression suggests the MCAT is the variable we can be most confident in saying that there's a relationship with Step-1 scores.

4. "the r value indicates how strongly MCAT scores can predict Step 1 across the entire student population" This is true. The coefficient of determination informs how much variance is explained, and thus how much of y you can predict with x. The parabola thing is impertinent - the relationship between step-1 scores and MCAT scores is linear, not quadratic.

Here's a quote from the authors of the paper: "Univariate analysis (Table 2) suggests that all measured variables except total medical student enrollment are significant predictors of Step 1 and Step 2 scores, with MCAT having the highest correlation. Such corresponds with other studies utilizing different data sets, which indicate that MCAT is a strong predictor of medical school success, and thus positively correlates with Step scores".

The AAMC: "Using MCAT total scores and undergraduate GPAs provides better prediction of Step 1 scores than using either one alone." So the AAMC thinks the MCAT is a predictor of Step-1, and their data indicates that, from the variables on the pre-med application, it's the best predictor of Step-1 scores and preclerkship course performance.

Do you disagree with me about the predictive validity of the MCAT? That's the crux of this thread and my posts. If you don't like that paper, fair enough. There are plenty of papers showing a strong relationship between preclerkship course performance, step-1 scores and the MCAT.
View attachment 283343View attachment 283344

Idk I beat my buddy on the MCAT by 4 points and he beat me by 20 on step

 

[the Mannis]

Idk I beat my buddy on the MCAT by 4 points and he beat me by 20 on step

hey stuff happens...

 

[Epilepsy365]

Ah excellence. More time about the MCAT not being a good quality measure.

Big whigs out there think that it’s time to make the MCAT more comprehensive by adding one more section and making it a 9 hr exam.

 

[theKingLT]

Idk I beat my buddy on the MCAT by 4 points and he beat me by 20 on step

That's meta-analysis level proof right there

Sent from my LG-H918 using SDN mobile

 

[Deecee2DO]

The problem with the MCAT is the material tested. it should be a physiology/biochem exam-you know, material thats actually somewhat relevant to med school. CARS chem physics and orgo gotta go. Psych can stay and maybe genetics. It would at least make applicants feel like they are better utilizing their time studying their ass off for 3 months

 

[751866]

2. The R-squared from a regression tells you how much of the variance in the dep. variable is predictable from the ind. variable/variables. You can predict Step-1 scores more accurately with MCAT scores than any other variable on the pre-med application. 36-50% of the variance in Step-1 scores can be accounted for by differences in MCAT scores.

You're conflating prediction and variance explained the same way that people think a logistic regression with high AUC is good at predicting the outcome. That's not sufficient in itself when talking about prediction. Further, everything you're talking about has zero cross validation or external validation in the paper which is really what matters. Again, the paper may offer R-squared but they miss a ton of other necessary components in assessing predictive utility.

Some of the factors leading to differences in Step-1 scores are also relevant in driving differences in MCAT scores. That was my point. Do you really dispute that? Some combination of factors leads to differences in Step-1 scores.

I agree in a logical sense that it's probably true, but you can't get that from interpreting the paper. It's a reasonable thought, but to suggest the paper shows that is absolutely conflating correlation and causation.

The stronger the correlation, the more confidence we have that there's a causal relationship.

Nope, not a chance. You're falling deeper into the correlation equals causation trap. Again, this is like saying that progression through the 21st century causes Autism because the correlation of incidence and time is so high-- or any time series for that matter. What do you think of the below correlation (and necessarily, the R-squared) on scientific spending is causing a rise in hanging suicides? Look at that correlation!!!!! Clearly this isn't the case.

Credit: 15 Insane Things That Correlate With Each Other

I emphasized the fact that other factors matter too since it's not accounting for all or even most of the variance.

Sure, reasonable point to make, but it doesn't mean that "Half the differences in step scores are what drives differences in MCAT scores" as you said.

The paper showed the MCAT had by far the highest R-squared among the independent variables. The multivariable linear regression returned the following p-values: GPA ~ .15, School type private vs public ~ .19, Full-time faculty-student ratio ~ .57, MCAT ~ .0002. The difference is staggering.

The paper also used a method of variable selection which is notoriously bad at picking "true" variables before sticking them in the model. When you do what they did, the p-values in the multivariable model are totally invalid without correction, the estimates (coefficients) are likely biased (systematically different from the truth, on average) and what's in the model often isn't right. There is good literature that supports this is a bad practice yet medicine continues to use it. First reference
Second reference

The problem is you might be able to get away with good predictions (but even then, literature supports better methods) but you can't interpret the model practically because it's well know there are these problems.

3. I never said that a non-significant p-value indicates the absence of an effect/relationship. The drastic differences of the p-values in the multivariable linear regression suggests the MCAT is the variable we can be most confident in saying that there's a relationship with Step-1 scores.

True, you didn't say that specifically, but you focused on a low pvalue for the MCAT as if it means a lot. See my comment just above: the p-values are invalid (they don't mention that they corrected them) based on the way they chose to include variables in the multivariable model. P-values aren't related to "confidence" as these are separate topics; the complement of a confidence level is an alpha level which are both preset before seeing data and are irrelevant after seeing the data. P-value is just a measure of incompatibility of the data with a particular null hypothesis (in this case maybe that the true slope is zero relating Y to the particular predictor). P-values have nothing to do with confidence levels.

4. "the r value indicates how strongly MCAT scores can predict Step 1 across the entire student population" This is true. The coefficient of determination informs how much variance is explained, and thus how much of y you can predict with x. The parabola thing is impertinent - the relationship between step-1 scores and MCAT scores is linear, not quadratic.

The fact is that a correlation coefficient isn't even half of the picture in terms of prediction; what would answer the question of prediction is out-of-sample R-squared, calibration, quantiles of prediction errors, average prediction errors, and a calibration curve. This is not really in dispute as there is TONS of literature to the point where this is routinely discussed in texts for working with and evaluating prediction models. The "parabola thing" is relevant because it's an easy counter example to your incorrect generalization that a Pearson correlation tells us how well the variable predicts the other; your argument doesn't hold to easy generalizations because the definition of the metric you are using isn't a generalized measure and is almost never a component used in assessing predictive ability. The other part about the MCAT and step 1 scores might be linear, but you don't really know that (I'd buy it), but at best, we have weak proof of that in the paper.

The AAMC: "Using MCAT total scores and undergraduate GPAs provides better prediction of Step 1 scores than using either one alone." So the AAMC thinks the MCAT is a predictor of Step-1, and their data indicates that, from the variables on the pre-med application, it's the best predictor of Step-1 scores and preclerkship course performance.
Do you disagree with me about the predictive validity of the MCAT? That's the crux of this thread and my posts. If you don't like that paper, fair enough.

I'm completely on your side with the idea that the MCAT (along with some other variables) are pretty decent at predicting medical school performance (and should remain in place), but I disagree with a lot of your interpretation of the statistical aspects of a poorly executed paper. And so, the issue I take is with misinterpretation of many aspects of what was done especially when these are well known misconceptions or don't address the stated goal of prediction.

I don't think there have actually been many high quality papers studying this, mostly because medicine (physicians) doesn't know it's foot from a TV when it comes to statistics.

I think getting rid of the MCAT, making Step 1 p/f are truly dumb moves. They're standardized, cumulative hurdles that everyone needs to take; the former is validated to rank order candidates by some combination of knowledge, critical thinking, and natural intelligence (not arguing the specific mix, but claiming the guy getting a 500 is just as intelligent as the 525 gal, on average, that's just ludicrous). The Step exams are at least designed and validated to distinguish between minimally competent examinees and incompetent (for the purpose of the body of knowledge and problem solving) and is probably well validated for also further rank ordering candidates (again by some combination of natural intelligence, knowledge, and critical thinking).

 

[751866]

The problem with the MCAT is the material tested. it should be a physiology/biochem exam-you know, material thats actually somewhat relevant to med school. CARS chem physics and orgo gotta go. Psych can stay and maybe genetics. It would at least make applicants feel like they are better utilizing their time studying their ass off for 3 months

After getting into practice you might disagree. I'm sure a bunch of people will say it's never helped them, but I'd be willing to bet if someone had some numbers, people who did better in these subjects more often think they're useful in practice than compared to people who did worse in these subjects.

Critical thinking is clearly important all the way in medicine and if a test is designed well to discern this ability in candidates, then absolutely. I see a big difference in peers or students who have an engineering, mathematics, or physics background compared to the run of the mill, memorizing bio major, on average.
Chem, orgo: I mean acid base might not be the same as a gen chem but it's definitely there, and pretty much all the other concepts from these two come into play with drug interactions, metabolism, physiology, drug distribution.... it goes on an on...some of the best docs are able to explain why a certain medication delivery is better (liposomal for example) or why certain medications can't get into the CSF...this is all basic chemistry. Orgo also helps with abstract thinking and mechanistic thinking.
Physics literally underlies everything you do (chem and orgo too): hypertension, trauma, Uhthoff's phenomenon... the goal isn't to memorize in medicine, it's to learn to problem solve and understand some common things very well so you can transport this knowledge to new problems or less familiar patient presentations.

Is it necessary? Easy for people to say no, hard to show the risks of not doing it: closest we can get is to say look at how an NP might "practice medicine" compared with someone who went to medical school. Foundational knowledge is different, med students complain about so much being irrelevant (yet having no frame of reference), but then one is clearly better at practicing medicine, because surprise, they studied foundations of medicine. Trying to pretend critical thinking, orgo, physics, chemistry, and so on aren't foundational in medicine is reaching.

 

[751866]

how do you guys have this much time on your hands haha

This doesn't take very long to recognize some common misconceptions perpetuated by medical schools, MPH/epi programs, test prep providers, and crappy online resources geared toward medical people.

 

[Deecee2DO]

After getting into practice you might disagree. I'm sure a bunch of people will say it's never helped them, but I'd be willing to bet if someone had some numbers, people who did better in these subjects more often think they're useful in practice than compared to people who did worse in these subjects.

Critical thinking is clearly important all the way in medicine and if a test is designed well to discern this ability in candidates, then absolutely. I see a big difference in peers or students who have an engineering, mathematics, or physics background compared to the run of the mill, memorizing bio major, on average.
Chem, orgo: I mean acid base might not be the same as a gen chem but it's definitely there, and pretty much all the other concepts from these two comes into play with drug interactions, metabolism, physiology, drug distribution.... it goes on an on...some of the best docs are able to explain why a certain medication delivery is better (liposomal for example) or why certain medications can't get into the CSF...this is all basic chemistry. Orgo also helps with abstract thinking and mechanistic thinking.
Physics literally underlies everything you do (chem and orgo too): hypertension, trauma, Uhthoff's phenomenon... the goal isn't to memorize in medicine, it's to learn to problem solve and understand some common things very well so you can transport this knowledge to new problems or less familiar patient presentations.

Is it necessary? Easy for people to say no, hard to show the risks of not doing it: closest we can get is to say look at how an NP might "practice medicine" compared with someone who went to medical school. Foundational knowledge is different, med students complain about so much being irrelevant (yet having no frame of reference), but then one is clearly better at practicing medicine, because surprise, they studied foundations of medicine. Trying to pretend critical thinking, orgo, physics, chemistry, and so on aren't foundational in medicine is reaching.

haha im going to have to disagree with 100% of this. There are other ways to test critical thinking you dont need useless orgo physics and CARS to do so. Test critical thinking in physio/bchem etc concepts that youre more likely to actually see again in preclinical years in med school

 

[751866]

haha im going to have to disagree with 100% of this. There are other ways to test critical thinking you dont need useless orgo physics and CARS to do so. Test critical thinking in physio/bchem etc concepts that youre more likely to actually see again in preclinical years in med school

Can you explain Uhthoff's phenomenon with a reasonable framework? What about why a patient with myopia has changes in vision until corrected (and how correction works)? (Rhetorical question, because a) absolutely, you took physics and know about some basic electronics concepts to propose a mechanism that is biologically and physically plausible and b) if you didn't, this is the internet so it's quick to look up). Do you want to shrug your shoulders when your patient with MS asks why this happens? This is low-level stuff so a physician telling the patient "IDK!" isn't really acceptable.

Looking beyond preclinical medicine, I can tell you there is 100% instances where thinking about basic concepts in orgo or physics has: helped explain something to a patient, helped interpret a study, recognized why certain interactions occur, recognized why a treatment works... list goes on. Doctor's arent' supposed to shrug their shoulders and said "I don't know" when a simple understanding of steric hindrance would help them explain to a patient why a certain medication works. It's easy to think things are irrelevant when we don't do well at them or don't have perspective to know how they're important. Talk to a clinician who you think is really brilliant (you know, the one who explains everything to you like you're 5 but all of a sudden you remember how the basic science all ties in and what it means for the pathophysiology). They can do this because they understand the fundamentals that underpin the topic very well. Go talk to a radiologist or rad onc person about why certain radiation is used for certain imaging or therapy; they won't say "beats me!" If they're good at their job, they'll say "oh some study showed..." If their great they'll say "this study showed... which makes a lot of sense because the density of this tissue and the energy in the... make this....instead of..."

I'm not saying it's tested optimally, but teaching to a test doesn't select better candidates when you need people who can think. In undergrad, didn't you notice the kids who complained that the test was "nothing like the homework or lecture" when in fact, every topic in lecture/homework was on the test but because it was a slightly stronger electrophile, a slightly less stable conformation... things fell apart for the kids who didn't understand the topic. They do great when they repeat the homework because it's the same problem, but they're not learning. So, I think putting preclinical type scenarios on the MCAT wouldn't hurt (and yeah, sure add physiology, but only if physiology [or any subject] is a hard entrance requirement-- if they plan to teach something they shouldn't necessarily see how good you are at it first...], but I don't think it's a better answer because it's more concrete. There's no secret that people who can solve abstract or less-familiar problems are generally more successful at solving familiar problems, but the people who are great at solving familiar problems aren't necessarily good at new situations (and medicine has a lot of unfamiliar territory you'll need to navigate).

The kids who I remember as standing out in medical school, including as we got to clinical years, were the kids with prior education that taught them to think: mathematics, engineering, physics, philosophy. One of the surest ways to detect people who best understand the core of a subject is to test them on it in new, unfamiliar, and abstract ways (you just need a good test to do that).

I'm open to seeing different testing, but to deny their role in the foundations of medicine is pretty outlandish and suggesting to test concepts (to be) taught in med school isn't the whole answer and probably isn't entirely relevant.

 

[Select All That Apply]

@delimitedtab You told a patient that steric hindrance was relevant to their care and they actually believed you? Who was this patient? Walter White?

 

[watermelon master]

@delimitedtab You told a patient that steric hindrance was relevant to their care and they actually believed you? Who was this patient? Walter White?

No. You see they broke it down into words that a 5 year old could understand. They didn’t just shrug their shoulders and say “idk”

 

[Select All That Apply]

No. You see they broke it down into words that a 5 year old could understand. They didn’t just shrug their shoulders and say “idk”

All things are made up of atoms. All atoms come with a given size, weight, and attraction to other atoms. Atoms combine to form molecules. Certain atoms combine to form functional groups that contribute to the effect of medications. Steric hindrance is when there is a stress related factor due to too many atoms occupying a given space that results in stress on the overall molecule. Steric hindrance can impact the effectiveness of the medication you are taking, therefore this is why you're getting the medication as the route and dose you are currently receiving it.

Btw, there is some vague connotation pharmacodynamics and also why I can't give you Percocet, Dilaudid, and Morphine within the span of one hour even if you threaten to hit me and sue me. It may be that respiratory failure and Narcanning you will be unpleasant could be more pertinent here, but you didn't listen to those reasons so now I am hitting you up with Organic Chemistry. Also, I can ask the physician to see if you can get Seroquel or Haldol because you appear to be very aggressive even with security on top of you.

I had an elderly woman recently tell me that I was abusing her for making her go through a standard EKG. I explained to her that she had presentation of elevated ST which was an indicator of MI and her heart potentially had some new presentation of dead tissue. She told me that the only thing that would be dying was me if she was forced to be attached to another set of wires.

Really curious who these patients are that will actually listen to steric hindrance. Maybe he's conflating patients for residents.

 

[751866]

@delimitedtab You told a patient that steric hindrance was relevant to their care and they actually believed you? Who was this patient? Walter White?

No. You see they broke it down into words that a 5 year old could understand. They didn’t just shrug their shoulders and say “idk”

All things are made up of atoms. All atoms come with a given size, weight, and attraction to other atoms. Atoms combine to form molecules. Certain atoms combine to form functional groups that contribute to the effect of medications. Steric hindrance is when there is a stress related factor due to too many atoms occupying a given space that results in stress on the overall molecule. Steric hindrance can impact the effectiveness of the medication you are taking, therefore this is why you're getting the medication as the route and dose you are currently receiving it.

Btw, there is some vague connotation pharmacodynamics and also why I can't give you Percocet, Dilaudid, and Morphine within the span of one hour even if you threaten to hit me and sue me. It may be that respiratory failure and Narcanning you will be unpleasant could be more pertinent here, but you didn't listen to those reasons so now I am hitting you up with Organic Chemistry. Also, I can ask the physician to see if you can get Seroquel or Haldol because you appear to be very aggressive even with security on top of you.

I had an elderly woman recently tell me that I was abusing her for making her go through a standard EKG. I explained to her that she had presentation of elevated ST which was an indicator of MI and her heart potentially had some new presentation of dead tissue. She told me that the only thing that would be dying was me if she was forced to be attached to another set of wires.

Really curious who these patients are that will actually listen to steric hindrance. Maybe he's conflating patients for residents.

I’ll keep it short and simple:
1) nice attacks more about a caricature rather than the actual statements
2) I actually haven’t had to explain where steric hindrance was important, but I did see a med Onc do it when a patient asked about chemo mechanisms between certain drugs (because he understood it, he could use analogies the patient understood—granted the patient had a masters degree); I gave common and less common examples of topics relevant to clinical medicine
3) if the only thing you can’t attempt to pick at is steric hindrance, I think I conveyed the point
4) I don’t use medical or technical terminology with patients because you should be able to explain the words to a lay person and you risk losing their understanding
5) the straw man comebacks are evidence that you lack a substantive argument; clearly all patients don’t ask why and then don’t get a basic science explanation in lay terms
6) I’m not super at it, but I’ve seen some clinicians who are phenomenal at putting complex topics into layman’s terms. Most are not

Also, I have no problem saying I don’t know to a patient, but overall, at minimum you can’t deny that these concepts are foundational in clinical medicine and pretending like they’re not is disingenuous.

 

[Select All That Apply]

@delimitedtab I did comment on your "actual statements". Me touching on them obviously touched you the wrong way because now you're distancing yourself from having written... "Doctor's arent' supposed to shrug their shoulders and said "I don't know" when a simple understanding of steric hindrance would help them explain to a patient why a certain medication works." Curious if you're thinking that my actual attempt of explaining steric hindrance was a caricature when you hadn't provided an actual example in the first place (per your following statement). How does one provide a metaphorical caricature of something that does not exist?

Am I obligated to discuss Uhthoff's phenomenon in my response? Do you not understand why a discussion about either Uthoff's phenomenon or steric hindrance in terms of informational delivery from a physician to a patient can be an impractical standard? Do I need to pretend like either of these topics matter when the purpose of both examples was illustrative and not demonstrative of some core point?

An inability to break down a complex topic into the base parts for a patient is an incredibly high ask considering the vast majority of a patient demographic is far less invested in the why or the how and far more invested in just having maximum healthcare outcomes. Even Feynman an established physicist whose primary interest was in being able to teach physics in a way that any undergraduate student could understand it invested many years to the endeavor and still constantly revised his methodology thereafter. Expecting physicians to be able to confer the knowledge that they have to their patients is a much larger mountain by my estimation considering this patient is going to be an exception and not the rule.

Don't understand why you have confidence to call my examples out as straw man when you don't know my specialty, patient demographic, or acuity. It seems like you deal with a very different category of patients in your day to day practice compared to mine. I'm sorry if my statements offended you which is why you feel like categorizing my responses as logical fallacies, I never intended to write a response that I thought you would be unable to internalize.

 

[deleted407021]

nerd s***

 

[751866]

@delimitedtab I did comment on your "actual statements". Me touching on them obviously touched you the wrong way because now you're distancing yourself from having written... "Doctor's arent' supposed to shrug their shoulders and said "I don't know" when a simple understanding of steric hindrance would help them explain to a patient why a certain medication works." Curious if you're thinking that my actual attempt of explaining steric hindrance was a caricature when you hadn't provided an actual example in the first place (per your following statement). How does one provide a metaphorical caricature of something that does not exist?

Am I obligated to discuss Uhthoff's phenomenon in my response? Do you not understand why a discussion about either Uthoff's phenomenon or steric hindrance in terms of informational delivery from a physician to a patient can be an impractical standard? Do I need to pretend like either of these topics matter when the purpose of both examples was illustrative and not demonstrative of some core point?

An inability to break down a complex topic into the base parts for a patient is an incredibly high ask considering the vast majority of a patient demographic is far less invested in the why or the how and far more invested in just having maximum healthcare outcomes. Even Feynman an established physicist whose primary interest was in being able to teach physics in a way that any undergraduate student could understand it invested many years to the endeavor and still constantly revised his methodology thereafter. Expecting physicians to be able to confer the knowledge that they have to their patients is a much larger mountain by my estimation considering this patient is going to be an exception and not the rule.

Don't understand why you have confidence to call my examples out as straw man when you don't know my specialty, patient demographic, or acuity. It seems like you deal with a very different category of patients in your day to day practice compared to mine. I'm sorry if my statements offended you which is why you feel like categorizing my responses as logical fallacies, I never intended to write a response that I thought you would be unable to internalize.

Pretty tricky on my phone, but I’ll try for kicks.
1) I’m standing by my statement, not distancing; my point of saying that I do think it’s okay to say “I don’t know” in some instances is to avoid someone claiming I think we need all the answers; I literally saw an attending explain steric hindrance using boats, and it worked for the patient
2) the caricatures are in your first two explanations; atoms, molecules, and functional groups and so on are not layman’s terms; telling a patient the boat is too big too get under the bridge is a layman concept people would understand;
3)I didn’t provide an actual example because it wasn’t needed to make the point that there are tons of applications
4) your reply doesn’t bother me, it was just I’ll-constructed
5) you don’t need to touch on Uhthoff’s phenomenon, but there are several less extreme examples of topics I gave
6) I agree this isn’t for all patients... I explicitly said this much. Like all explanations: right patient at the right time
7) the whole reason i brought this up, if you read earlier, is the claim that this stuff isn’t needed. Clearly it is. You’re shifting the argument to a straw man because I wasn’t making the argument that it’s practical all the time (because it’s obviously not)
8) Your examples weren’t in line with what I would consider layman’s terms, maybe the ekg being the closest, so I am pretty confident to say that’s not what I’m talking about. I don’t know why your background is relevant to the argument (which was originally, and still is, that these basic science concepts are important to physicians); you’re hung up on the more extreme example I gave but it still doesn’t deny the point that this is all relevant to medial education

You’re morphing the argument from “this is relevant for xyz” to “most patients won’t understand this!” ...that’s why I’m calling it a straw man.

You haven’t offended me, I understand your examples. Strangers on the internet don’t take up much space in my head.

 

[Select All That Apply]

@delimitedtab Okay.

 

[the Mannis]

You're conflating prediction and variance explained the same way that people think a logistic regression with high AUC is good at predicting the outcome. It's not sufficient in itself to talk about prediction. Further, everything you're talking about has zero cross validation or external validation in the paper which is really what matters. Again, the paper may offer R-squared but they miss a ton of other necessary components in assessing predictive utility.

I'm not only concerned with that specific paper. There are other papers showing the validity of the MCAT for predicting medical school performance. The AAMC has people dedicated to looking into this. The plots I've provided in previous posts were all from the AAMC's MCAT validity studies. My central claim is that the MCAT is a medium-to-strong predictor of medical school performance, especially step-1. The way to assess this is by judging the predictive validity of the MCAT. If you thought I made unjustifiable interpretations from that single paper, fine. But I'm not concerned with that single paper per se; the predictive validity of the MCAT is what matters.

Sure, reasonable point to make, but it doesn't mean that "Half the differences in step scores are what drives differences in MCAT scores" as you said.

Roughly half of the variance in Step 1 scores is explained by variance in MCAT scores. That is what R-squared tells you. I think there are shared factors leading to both of these differences. The 50% number isn't essential - it's a substantial proportion.

The predictive power of a test for some outcome measure is judged by the results of using a correlation or linear regression. In the study I cited, yes, they looked at both the MCAT and step-1 retrospectively. There are studies where they tracked medical students throughout medical school and residency and compared their step-1 performance to what the MCAT results predicted. This is assessing predictive validity. Here's one: (DOI) 10.1097/00001888-200510000-00010 Here's a figure from it:

The AAMC provides data themselves, and I already posted links to that data as well as some plots straight from the sources.

Nope, not a chance. You're falling deeper into the correlation equals causation trap. Again, this is like saying that progression through the 21st century causes Autism because the correlation of incidence and time is so high-- or any time series for that matter. What do you think of the below correlation (and necessarily, the R-squared) on scientific spending is causing a rise in hanging suicides? Look at that correlation!!!!! Clearly this isn't the case.

All else being equal, the stronger the correlation, the more likely it is that the relationship is causally connected. That's indisputable. Yes, correlation does not always equal causation, but there's a greater likelihood of a real relationship when the correlation is .95 rather than zero. The fact that there are tons of spurious and specious correlations doesn't negate the verity that many linear relationships are in fact causally connected. You posted the suicides by hanging and US spending on science and technology. Those two things are patently disparate. There's no intuitive basis to presume those two things would be casually connected. But here, we're talking about the MCAT and Step-1. They both are 8-hour exams where people have to answer multiple-choice questions that test the knowledge of the test-takers as well as their reasoning skills. These are very similar tasks. There's no a priori reason why you'd expect a correlation between these two things to be specious, whereas there is for suicide hangings and research spending.

The problem is you might be able to get away with good predictions (but even then, literature supports better methods) but you can't interpret the model practically because it's well know there are these problems.

If you can predict Step-1 scores with the MCAT, then the MCAT's legitimacy as a tool is validated. If you want to argue that this isn't sufficient to show a causal relationship, that's your prerogative. I can't imagine a sensible argument as to why these two things would have such a strong, linear relationship and still not be causally connected. From a practical perspective, the MCAT does what it was designed to do.

The fact is that a correlation coefficient isn't even half of the picture in terms of prediction; what would answer the question of prediction is out-of-sample R-squared, calibration, quantiles of prediction errors, average prediction errors, and a calibration curve. This is not really in dispute as there is TONS of literature to the point where this is routinely discussed in texts for working with and evaluating prediction models.

I admit I haven't read enough on prediction models. My understanding is that the predictive validity of a test is judged by the strength of its linear relationship with some outcome variable. It's what psychologists use for all sorts of research. While some psych research is nonsense, not all of it is. Some of the fundamental statistical techniques used by researchers of all stripes were developed by psychologists.

The other part about the MCAT and step 1 scores might be linear, but you don't really know that (I'd buy it), but at best, we have weak proof of that in the paper.

I posted data showing a linear relationship throughout the entire plot. It wasn't from the one paper though, it's the AAMC data.

I'm completely on your side with the idea that the MCAT (along with some other variables) are pretty decent at predicting medical school performance (and should remain in place), but I disagree with a lot of your interpretation of the statistical aspects of a poorly executed paper. And so, the issue I take is with misinterpretation of many aspects of what was done especially when these are well known misconceptions or don't address the stated goal of prediction.

I don't think there have actually been many high quality papers studying this, mostly because medicine (physicians) doesn't know it's foot from a TV when it comes to statistics.

Okay fair enough, you don't like the one paper. You made good points, some of my conclusions from that single article may have been unjustified. You sharpened up some of my arguments, thank you for that. Nonetheless, the aggregate of data from multiple papers substantiates my basic claim. The AAMC data and the other paper above in this post both address your skepticism with respect to predictive validity. The paper above directly addressed the predictive validity of the MCAT on medical school performance.

You think physicians are that statistically illiterate? I'd believe it; it's not emphasized nearly enough in undergrad or medical school. I think two semesters of statistics should be a requirement for pre-meds rather than calculus.

I think getting rid of the MCAT, making Step 1 p/f are truly dumb moves. They're standardized, cumulative hurdles that everyone needs to take; the former is validated to rank order candidates by some combination of knowledge, critical thinking, and natural intelligence (not arguing the specific mix, but claiming the guy getting a 500 is just as intelligent as the 525 gal, on average, that's just ludicrous). The Step exams are at least designed and validated to distinguish between minimally competent examinees and incompetent (for the purpose of the body of knowledge and problem solving) and is probably well validated for also further rank ordering candidates (again by some combination of natural intelligence, knowledge, and critical thinking).

This is spot-on.

 

[751866]

I'll start this post to be very clear that I agree the MCAT is probably a pretty good indicator of medical school and step success, at least in terms of passing. However my issue is not with your interpretation of the specific paper, but the interpretation in general is not accurate in the areas I pointed out and gave references for in several areas. The term prediction and predictive ability are specific terms with specific meaning in statistics and there is far more to demonstrating those qualities than you suggested with a pearson correlation or R-squared (somewhat of a redundant argument).

I'm not only concerned with that specific paper. There are other papers showing the validity of the MCAT for predicting medical school performance. The AAMC has people dedicated to looking into this. The plots I've provided in previous posts were all from the AAMC's MCAT validity studies. My central claim is that the MCAT is a medium-to-strong predictor of medical school performance, especially step-1. The way to assess this is by judging the predictive validity of the MCAT. If you thought I made unjustifiable interpretations from that single paper, fine. But I'm not concerned with that single paper per se; the predictive validity of the MCAT is what matters.

I agree that the MCAT is probably the best currently available, validated measure designed to order test takers based on a combination of factors in several domains. This includes the utility of the test to separate likely successful and unsuccessful students for medical school admissions.

Roughly half of the variance in Step 1 scores is explained by variance in MCAT scores. That is what R-squared tells you. I think there are shared factors leading to both of these differences. The 50% number isn't essential - it's a substantial proportion.

I understand the interpretation of R-squared as you can see in my prior posts. You may think there are shared factors and there might be, but there might not be, and you also claimed same causality of these differences, which is a much bigger stretch.

The predictive power of a test for some outcome measure is judged by the results of using a correlation or linear regression. In the study I cited, yes, they looked at both the MCAT and step-1 retrospectively. There are studies where they tracked medical students throughout medical school and residency and compared their step-1 performance to what the MCAT results predicted. This is assessing predictive validity.

I can't restate what I've previously said, but predictive ability is not at all defined by correlation and within a regression you need to look at out-of-sample R-squared (assuming OLS), calibration, and certain quantiles of prediction error to see if it is actually predicting well. A single, in-sample Pearson correlation doesn't tell you this, especially not in a multivariable setting.

All else being equal, the stronger the correlation, the more likely it is that the relationship is causally connected. That's indisputable.

Not only indisputable, it's easily proven incorrect.
The below are two, ridiculously obvious non-causal associations. The top one has a much higher correlation. These are pretty similar in other material regards (like plausibility). Your argument that the top panel is more likely causal doesn't hold water.

Yes, correlation does not always equal causation, but there's a greater likelihood of a real relationship when the correlation is .95 rather than zero.

Again, not true. Nonlinear relationships can have zero or very non zero correlations (Pearson and Spearman) but strong nonlinear relationships. If the non linear one is causal then your point is shown not generally true. You're circularly falling into the correlation = causation trap. It's not even an outlier case for correlation to not equal causation-- it's pretty much the norm (even if we ignore the fact that you keep ignoring non linear associations).

The fact that there are tons of spurious and specious correlations doesn't negate the verity that many linear relationships are in fact causally connected.

Sure, no one is arguing that. Nonlinear relationships may also be causal.

You posted the suicides by hanging and US spending on science and technology. Those two things are patently disparate. There's no intuitive basis to presume those two things would be casually connected.

Right, you're proving my point that your interpretation, in this paper and in general, is incorrect.

But here, we're talking about the MCAT and Step-1. They both are 8-hour exams where people have to answer multiple-choice questions that test the knowledge of the test-takers as well as their reasoning skills. These are very similar tasks. There's no a priori reason why you'd expect a correlation between these two things to be specious, whereas there is for suicide hangings and research spending.

Right. They are both proxies for intelligence, work ethic, content exposure, and probably some other factors, but the point being is that randomizing students to receive a 525 on the MCAT will not tend to cause higher Step 1 scores; if it did, then you'd have a stronger argument for causality. This is literally how randomized experiments (well designed clinical trials) work and why observational studies are much further from generating good causal inference than people in medicine think. If you really think this is causal, you'd stand by the notion that randomly assigned MCAT scores will tend to produce certain Step 1 scores and medical school performance (which we know is clearly false). When we randomize patients to receive an ARB or placebo and there is a large enough difference in blood pressures after medication, we might conclude its the ARB...

If you can predict Step-1 scores with the MCAT, then the MCAT's legitimacy as a tool is validated.

Prediction and validation isn't really binary; like I said sure you can always get predictions, but how is your method better than tossing darts or flipping a coin? There are several things that need to be looked at before we can actually make that claim. Is the MCAT likely a decent predictor? I think so, because it has a plausible relationship to important factors of Step 1 scores, but they really haven't shown it to be a great one. No one has attempted to quantify how close predicted and actual step 1 scores are based on some MCAT-employing prediction model (again on new data).

If you want to argue that this isn't sufficient to show a causal relationship, that's your prerogative. I can't imagine a sensible argument as to why these two things would have such a strong, linear relationship and still not be causally connected. From a practical perspective, the MCAT does what it was designed to do.

It's not my prerogative, exactly. My point was that your interpretations were largely incorrect. Again, I'm not arguing with your conclusion, I'm saying your presented rationale isn't particularly strong.

I admit I haven't read enough on prediction models. My understanding is that the predictive validity of a test is judged by the strength of its linear relationship with some outcome variable. It's what psychologists use for all sorts of research. While some psych research is nonsense, not all of it is. Some of the fundamental statistical techniques used by researchers of all stripes were developed by psychologists.

I think if you back up you'll see the issue here. You're stuck on linear and a single number that doesn't contain the information you need to make the assessment. Just because tons of people do it incorrectly doesn't mean it's right. Most clinicians with publications incorrectly and often egregiously employ and interpret statistical methods, but that doesn't mean it's right. Clinicians, like psychologists, are not statisticians. I agree that there are numerous methods developed by psychologists (often who have a very quantitative background including graduate statistics courses in a stat department), but that's neither here nor there when we're talking about well established methods of evaluating predictive ability. Psychologists aren't the ones pushing the envelop on that front; they gave us mystical things like factor analysis (although the same guy gave us more useful stuff like Spearman's rho) and terribly flawed and popular "agreement" statistics like Kappa.

I posted data showing a linear relationship throughout the entire plot. It wasn't from the one paper though, it's the AAMC data.

It might be linear, but we don't know, that's why we use statistics and probability.

Okay fair enough, you don't like the one paper. You made good points, some of my conclusions from that single article may have been unjustified. You sharpened up some of my arguments, thank you for that. Nonetheless, the aggregate of data from multiple papers substantiates my basic claim. The AAMC data and the other paper above in this post both address your skepticism with respect to predictive validity.

Again, I like the idea of MCAT being useful to predict step 1, but your data and the papers don't support the predictive utility. There are tons of missing pieces; at the very least, out of sample measures like R-squared, the model standard deviation, quantiles of prediction errors, calibration plots...What you're doing right now is the equivalent of declaring secondary hypertension because you measured a single blood pressure of 144/80 in a healthy adult around 30 years old...there's a lot missing to that picture.

You think physicians are that statistically illiterate? I'd believe it; it's not emphasized nearly enough in undergrad or medical school. I think two semesters of statistics should be a requirement for pre-meds rather than calculus.

As a group, yes. I agree at least 2 semesters of applied statistics should be a requirement, but ideally some lower level mathematics (through calc and linear algebra). Startling how many kids in my class complained they needed to "memorize these 7 pharm formulas" that came from plugging in and manipulating 2-3 original formulas...and the fact that for some reason courses like this tend to teach students problem solving skills they lack. Honestly, psychology and sociology should have been back-burnered as a requirement instead of statistics and mathematics because the schools do an okay job teaching the psych and people skills stuff (and those two subjects aren't rocket science), but the schools fail horribly at teaching statistics to their students (mainly blind leading the blind).

Very nice

What happened? You go from agreeing with that comment to giving the Borat?

 

[the Mannis]

I can't restate what I've previously said, but predictive ability is not at all defined by correlation and within a regression you need to look at out-of-sample R-squared (assuming OLS), calibration, and certain quantiles of prediction error to see if it is actually predicting well. A single, in-sample Pearson correlation doesn't tell you this, especially not in a multivariable setting.

The wiki for predictive validity states that you can determine the predictive power by using a linear regression. Here's a definition from wikipedia: "In psychometrics, predictive validity is the extent to which a score on a scale or test predicts scores on some criterion measure." You disagree with this definition?

I don't know enough statistical science to respond to your comments about calibration, cross-validation, prediction error, etc. This is frustrating. But answer this, if you don't mind. If you collect MCAT data, track students through medical school, gather their step-1 data, and plot a linear regression, how is this not assessing the predictive validity of the MCAT? The linear regression assesses how tightly the independent variable can predict the outcome variable. The tighter the fit, the better it is at predicting the outcome variable. Wouldn't cross-validation just be using the linear regression on new MCAT/step-1 data? If study after study with huge sample sizes from various schools show that the MCAT can predict scores on step-1, what is missing here? Are the AAMC and the authors of the 2005 paper making unjustified claims in your view?

Right. They are both proxies for intelligence, work ethic, content exposure, and probably some other factors, but the point being is that randomizing students to receive a 525 on the MCAT will not tend to cause higher Step 1 scores; if it did, then you'd have a stronger argument for causality. This is literally how randomized experiments (well designed clinical trials) work and why observational studies are much further from generating good causal inference than people in medicine think. If you really think this is causal, you'd stand by the notion that randomly assigned MCAT scores will tend to produce certain Step 1 scores and medical school performance (which we know is clearly false).

wait what??? I never argued that getting the score is the factor leading to a better step-1 score. The argument for causality was that the factors leading to differences in MCAT scores carry over into medical school and lead to some of the differences on step-1. The things that make students answer more questions correctly on the MCAT (the factors you listed) are behaving similarly in the future on step-1 to some extent. That's the case for causality, not the score itself... I trust you'll clear up that confusion, because I don't believe that you thought I was arguing that.

Prediction and validation isn't really binary; like I said sure you can always get predictions, but how is your method better than tossing darts or flipping a coin? There are several things that need to be looked at before we can actually make that claim. Is the MCAT likely a decent predictor? I think so, because it has a plausible relationship to important factors of Step 1 scores, but they really haven't shown it to be a great one.

The AAMC and the 2005 paper both state that MCAT data was predictive of future step-1 performance. How is it better than flipping a coin? If I flip a coin 100 times to guess if 100 students will score above or below 229, I'll be right 50% of the time, on average. If I assume a linear relationship with a correlation of 1 between the two exams, the equation will spit out step-1 scores when MCAT scores are inputted. Obviously the correlation is not 1, but the correlation of .6-.7 will lead to better predictions than just choosing heads or tails. I'll do a better job of predicting above/below 229 for 100 students with this method than if I were to flip a coin, especially for the left and right tails of the MCAT distribution. That's better, no?

Psychologists aren't the ones pushing the envelop on that front; they gave us mystical things like factor analysis (although the same guy gave us more useful stuff like Spearman's rho)

mystical lol I didn't know factor analysis was controversial

It might be linear, but we don't know, that's why we use statistics and probability.

Doesn't the data from the graph display a linear relationship...? What is missing from that plot?

What happened? You go from agreeing with that comment to giving the Borat?

hahaha I've never seen Borat, but I'll rescript it to what I wrote initially.
Thanks for the the thorough replies, this has been productive.

 

[751866]

The wiki for predictive validity states that you can determine the predictive power by using a linear regression. Here's a definition from wikipedia: "In psychometrics, predictive validity is the extent to which a score on a scale or test predicts scores on some criterion measure." You disagree with this definition?

I think it's an incomplete definition. You go through a model fitting and calibrating process, so yes you can use a regression (again let's assuming a linear regression), but that's like saying you worked up the hypertension to show it was secondary hypertension. There's a few right ways this could go and some wrong ones (namely, just taking a single BP). The problem is that prediction is actually very hard and which is why there are multiple components you need to look at, but you need to first fit the regression to get these components.

I don't know enough statistical science to respond to your comments about calibration, cross-validation, prediction error, etc.

Basically, I can fit a model and fine tune it to what I think looks really good (even some model based statistics like R-squared and the model standard deviation, calibration plot [actual vs predicted scatter plot] look good), but the question is whether I fit the model to the data too much or if I generalized the relationship appropriately (in the prediction context). So, I can use methods with the data at hand to simulate what the performance might look like on new data (although we're not technically using new data) OR I can get truly new data, plug in the independent variables to the equation we developed, and then see how close the new observed Y values are to the new predicted Y values; you want to look at how large the typical error of prediction is, including some of the quantiles such as the 90th percentile absolute error (as in, how large of a difference was their between actual and predicted where 90% of all errors are not larger?).

This is frustrating. But answer this, if you don't mind. If you collect MCAT data, track students through medical school, gather their step-1 data, and plot a linear regression, how is this not assessing the predictive validity of the MCAT?

It is, but poorly. Eye ball methods aren't always great so there are several quantities that we want to look at. What the AAMC should do (maybe they have internally, I don't know) is literally develop a model to predict step 1. Then in a prospective manner, use that model to assign predicted step 1 scores to students and when the students actually take and receive a step 1 score, compare these values in aggregate using things like we discussed. Just showing the correlation, as I said, I like showing a single isolated elevated BP and saying it's secondary hypertension. The other issue is that the MCAT might be a good predictor, but maybe we can do better; correlation coefficients are 2 dimensional, so it's not very useful in itself in most all cases because more than 1 independent variable is important.

The linear regression assesses how tightly the independent variable can predict the outcome variable. The tighter the fit, the better it is at predicting the outcome variable. Wouldn't cross-validation just be using the linear regression on new MCAT/step-1 data? If study after study with huge sample sizes from various schools show that the MCAT can predict scores on step-1, what is missing here? Are the AAMC and the authors of the 2005 paper making unjustified claims in your view?

What your saying about tightness of fit needs to be on data other than those used to assess the relationship and create a model in the first place, and the things we talked about will help assess that. Cross validation could be thought of as new data; I'm referring specifically to things they could do with the data at hand that would be better than nothing (what they did, unless I missed this in their paper), and this is under the broader category of resampling, bootstrapping, k-folds-- but if they could get brand new data that weren't used before to make the model, sure that's ideal. I think the issue is somewhat philosophical and epistemological because they haven't done enough in terms of good practice to demonstrate strength of prediction. If I told you I had a model to predict whether it will rain today would you like to use it and make a bet, you should ask for more than just a simplistic summary statistic, especially when it lacks recognition of the other important variables.

I wouldn't be surprised if MCAT is a good predictor, but I just don't think they've done enough things publicly to show that; I'm sure their internal development is more than what they present in these papers. I think the 2005 paper specifically is a of proof of concept but needs to be taken further (again, I agree on your suggestion that the MCAT is a good predictor, and I think it's one of a few that we could use in concert, but I haven't seen a paper doing the full work up).

wait what??? I never argued that getting the score is the factor leading to a better step-1 score. The argument for causality was that the factors leading to differences in MCAT scores carry over into medical school and lead to some of the differences on step-1. The things that make students answer more questions correctly on the MCAT (the factors you listed) are behaving similarly in the future on step-1 to some extent. That's the case for causality, not the score itself... I trust you'll clear up that confusion, because I don't believe that you thought I was arguing that.

My misunderstanding then-- you were saying higher correlations...causal...MCAT...Step 1, so I misunderstood and thought you were saying something like that. Seems like we're on the same page then.

The AAMC and the 2005 paper both state that MCAT data was predictive of future step-1 performance. How is it better than flipping a coin? If I flip a coin 100 times to guess if 100 students will score above or below 229, I'll be right 50% of the time, on average. If I assume a linear relationship with a correlation of 1 between the two exams, the equation will spit out step-1 scores when MCAT scores are inputted. Obviously the correlation is not 1, but the correlation of .6-.7 will lead to better predictions than just choosing heads or tails. I'll do a better job of predicting above/below 229 for 100 students with this method than if I were to flip a coin, especially for the left and right tails of the MCAT distribution. That's better, no?

Somewhat rhetorical since my point was that all the other stuff I described about prediction is how we know if it's better than flipping a coin or how a particular model to predict Step 1 with {some set of predictors} is better than another model with {a different set of predictors}-- which is an important question for admission purposes.

mystical lol I didn't know factor analysis was controversial

Principal components analysis is not generally controversial, but factor analysis is and the two are often conflated. Both are kinds of ways to approach underlying covariance; generally the former is for dimension reduction where the latter is finding underlying structure and people try to put meaning on this structure. Factor analysis, as I have seen and heard, is a bit of hocus pocus. Mathematically it's fine, but finding "hidden constructs" in the data then attempting to name these weighted-average "factors" into something meaningful is where it starts to get hairy; people even further eff this up by "finding new factors/constructs", creating scores and then using them in regression as if they actually mean things. There can be good factor analyses done that are reasonable, but psychological ideas are very hard to wrangle. I recently was talking with a friend who has a good math background (bachelors in math) who did a PhD in quantitative psych and he was saying how the more research he does the more he sees why social sciences get crapped on for poorly done hand wavy things; it's hard enough that the things people are trying to measure are often not measured by the variables or scales they create and validate, but then things are applied so poorly on top of that. (This is in contrast to PCA for various reasons.)

Doesn't the data from the graph display a linear relationship...? What is missing from that plot?

I'm not trying to be difficult-- it's just as simple as that it is linear in the sample (roughly), although one of them had a clear nonlinearity in the lower end of x. In general, looking at the sample to see what the relationship is can only tell us what it looks like in the sample. I think it's reasonable to assume rough linearity in this case for various reasons, but in general, if we took a clinical trial paper and just looked at the graph to assert the relationship is linear or not wouldn't be a good idea (again in philosophical and epistemological terms, which statistics uses heavily).

I think we should PM if we chat more on this so the thread isn't totally hijacked, even though this is somewhat relevant, we're not really disagreeing on the main point.

 

[IsleyOfTheNorth]

When I was in the military, we were expected to fold our underpants into 4-inch squares. There is no conceivable use for this skill other than as a test of our ability to follow instructions.

I view the MCAT as a hoop which takes time and effort to jump through. Does it predict medical school success? No idea. But it does require dedication, practice, good study habits, and the ability to thrive under pressure to get a "passing" score.

In my opinion, I don't see the MCAT as a predictor, but rather as a filter.

 

[Osteosaur]

Part of the MCAT is being a hoop/test of hill, and a general test of standardized testing ability. I spent 9 months preparing while working and pouring over a test book for 2-3 hours a night. But for all that, supposedly, COMLEX correlates way better with grades than MCAT with students at my school.

I think one of the problems with the author's idea is that being perceived as a superstar by your college professors or people in your community is also not going to correlate well with being a physician. There does need to be some standardized measure. Maybe its a matter of using the MCAT to winnow in on those college subjects that will reappear in medical school and boards in some way. CARS is an absolute disgrace, and its heartbreaking if someone is kept out of medical school because they can't parse a poorly written essay on Ayne Rand.

 

[Ho0v-man]

Part of the MCAT is being a hoop/test of hill, and a general test of standardized testing ability. I spent 9 months preparing while working and pouring over a test book for 2-3 hours a night. But for all that, supposedly, COMLEX correlates way better with grades than MCAT with students at my school.

I think one of the problems with the author's idea is that being perceived as a superstar by your college professors or people in your community is also not going to correlate well with being a physician. There does need to be some standardized measure. Maybe its a matter of using the MCAT to winnow in on those college subjects that will reappear in medical school and boards in some way. CARS is an absolute disgrace, and its heartbreaking if someone is kept out of medical school because they can't parse a poorly written essay on Ayne Rand.

yeah I kinda interpret the bolded part as code for “I convinced everyone else that I’m amazing. It’s not fair that objective evidence should exist that states otherwise!” We’ve all seen these types in med school who paint themselves as a super star with no flaws despite barely hanging on.

 

[751866]

yeah I kinda interpret the bolded part as code for “I convinced everyone else that I’m amazing. It’s not fair that objective evidence should exist that states otherwise!” We’ve all seen these types in med school who paint themselves as a super star with no flaws despite barely hanging on.

Right. The PhD holder seemed to think that necessarily gave them an advantage or a golden ticket to the school of their choice. The student couldn’t hack it on the standardized test even after multiple attempts with focused studying.