Question on a paper I am working on...

[Doctor_Strange]

Hi,

I am studying exercise effects on working memory using a meta analysis. Is it allowed to group different working memory tests together for a single study? As in, I have Study 1 that measures WM using 2 different tests, but I want to conduct a meta regression on it and if I am allowed to combine the 2 different tests that are measuring working memory, I can than do my regression...

---

Members don't see this ad.

 

[WisNeuro]

Hi,

I am studying exercise effects on working memory using a meta analysis. Is it allowed to group different working memory tests together for a single study? As in, I have Study 1 that measures WM using 2 different tests, but I want to conduct a meta regression on it and if I am allowed to combine the 2 different tests that are measuring working memory, I can than do my regression...

Short answer, depends.

Are test scores raw scores? Standardized Scores? Are the population samples similar in demographics? And so on and so forth. Hard to know the answer to the question without knowing the characteristics of both the test and the samples.

 

[Doctor_Strange]

Sorry, let me add the following details:

• I am doing a systematic review with a meta analysis so by a function of my strict search strategy and inclusion/exclusion criteria the population samples are quite similar (the only real difference is age which is in fact what I want to regress to the impact of age).
• The test scores are all post-test.

 

[Doctor_Strange]

I am using a program called Comprehensive Meta Analysis and it does essentially all of the calculations. The final meta regression will be in units of standard difference in means.

edit: this is where I have added the graphs for input from others https://forums.studentdoctor.net/th...w-regarding-cognition-working-memory.1231131/

 

[Ollie123]

I am using a program called Comprehensive Meta Analysis and it does essentially all of the calculations. The final meta regression will be in units of standard difference in means.

edit: this is where I have added the graphs for input from others https://forums.studentdoctor.net/th...w-regarding-cognition-working-memory.1231131/

Your responses don't help answer the question. This is a conceptual question, not really a statistical one. There are certainly ways to include multiple measures from the same study...the question is just whether or not you should do that given the goals of the project, the nature of the measures, etc. Do you have a faculty mentor or someone working on this project with you? This is the sort of thing you should be discussing with someone heavily involved in the project who knows the area....

 

[Doctor_Strange]

Unfortunately, my mentor is not well versed in this particular analysis.

 

[Doctor_Strange]

I need to measure age as a moderator given a recommendation by one of the reviewers who commented on my initial manuscript submission. So it definitely is needed, and I think I am more concerned statically not conceptually. Conceptually I know it is appropriate to include given my goals as well as review prior reviews and how they included their regressions. I just don't know the problems that arise if any if I combine the effect size of a digital span forward test and the effect size of a n back task basically I have a unit of analysis dilemma..

 

[WisNeuro]

Hard to know much without seeing all of the data. Are there any quant people in your department who could consult with on this?

 

[Ollie123]

The neuropsych folks here are better poised to address this particular issue, but last I knew there was significant debate about whether DS Forward is really a test of working memory. I thought the general consensus was "No" but again, will defer to others on the matter. That's really the critical issue here and that is what I meant by it being a conceptual one, not a statistical one. You can combine anything in a meta-analysis, the question is whether you should.

 

[WisNeuro]

The neuropsych folks here are better poised to address this particular issue, but last I knew there was significant debate about whether DS Forward is really a test of working memory. I thought the general consensus was "No" but again, will defer to others on the matter. That's really the critical issue here and that is what I meant by it being a conceptual one, not a statistical one. You can combine anything in a meta-analysis, the question is whether you should.

I do not technically consider DS forward WM, it's really just basic verbal attention.

 

[psych.meout]

The neuropsych folks here are better poised to address this particular issue, but last I knew there was significant debate about whether DS Forward is really a test of working memory. I thought the general consensus was "No" but again, will defer to others on the matter. That's really the critical issue here and that is what I meant by it being a conceptual one, not a statistical one. You can combine anything in a meta-analysis, the question is whether you should.

Exactly. You can't really be conflating two measures like that in your analysis without substantiating that they are measuring the same construct(s) in the same way(s). Just because they seem to similar doesn't mean that they actually are. You don't want to be caught making an apples-to-oranges comparison.

You should do a lit search and consult with any neuropsych faculty to see what the existing research says about this and make sure you have some sources to cite that support what you are doing, both conceptually and statistically.

 

[Doctor_Strange]

Thanks for the replies from everyone--genuinely appreciate it.

Long story short: I wrote this paper my last semester in college, I am now in my gap year no longer at that school, and since it was a small liberal arts college literally no one there every conducted a meta analysis. I went to the statistical expert int he psych department and he even said he really could not help me then. I have had to learn meta analyses essentially on my own by watching YouTube videos, reading previous reivews, etc.

Using DS-F as an example, and not to get into a debate regarding this, but previous studies that examined the effect of exercise on cognition identified DS-F among several other tests that measure working memory. I am aware that others out there may disagree, but I have explicitly stated in my manuscript that the studies that I have included follow the thinking paradigm of previous reviews like the one I just mentioned. So putting that aside, from @psych.meout 's comment, I feel comfortable and confident to state in my manuscript that the measures that are combined do indeed measure the same construct (working memory), but in different ways. I have found 2 reviews that displayed tabular "combined effect sizes" in a manner that I wish to do. So I will make sure to cite that.

I know this is an odd way to get help since I cannot show the entire manuscript but long story short this paper was needing revisions in this particular area, everything else was deemed appropriate.

 

[WisNeuro]

You still need to know if the scores were standardized in some way or not. If they were, they have already been corrected for age, presumably, so doing so again will lead to a flawed analysis. Honestly, this needs a consultation with a stats person in the area. Depending on how much the publication matters to you, you could always hire a neuro/quant person for a few hours of time.

 

[Doctor_Strange]

You still need to know if the scores were standardized in some way or not. If they were, they have already been corrected for age, presumably, so doing so again will lead to a flawed analysis. Honestly, this needs a consultation with a stats person in the area. Depending on how much the publication matters to you, you could always hire a neuro/quant person for a few hours of time.

From what I have retrieved from my 15 articles, none of the scores were standardized. My meta analysis, however, transformed the data ultimately in standardized mean differences.

 

[psych.meout]

Thanks for the replies from everyone--genuinely appreciate it.

Long story short: I wrote this paper my last semester in college, I am now in my gap year no longer at that school, and since it was a small liberal arts college literally no one there every conducted a meta analysis. I went to the statistical expert int he psych department and he even said he really could not help me then. I have had to learn meta analyses essentially on my own by watching YouTube videos, reading previous reivews, etc.

Using DS-F as an example, and not to get into a debate regarding this, but previous studies that examined the effect of exercise on cognition identified DS-F among several other tests that measure working memory. I am aware that others out there may disagree, but I have explicitly stated in my manuscript that the studies that I have included follow the thinking paradigm of previous reviews like the one I just mentioned. So putting that aside, from @psych.meout 's comment, I feel comfortable and confident to state in my manuscript that the measures that are combined do indeed measure the same construct (working memory), but in different ways. I have found 2 reviews that displayed tabular "combined effect sizes" in a manner that I wish to do. So I will make sure to cite that.

I know this is an odd way to get help since I cannot show the entire manuscript but long story short this paper was needing revisions in this particular area, everything else was deemed appropriate.

Please, correct me if I'm wrong. Are you comparing studies where only DS-Forward was used to those where only DS-Backwards was used? Or, are you comparing studies where the same kind of task(s) was used, e.g. both DS-Forward?

 

[WisNeuro]

Please, correct me if I'm wrong. Are you comparing studies where only DS-Forward was used to those where only DS-Backwards was used? Or, are you comparing studies where the same kind of task(s) was used, e.g. both DS-Forward?

Based on previous tables, looks like DSF was one set of analyses, DSB was another set, and a 2-step n-back test was another.

Edit - Looks like they also used LNS and Tower of London as a WM task. I could make an argument for most of those, but the ToL is clearly an EF task.

 

[psych.meout]

Based on previous tables, looks like DSF was one set of analyses, DSB was another set, and a 2-step n-back test was another.

Ah, ok, makes more sense.

Edit - Looks like they also used LNS and Tower of London as a WM task. I could make an argument for most of those, but the ToL is clearly an EF task.

Really? I know working memory is a core facet of executive functioning and surely it is involved in the Tower of London test and letter-number sequencing task, but the latter two clearly aren't tapping into exactly all the same functions as digit span tasks. I guess I don't have all the required info or I'm just not getting why you would include them and overly complicate and possibly confound a simpler, yet more sound, analysis of working memory just using digit span.