Stats Question

This forum made possible through the generous support of SDN members, donors, and sponsors. Thank you.

Marissa4usa

Full Member
15+ Year Member
Joined
Sep 5, 2007
Messages
523
Reaction score
56
I am trying to figure out the appropriate statistical procedures to analyze data collected at a clinic that after years of providing treatment-as-usual made some drastic programmatic changes, and now wants to compare how well patients do under this new treatment approach. This is not a randomized clinical control - essentially, every patient will be his/her own control in this procedure, and we're comparing each person's slope before and after implementing their new approach to clinical care.

Somewhat surprisingly, I'm struggling to find information on how to do these kind of analyses. It looks like time-series analyses would be the right approach, but I am only superficially familiar with those types of analyses.

Any suggestions are welcome!
 
I am trying to figure out the appropriate statistical procedures to analyze data collected at a clinic that after years of providing treatment-as-usual made some drastic programmatic changes, and now wants to compare how well patients do under this new treatment approach. This is not a randomized clinical control - essentially, every patient will be his/her own control in this procedure, and we're comparing each person's slope before and after implementing their new approach to clinical care.

Somewhat surprisingly, I'm struggling to find information on how to do these kind of analyses. It looks like time-series analyses would be the right approach, but I am only superficially familiar with those types of analyses.

Any suggestions are welcome!

Wrote something else before I saw the part about each serving as their own control. Piecewise is the way to go. I can think of some random geeky things you can do with timeseries in this scenario, but it would be epic overkill if this is just internal data and not something you want to publish in a top journal.

Don't know the purpose of the analyses or who will be looking at them, but don't discount the value of just plotting and overlaying the before/after curves. If this is a program eval, its all folks are likely to understand anyways (depending on the setting of course).
 
Last edited:
Hi Marissa,
I'd look into a piecewise latent growth model, centering each individuals time at the transition point. Feel free to DM me if you like.

Thanks, I will message you!

Wrote something else before I saw the part about each serving as their own control. Piecewise is the way to go. I can think of some random geeky things you can do with timeseries in this scenario, but it would be epic overkill if this is just internal data and not something you want to publish in a top journal.

Don't know the purpose of the analyses or who will be looking at them, but don't discount the value of just plotting and overlaying the before/after curves. If this is a program eval, its all folks are likely to understand anyways (depending on the setting of course).

The intent is to ultimately publish these (as well as a program evaluation). They hired me as their research postdoc to make sense of their data. Plotting sounds would be a good first step but probably not suffice, but I'd love to hear what time series analyses can do.
 
Last edited:
I was half-joking about time series, but PM me some details if you want.

It is primarily used when you have a LOT of repeated measures and things change rapidly. I dont mean weekly ratings for 12 weeks....more like stock prices sampled every 10 minutes for 30 years. A more clinical example would be EEG or psychophya data where you might be sampling thousands of times a second and need to decompose the signal to search for patterns. It is (generally) more focused on looking for cyclical changes and trends than what you are thinking about (which sounds like a 1x event?).

I have toyed with the idea of using some principles from it to see if I could decompose rapid fluctuations (e.g. acute stressors) from more chronic changes (e.g. full blown lapse/relapse). I never had the right data for it and haven't even fully thought through the actual implementation.

This would be wildly inappropriate for a program eval and would likely not directly answer the questions they are asking. Has been excellent mental fodder during my commutes though...
 
I am trying to figure out the appropriate statistical procedures to analyze data....

... essentially, every patient will be his/her own control in this procedure, and we're comparing each person's slope before and after implementing their new approach to clinical care.

You've been given some solid advice regarding statistics for analyzing if there has been a significant change in the slope pre- versus post-change. Depending on the type of data and how it is measured things like time-series analysis, statistical process control, etc. MAY be some options. Statistical analysis of such repeated measures from the same subject can be problematic (e.g. due to issues with autocorrelation). Also, assuming multiple clients, you face the issue of doing individual analyses on each client (time consuming) or aggregating the group data (blechhh!, speaking as a behavior analyst). Note that these methods, best case scenario, would only demonstrate that there was a significant change in the slope. In the absence of any experimental control, you should not and cannot conclude that the change in client data is functionally related to any specific changes that you made in your way of doing things.

There are, however, several research design methods that don't involve stats where you use individual subjects as their own controls (I currently teach a whole graduate class on these "single case design" methods). If you've already implemented the changes across the board with all clients, then you really are limited to only one such experimental design at this point- a reversal/withdrawal design where you would go back to the conditions in the clinic before you made the changes until the data stabilizes, and the re-implement the changes. If the data returns to baseline levels when you revert to pre-change conditions and goes back to post change levels after you re-implement the change, you can conclude that there is a functional relationship between the changes you made and the change in the client's performance. There are ethical issues with using such a design in an applied setting, including doing something that would lead to client regression. You could potentially do this with just a few of the clients. Otherwise, you are pretty limited in what you could use, post-hoc, for experimental designs. If you did not implement the changes for each client at the same time, you might have accidentally stumbled upon a multiple baseline across subjects design. Going forward, if the agency is willing to use the older pre-change procedures for awhile with some new clients, you might be able to do non-concurrent multiple baseline (but with no overlap between the baselines in the different groups, it would be a pretty weak design).

TL/DR- there may be some ways to show that the data changed significantly after the change in procedures. However, this would not show that the change in the procedures caused the change in the data. There are some single-case design options for showing causation, but having already made the changes in procedures, you're kind of limited.
 
Top