Have you worked with population-level data before? Reporting every single analysis performed is infeasible and distracting. If I have no idea what causes increased obesity rates, I'm going to have to go looking for variables. I don't have any preconceived notion of a variable in my head. I know that this population has increased obesity but I have no idea why so I'm going to go into this in an unbiased manner and see what variables might be associated with it.
Right, this is called exploratory data analysis, but it doesn't preclude you from stating what you did, especially when you're presenting p-values. Understanding what a p-value is and isn't, and the assumptions in calculating a p-value should make this apparent. It's a sorry excuse to say clear and complete reporting is "distracting"-- I've heard the same argument numerous times from respected researchers when they're presented with important aspects such as model assumption verification and missing data methodology.
In some aspects, this is better than going into this with a specific hypothesis because then you run the risk of having coincidental or confounded findings. If I look at all the variables in a population that are measured, I would probably find a few variables associated with obesity, e.g. hypertension, soda company revenue, etc. Hell, I might even report a p value for these comparisons.
So you're making the argument that blindly swinging in the dark until you hit something is
less likely to uncover a coincidental finding than trying to use current knowledge to formulate a specific hypothesis or set of hypotheses? That's an odd argument to make. Again, I'm not totally against data mining or exploration, but I'm against people reporting what they deem relevant out of the pile of analyses they ran just to present "new association X" with a p-value. You don't need p-values (or CIs) to show association that might be worth further investigation, especially in a hypothesis generating or exploratory study. An understanding of many assumptions used in the methods makes this clear that observational p-values don't mean
nearly as much as from randomized experiments, but you can still come up with new ideas without using inferential statistics such as p-values and CIs.
The key difference is that I know what kind of study this is - to discover a model that can be used to understand obesity. Others who don't have my training probably wouldn't understand and ding me for running regression on so many variables.
You may know for your paper, but others reading might not. Good science is clearly detailed so the reader knows what was done and how. Earlier you say you're just shaking the tree to generate ideas, now you're using that 1 out of 18 models to understand the disease.
Since you're bringing this to the table, what is your training? Genuinely curious because I can think of quite a few people who have at least bachelors but up to a PhD in Statistics who have dinged people for trying not to report the random things they did, then finding something by mining, and then trying to draw far stronger conclusions than are warranted by the study. It's perfectly okay to data mine, but you have to report it as such, and you definitely have to limit your conclusions because of the nature of what was done.
I think you would be hard-pressed to discover a more effective way of answering such research questions. In your world, how would you answer this same question? Remember, you have no idea what causes obesity here. Or, if you want a better example, what causes SIDS.
I can think of several ways that are more effective and are more open about what's being done. First and formost, visualizing data and looking at descriptive measures get you pretty far for understanding relationships in the data and this doesn't open you up to risks of Type I or Type II errors. For formal tests, it's real easy to state you're using a lenient alpha such as .15 for variable screening (which is pretty reasonable since you're digging for something and won't be pushing strong conclusions, just generating a future study question)-- problems arise when people reviewing for journals don't know stats and think every significance threshold needs to be .05, so you don't see more judgement when selecting alpha. For your question, I would consider utilizing a LASSO regression, report what I did, and temper the conclusions I try to draw. At the very least, it's not hard to say "14 regression models were fit but only the two with significant omnibus tests are presented." Whatever I end up using though, the important part is clear reporting and appropriate conclusions in the context of the analyses performed. It's coming off that you're arguing against the basic idea of clear and complete reporting.
There's no model from which you can derive testable hypotheses. How would you proceed?
You don't need a model to come up with a hypothesis, you need subject matter expertise, and when you have none for a specific issue, you can bridge concepts that are better described-- this is scientific thinking. You don't need to think to have a computer iterate through tons of models and set aside results from a subset.
There's a reason this whole argument of lowering the significance threshold is even in play, and what you're describing is a large contributor to the underlying issue. It's not the clinical trials with preregistered study protocol and far better detailing of methods as much as it is everything else.
