My previous job was at a startup doing BMI, for research. For the first time I had the chance to work with expensive neural signal measurement tools (mainly EEG for us, but some teams used fMRI). and quickly did I learn how absolute horrible the signal to noise ratio (SNR) was in this field.
And how it was almost impossible to reproduce many published and well cited result. It was both exciting and jarring to talk with the neuroscientist, because they ofc knew about this and knew how to read the papers but the one doing more funding/business side ofc didn't really spend much time putting emphasis on that.
One of the team presented a accepted paper that basically used Deep Learning (Attention) to predict images that a person was thinking of, from the fMRI signals. When I asked "but DL is proven to be able to find pattern even in random noise, so how can you be sure this is not just overfitting to artefact?" and there wasn't really any answer to that (or rather the publication didn't take that in to account, although that can be experimentally determined). Still, a month later I saw tech explore or some tech news writing an article about it, something like "AI can now read your brain" and the 1984 implications yada yada.
So this is indeed something probably most practitioners, masters and PhD, realize relatively early.
So now that someone says "you know mindfulness is proven to change your brainwaves?" I always add my story "yes, but the study was done with EEG, so I don't trust the scientific backing of it" (but anecdotally, it helps me)
There are lots of reliable science done using EEG and fMRI; I believe you learned the wrong lesson here. The important thing is to treat motion and physiological sources of noise as a first-order problem that must be taken very seriously and requires strict data quality inclusion criterion. As far as deep learning in fMRI/EEG, your response about overfitting is too sweepingly broad to apply to the entire field.
To put it succinctly, I think you have overfit your conclusions on the amount of data you have seen
I would argue in fact almost all fMRI research is unreliable, and formally so (test-retest reliabilities are in fact quite miserable: see my post below).
EDIT: The reason being, with reliabilities as bad as these, it is obvious almost all fMRI studies are massively underpowered, and you really need to have hundreds or even up to a thousand participants to detect effects with any statistical reliability. Very few fMRI studies ever have even close to these numbers (https://www.nature.com/articles/s42003-018-0073-z).
That depends immensely on the type of effect you're looking for.
Within-subject effects (this happens when one does A, but not when doing B) can be fine with small sample sizes, especially if you can repeat variations on A and B many times. This is pretty common in task-based fMRI. Indeed, I'm not sure why you need >2 participants expect to show that the principle is relatively generalizable.
Between-subject comparisons (type A people have this feature, type B people don't) are the problem because people differ in lots of ways and each contributes one measurement, so you have no real way to control for all that extra variation.
Precisely, and agreed 100%. We need far more within-subject designs.
You would still in general need many subjects to show the same basic within-subject patterns if you want to claim the pattern is "generalizable", in the sense of "may generalize to most people", but, precisely depending on what you are looking at here, and the strength of the effect, of course you may not need nearly as much participants as in strictly between-subject designs.
With the low test-retest reliability of task fMRI, in general, even in adults, this also means that strictly one-off within-subject designs are also not enough, for certain claims. One sort of has to demonstrate that even the within-subject effect is stable too. This may or may not be plausible for certain things, but it really needs to be considered more regularly and explicitly.
Yes on many of those fronts, although not all those papers support your conclusion. The field did/does too often use tasks with to few trials, with to few participants. That always frustrated me as my advisor rightly insisted we collect hundreds of participants for each study, while others would collect 20 and publish 10x faster than us.
Yes, well "almost all" is vague and needs to be qualified. Sample sizes have improved over the past decade for sure. I'm not sure if they have grown on median meaningfully, because there are still way too many low-N studies, but you do see studies now that are at least plausibly "large enough" more frequently. More open data has also helped here.
EDIT: And kudos to you and your advisor here.
EDIT2: I will also say that a lot of the research on fMRI methods is very solid and often quite reproducible. I.e. papers that pioneer new analytic methods and/or investigate pipelines and such. There is definitely a lot of fMRI research telling us a lot of interesting and likely reliable things about fMRI, but there is very little fMRI research that is telling us anything reliably generalizable about people or cognition.
The small sample sizes is rational response from scientists in the face of a) funding levels and b) unreasonable expectations from hiring/promotion committees.
cog neuro labs need to start organizing their research programs more like giant physics projects. Lots of PIs pooling funding and resources together into one big experiment rather than lots of little underpowered independent labs. But it’s difficult to set up a more institutional structure like this unless there’s a big shift in how we measure career advancement/success.
+1 to pooling funding and resources. This is desperately needed in fMRI (although site and other demographic / cultural effects make this much harder than in physics, I suspect).
I'm not an expert, but my hunch would be that a similar Big(ger) Science approach is also needed in areas like nutrition and (non-neurological) experimental psychology where (apparently) often group sizes are just too small. There are obvious drawbacks to having the choice of experiments controlled by consensus and bureaucracy, but if the experiments are otherwise not worthwhile what else is there to do?
I think the problems in nutrition are far, far deeper (we cannot properly control diet in most cases, and certainly not over long timeframes; we cannot track enough people long enough to measure most effects; we cannot trust the measurement i.e. self-report of what is consumed; industry biases are extremely strong; most nutrition effects are likely small and weak and/or interact strongly with genetics, making the sample size requirements larger still).
I'm not sure what you mean by "experimental psychology" though. There are areas like psychophysics that are arguably experimental and have robust findings, and there are some decent-ish studies in clinical psychology too. Here the group sizes are probably actually mostly not too bad.
Areas like social psychology have serious sample size problems, so might benefit, but this field also has serious measurement and reproducibility problems, weak experimental designs, and particularly strong ideological bias among the researchers. I'm not sure larger sample sizes would fix much of the research here.
> Areas like social psychology have serious sample size problems, so might benefit, but this field also has serious measurement and reproducibility problems, weak experimental designs, and particularly strong ideological bias among the researchers. I'm not sure larger sample sizes would fix much of the research here.
I can believe it; but a change doesn't have to be sufficient to be ncessary.
But none of this (signal/noise ratio, etc) is related to the topic of the article, which claims that even with good signal, blood flow is not useful to determine brain activity.
I'm not sure I understand. Wouldn't any prediction result above statistical random (in the image mind reading study) be significant? If the study was performed correctly I don't really need to know much about fMRI to tell whether it's an interesting result or not.
The difference is that EEG can be used usefully in e.g. biofeedback training and the study of sleep phases, so there is in fact enough signal here for it to be broadly useful in some simple cases. It is not clear fMRI has enough signal for anything even as simple as these things though.
Yes, there are a few medical cases where fMRI makes good simple basic sense, and TBI/Concussion sounds immediately like one of those to me. I seem also to recall them being useful in some cases prior to brain surgeries and the like.
This all makes sense because fMRI tracks metabolic activity via oxygenation changes, which is much more clearly and plausibly related to tissue health and recovery. In these cases, it is also most likely being used within-subject (i.e. longitudinally) to make comparisons to baselines, rather than in an attempt to make speculative inferences about the mind using groups of people, and likely is a simple comparison to baseline rather than bespoke statistical analyses relying on questionable assumptions about the BOLD response being related to overly-specific kinds of neural activity.
fMRI can track oxygenation changes, and indirectly where the blood flow is, or isn't, and perhaps some ideas on where to get it.
All to say, this application might not fall in the 40%.
I just find articles like these can't help but feel like they have an agenda to undermine something instead of simply acknowledge the kinds of things it is and isn't working for.
There's no doubt these researchers have found something, but the need for sensationalistic headlines is well known in academia as well.
Sometimes it's noticeable where the research is specific in scope, but the findings are more general and broad.
> fMRI's are being used in TBI/Concussion recovery
Interesting. Do you happen to have any more information on this topic? I ask because I was under the impression that concussions are a functional/metabolic injury and not a structural injury, therefore, concussions are not visible on any type of fMRI, CT Scan, etc.. Though, I haven't looked into this topic for almost half a decade, so I imagine things have likely progressed.
Concussions seem to be pretty physiological - first they're a brain bleed, and blood doesn't seem to pump the same as it did before the concussion... resulting in different symptoms.
That might be what you're referring to as functional?
Metabolically, or otherwise, if the brain can't operate, other things in the body such as metabolism would be impacted for sure when it can't oversee and run as it normally can?
While I'm not sure if a concussion directly is visible or not (some have sizeable enough brain bleeds that can be visible), concussions to the extent that they are a change in blood circulation changes and issues, can be visualized on fMRI, etc, where it's not regular, those areas suffer in a brain.
Things luckily have progressed and quite exciting.
Out of convenience, I'll share one I know about (no affiliation) that lay out their therapies and the science behind it as well.
Effectively (I hope I'm getting this accurately) it seems the blood vessels in the brain also have signalling from the blood and oxygen that gets affected which affects things downstream from there.
These guys do an fMRI baseline, have you jump on a bike, fMRI again, see what's not getting blood, and then give you exercises and activites for those regions of the brain. It's pretty interesting.
90% of papers I read in computer science / computer security speak of software written or AI models they trained that are nowhere to be found. Not on git nor via email to the authors.
There's fancier ML studies on EEG signal but probably not consistent enough for clinical work. For now, the one thing EEG can reliably tell is if you're having a seizure or not, if you're delirious (or in a coma) or not, or if you're asleep.
> When I asked "but DL is proven to be able to find pattern even in random noise, so how can you be sure this is not just overfitting to artefact?"
So here you say quite a mouthful. If you train it on a pattern it'll see that pattern everywhere - think about the early "Deep Dream" trippy-dogs-pictures nonsense that was pervasive about eight or nine years ago.
I repaired a couple of cameras for someone who was working with a large university hospital about 15 years ago, where they were using admittedly 2010s-era "Deep Learning" to analyse biopsy scans for signs of cancer. It worked brilliantly, at least with the training materials, incredible hit rate, not too terrible false positive rate (no biggie, you're just trying to decide if you want to investigate further), really low false negative rate (if there was cancer it would spot it, for sure, and you don't want to miss that).
But in real-world patient data it went completely mental. The sample data was real-world patient data, too, but on "uncontrolled" patients, it was detecting cancer all over the place. It also detected cancer in pictures of the Oncology department lino floor, it detected cancer in a picture of a guy's ID badge, it detected cancer in a closeup of my car tyre, and it detected cancer in a photo of a grey overcast sky.
Aw no. Now what?
Well, that's why I looked at the camera for them. They'd photographed the biopsies with one camera on site, from "real patients", but a lot of the "clear" biopsies were from other sites.
You're ahead of me now, aren't you?
The "Deep Learning" system had in fact trained itself on a speck of shit on the sensor of one of the cameras, the one used for most of the "has cancer" biopsies and most of the "real patient under test" biopsies. If that little blob of about a dozen slightly darker pixels was present, then it must be cancer because that's what the grown-ups told it. The actual picture content was largely irrelevant because the blob was consistent across all of them.
I'm not too keen on AI in healthcare, not as a definitive "go/no-go" test thing.
I remember reading a paper back in grad school where the researchers put a dead salmon in the magnet and got statistically significant brain activity readings using whatever the analysis method à la mode was. It felt like a great candidate for the Ig Nobel awards.
That was our paper! We showed that you can get false positives (significant brain activity in this case) if fMRI if you don't use the proper statistical corrections. We did win an Ig Nobel for that work in 2012 - it was a ton of fun.
I was a grad student at UCSD when Ed Vul published Voodoo Correlations in Social Neuroscience [1], which stoked a severe backlash from the fMRI syndicate resulting in a title change to Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition [2]. There is a lot of interesting commentary around this article (e.g., “Voodoo” Science in Neuroimaging: How a Controversy Transformed into a Crisis [3]). To me it was fascinating to watch Vul (an incredibly rare talent, perhaps a genius), take on an entire field during his 1st year as assistant professor.
I'll get raked for this, but as someone in the field, I can say with high confidence that the majority of comments in this thread are not from imaging experts, and mostly (mis)informed by popular science articles. I do not have the time to properly respond to each issue I see. The literature is out there in any case.
> I do not have the time to properly respond to each issue I see. The literature is out there in any case.
I think your expertise would be very welcome, but this comment is entirely unhelpful as-is. Saying there are bad comments in this thread and also that there is good literature out there without providing any specifics at all is just noise.
You don't have to respond to every comment you see to contribute to the discussion. At minimum, could you provide a hint for some literature you suggest reading?
I have also published and worked for some years in this field, if that helps.
The literature is huge, and my bias is that I believe most of the only really good fMRI research is methodological research (i.e. about what fMRI actually means, and how to reliably analyze it). Many of the links I've provided here speak to this.
I don't think there is much reliable fMRI research that tells us anything about people, emotions, or cognition, beyond confirming some likely localization of function to the sensory and motor cortices, and some stuff about the Default Mode Network(s) that is of unclear importance.
A lot of the more reliable stuff involves the Human Connectome Project (HCP) fMRI data, since this was done very carefully with a lot of participants, if you want a place to start for actual human-relevant findings. But the field is still really young.
> Saying there are bad comments in this thread and also that there is good literature out there without providing any specifics at all is just noise.
Nah, it's not noise. It's a useful reminder not to take any comments too seriously and that this topic is far outside the average commenter's expertise.
I'll co-sign SubiculumCode's comment -- there's a lot of yelling about how bad fMRI is generally, which is not particularly fair to the fMRI research (or at least the better parts of it) or related to the argument.
The BOLD signal, the thing measured by fMRI, is a proxy for actual brain activity. The logic is that neural firing requires a lot of energy and so active neurons will being using more oxygen for their metabolism, and this oxygen comes from the blood. Thus, if you measure local changes in the oxygenation of blood, you'll know something about how active nearby neurons are. However, it's an indirect and complicated relationship. The blood flow to an area can itself change, or cells could extract more or less oxygen from the blood--the system itself is usually not running at its limits.
Direct measurements from animals, where you can measure (and manipulate) brain activity while measuring BOLD, have shown how complicated this is. Nikos Logathetis and Ralph Freeman's groups, among many others did a lot of work on this, especially c. 2000-2010. If you're interested, you could check out this news and views on Logathetis's group's 2001 Nature paper [1]. One of the conclusions of their work is that BOLD is influenced by a lot of things but largely measure the inputs to an area and the synchrony within it, rather than just the average firing rate.
In this paper, the researchers adjust the MRI sequences to compare blood oxygenation, oxygen usage, and blood flow and find that these are not perfectly related. This is a nice demonstration, but not a totally unexpected finding either. The argument in the paper is also not "abandon fMRI" but rather that you need to measure and interpret these things carefully.
In short, the whole area of neurovascular coupling is hard--it includes complicated physics (to make measurements), tricky chemistry, and messy biology, all in a system full of complicated dynamics and feedback.
I hide any thread that deals with education, education funding, or teaching in general for that specific reason. It really saddens me to see that this place is full of so much misinformation and anecdotes made into data (and usually with much more self-confidence than other forums, which is interesting to me).
It's why I generally only ask questions, or ask for clarification instead of directly challenging something I think might be wrong now in threads that aren't related to something I have deeeeep personal knowledge of. I know when I'm out of my area, and don't want to add to the ignorance.
A great habit - especially when your question is an irresistible, easily-addressed homerun to a domain expert wandering through the thread looking for an entry-point.
Challenging something with a question about it, is not adding to ignorance - if a statement/study/fact/belief can't hold to up to questions, from actual opposing critics, what's the point of that position existing?
Being all "PC" and "nice" about stuff that is what it is, or isn't -- THAT adds to ignorance.
I guess maybe I didn't phrase that correctly - I ask challenging questions, but don't state the things I "know" without clarification first. I meant that I don't just pop off with "yeah, but in reality it's x, y, z" because I know that I'm probably ignorant of facts. I'll ask about x, y, or z first.
I'm not a specialist by any means, although I am a patient of an fMRI. One thing I will note is that in the eventual, resultant paperwork from the broad array of tests I had, the fMRI was not noted whatsoever, neither was it discussed with me by any of the numerous neurologists or surgeons involved in my case. I was quite curious as to why it was performed at all, but presumably it was some formality to check a box.
It makes sense they wouldn't look at it, there are very few, if any, well-validated clinical uses for it. However, they might have taken it as a baseline for later comparison, and it is definitely plausible when surgery is involved that visible abnormalities could be seen in fMRI that might not show up in MRI, either now or later.
I don't think there would be much clear guidance for them on how to interpret any such fMRI abnormality on its own, but it might still be something useful for further investigations, and this might especially be the case for surgery. It might also have been done as part of research, if you consented to anything like that?
There are literally less than 20 top level comments and this one is (at least for me) the 2nd or 3rd.
Instead of a nothingburger, you could have used your academic prowess to break down the top 1/2 misconceptions with expertise.
You might not have time to respond to all the comments but a couple of clarifications could have helped anyone else who doesn't comment without experience.
Just saying that next time you can be the change you want to see in HN instead of wasting text telling us how ignorant we are.
This study is validating a commonplace fMRI measure (change in blood-oxygenation-level-dependent or BOLD signal) by comparing it with a different MRI technique, one that uses a multiparametric quantitative BOLD model, a different model for BOLD derived from two separate MRI scans which measure two different kinds of signal (transverse relaxation rates), and then multiply/divide by a bunch of constants to get at a value.
I'm a software engineer in this field, and this is my layman-learns-a-bit-of-shop-talk understanding of it. Both of these techniques involve multiple layers of statistical assumptions, and multiple steps of "analysing" data, which in itself involves implicit assumptions, rules of thumb and other steps that have never sat well with me. A very basic example of this kind of multi-step data massaging is "does this signal look a bit rough? No worries, let's Gaussian-filter it".
A lot of my skepticism is due to ignorance, no doubt, and I'd probably be braver in making general claims from the image I get in the end if I was more educated in the actual biophysics of it. But my main point is that it is not at all obvious that you can simply claim "signal B shows that signal A doesn't correspond to actual brain activity", when it is quite arguable whether signal B really does measure the ground truth, or whether it is simply prone to different modelling errors.
In the paper itself, the authors say that it is limited by methodology, but because they don't have the device to get an independent measure of brain activation, they use quantitative MRI. They also say it's because of radiation exposure and blah blah, but the real reason is their uni can't afford a PET scanner for them to use.
"The gold standard for CBF and CMRO2 measurements is 15O PET; but this technique requires an on-site cyclotron, a sophisticated imaging setup and substantial experience in handling three different radiotracers (CBF, 15O-water; CBV, 15O-CO; OEF, 15O-gas) of short half-lives8,35. Furthermore, this invasive method poses certain risks to participants owing to the exposure to radioactivity and arterial sampling."
This is why I love this site. You get input from so many specialized folks! I appreciate you contributing your expertise and I also appreciate you calling out the limits to that knowledge.
Two points I'm hoping you can help clarify:
> Researchers ... found that an increased fMRI signal is associated with reduced brain activity in around 40 percent of cases.
So it's not just that they found it was uncorrelated, they found it was anticorrelated in 40% of cases?
And you are suggesting that conclusion suffers from the same potential issues as these fMRI studies in general?
Like you mention, it seems to me if we wanted to really validate the model, we'd have to run the same experiment with two, three, or maybe even more different modalities (fMRI, PET with different tracers, etc).
If you have a PET/MR system [0], you can probably do this "gold standard" comparison, and I know that one is used for research studies. I think you can piggy-back off a different study's healthy controls to write a paper like this, if that study already uses PET/MR and if adding an oxygen metabolite scan isn't a big problem. But that's speaking as someone who does not design experiments.
This study was really highlighting a statistical issue which would occur with any imaging technique with noise (which is unavoidable). If you measure enough things, you'll inevitably find some false positives. The solution is to use procedures such as Bonferroni and FDR to correct for the multiple tests, now a standard part of such imaging experiments. It's a valid critique, but it's worth highlighting that it's not specific to fMRI or evidence of shaky science unless you skip those steps (other separate factors may indicate shakiness though).
When we published the salmon paper, approximately 25-35% of published fMRI results used uncorrected statistics. For myself and my co-authors, this was evidence of shaky science. The reader of a research paper could not say with certainty which results were legitimate and which might be false positives.
Hey, I know you got a lot of flack for the article. So, I just wanted to thank you for having the courage to publish it anyways and go through all of that for all of us.
I go back to the study frequently when looking at MRI studies, and it always holds up. It always reminds me to be careful with these things and to try to have other be careful with their results too. Though to me it's a bit of a lampooning, surprisingly it has been the best reminder for me to be more careful with my work.
So thank you for putting yourself through all that. To me, it was worth it.
> a statistical issue which would occur with any imaging technique
I sounds like it goes beyond that: If a certain mistake ruins outcomes, and a lot of people are ruining outcomes and not noticing, then there's some much bigger systematic problem going on.
While I would agree that the prevalence of the problem has been minimized in fMRI during the last 15 years, I disagree that our critique does not hold up. The root of our concern was that proper statistical correction(s) need to be completed in order for research results to be interpretable. I am totally biased, but I think that remains worthwhile.
In related news: ironically, Psychedelics disrupt normal link between brain’s neuronal activity and blood flow - thus casting some doubt on findings that under psychedelics more of the brain is connected (since fMRI showed elevated blood flow, suggesting higher brain activity).
As a caveman pondering "Stoned Ape Theory" during the rise of MRI in the 80s, having done light reading of Huxley, McKenna et. al, the claim that vascular variations were so tied to thought patterns in a purely calm and cognitive activity was fascinating. To see the brain of someone as they went through a deck of cards and paused to look at each... astounding! But frustrating also. My first question always was, was the person's hands busy going through the deck and holding up the cards, focusing on them... or were they merely shown the cards sitting still? It seemed the popsci articles often glossed over that information, and any simple "control for coordinated body movement" played second fiddle to the novelty of it all. Then I worked in a club where I was often surrounded by tripping people. I'd fetch them glasses of water and they would always drink. Do you know you can smell them, they smell like fear? The experience has every sweat gland working overtime. When I learned that I greeted this "tripping people MRIs light up indicating enhanced brain connectivity" with a grain of salt. I would not be the least bit surprised if the sweat gland thing also has the brain's vascular system in overdrive.
This isn’t entirely news to people in the field doing research, but it’s important information to keep in mind when anyone starts pushing fMRI (or SPECT) scans into popular media discussions about neurology or psychiatry.
There have been some high profile influencer doctors pushing brain imaging scans as diagnostic tools for years. Dr. Amen is one of the worst offenders with his clinics that charge thousands of dollars for SPECT scans (not the same as the fMRI in this paper but with similar interpretation issues) on patients. Insurance won’t cover them because there’s no scientific basis for using them in diagnosing or treating ADHD or chronic pain, but his clinics will push them on patients. Seeing an image of their brain with some colors overlayed and having someone confidently read it like tea leaves is highly convincing to people who want answers. Dr. Amen has made the rounds on Dr. Phil and other outlets, as well as amassing millions of followers on social media.
Dr. Mike, a rare YouTube doctor who is not peddling supplements and wares, and thus seems to be at the forefront of medical critical thinking on the platform, interviewed Dr. Amen recently[0]. I haven't finished the interview yet, but having watched some others, generally the approach is to let the interviewee make their grandiose claims, agree with whatever vague generalities and truisms they use in their rhetoric (yes it's true, doctors don't spend enough time explaining things to patients!), and then lay into them on the actual science and evidence.
Dr. Mike did an incredible job in that interview. He gave Dr. Amen all the rope to hang himself with his own words. When you're hawking a diagnosis method and you're not interested in building up the foundation of evidence for it by doing a double blinded, randomized controlled study. And that the results of said study would change how your treating patients it's pretty clear who the snake oil salesman is
I also thought the rest of the interview was really worthwhile - they talked a lot about real problems in the medical industry from different perspectives. What a great and critical discussion from Dr. Mike. If Amen had conceded the point they could have moved on. There could be real findings to be had there, and some may even match his conclusions, but many likely will not, and the whole thing could also be pure fiction. We should want better answers to these questions. It's unfortunate to watch someone as seemingly intelligent and well-informed as Amen come across as shilling snake oil, and/or just being hung up on his ego, at the end of it all. Scientific literacy is so critical, because it's easy to cloak pseudoscience behind high-tech smokescreens.
Unfortunately I worry about the rebound effect, where even though the entire interview was debunking his claims this could still on average increase amen’s popularity.
I'm no expert in medicine, but I watched that entire video and your analogy about performance and rope doesn't fit well with how it came across to me.
I actually thought the interviewer was a little disingenuous. He said things like "We're on the same team" and "I'm not trying to trap you", then proceeded to lob his guest with criticisms from the other team and questions aimed to maneuver him into a contradiction. There's nothing inherently wrong with that, but if you're going to do it, be forthright you're engaging in a debate.
Earlier in the interview he could have put his cards on the table and plainly stated "Myself and others in the medical community are skeptical of the efficacy of imaging on outcomes, and a rigorous, double-blind study would lend dramatic support for us to adopt what you're touting."
Then they could have had the conversation he was clearly after, focused on that issue.
Instead it felt like I was watching for ages as he took a winding route to get there, then the interview cut off abruptly when they finally really did.
The overlays applied in editing while helpful and fair in some cases, at other times came across as one-sided. It's a shame we can't see a follow-up where the interviewee has an opportunity to respond (or squirm) in light of them.
For the record I would very much love to see additional research and gold-standard, double-blind studies. In the meantime I'll treat this as "Hey, we've got this interesting thing we can measure, we're seeing some good results in our practice" without over-emphasizing the confidence in this one diagnostic.
I did find the bit interesting about how having a gauge you can viscerally see impacted patients' engagement in care. Both agreed on the potential usefulness of that aspect, and conceded the difference in profiles between patients coming to Dr. Amen vs. ordinary front-line family physicians.
Back in 2009 I remember reading about how dead salmon apparently turns up brain activity in fMRI without proper statistical methods. fMRI studies are something frequently invoked unscientifically and out of context.
I think technically there's some statistical correction you apply to the voxels to avoid this. But yea...most hypotheses from fMRI are considered hypotheses until there's some other modality, i.e. electrical recordings, etc that confirm it.
i.e. the well regarded studies, i.e. Kanwisher and the visual processing areas, have follow up studies on primates and surgical volunteers w/ actual electrical activity correlating w/ visual stimuli etc
>> Seeing an image of their brain with some colors overlayed ... is highly convincing
Indeed, there's been quite a few studies [1] that find just including any old image of a brain with stuff highlighted will cause a paper to be perceived as more scientifically credible.
I saw a clinical report of his on a patient, he puts a graphic in their report of their "brain scan" but it's basically a vector graphic of the brain w/ a multicolor MS Paint gradient...
Pop science guru-ing is a giant flashing red sign for me. I am never even a little surprised when the latest “sense maker” or pop science guru comes out as a complete loon or is consumed by some kind of scandal.
Influencers in general are always suspect. The things that get you an audience fast are trolling or tabloid-ish tactics like conspiracism.
There are good ones but you have to be discerning.
I might be oversimplifying, but isn't a lot of our neurological understanding about ADHD based on "fMRI shows decreased activity in the frontal cortex"? Or for that matter, our neurological understanding of a lot of mental health conditions.
I know the actual diagnosis is several times more layered than this attempt at an explanation, but I always felt that trying to explain the brain by peering at it from outwards is like trying to debug code by looking at a motherboard through a bad microscope.
I do not think there is much neurological understanding about ADHD at all from current fMRI research, there are far too many quality and reliability issues here, not just on the fMRI end or limited amount of data overall, but in the measurement and diagnosis of ADHD itself (i.e. ADHD subtypes, and of course ADHD is a complicated diagnosis with many components manifesting to different degrees in different individuals, which makes it very hard to cleanly link to messy fMRI signals).
Or, as I have commented elsewhere here, the idea that statements like "fMRI shows decreased activity" are ever valid is just fundamentally suspect (lower BOLD response could mean less inhibition or less excitation, and this is a rather crucial difference that fMRI simply can't distinguish). EDIT: Or to be more precise: it may well be that fMRI research suggests less metabolic activity in certain regions, but this could mean the region is actually firing more than normal, less than normal, is more efficient than normal, etc., and interpreting anything about what is functioning differently in ADHD, given this uncertainty, is what is going to be suspect.
Yes this is true, but we actually have a lot more data to back this on than exclusively fMRI analysis - for example the ADHD medication guanfacine works only because the alpha 2 receptors happen to be wired differently in the prefrontal cortex than it is in other areas of the brain (a2 is inhibitory for most brain regions, but in the PFC they're positioned to amplify connections between neurons) , so by stimulating alpha 2 we allow for a more “top down” control from the prefrontal cortex than we do without, which improves executive function.
So that is one extremely robust way to understand neurological conditions like ADHD or Parkinson’s
As someone who used to work at the Cognitive Neurophysiology Lab in the Scripts Institute-- doing some work on functional brain image-- I can confirm this was not news even thirty years ago. I guess this is trying to make some point to lay people?
Are there proposed reasons for increased blood flow to brain regions other than neural activity? Are neurons flushing waste products or something when less active?
fMRI has been abused by a lot of researchers, doctors, and authors over the years even though experts in the field knew the reality. It’s worth repeating the challenges of interpreting fMRI data to a wider audience.
The way I understood it is that while individual fMRI studies can be amazing, it is borderline impossible to compare them when made using different people or even different MRI machines. So reproducibility is a big issue, even though the tech itself is extremely promising.
The article is pointing out that one of the base assumptions behind fMRI, that increased blood flow (which is what the machine can image) is strongly correlated to increased brain activity (which is what you want to measure) is not true in many situations. This means that the whole approach is suspect if you can't tell which situation you're in.
fMRI ususally measures BOLD, changes in blood oxygenation (well, deoxygenation). The point of the paper is that you can get relative changes like that in lots of ways: you could have more or less blood, or take out more/less oxygen from the same blood.
These can be measured themselves separately (that's exactly what they did here!) and if there's a spatial component, which the figures sort of suggest, you can also look at what a particular spot tends to do. It may also be interesting/important to understand why different parts of the brain seem to use different strategies to meet that demand.
This isn't really true. The issue is that when you combine data across multiple MRI scanners (sites), you need to account for random effects (e.g. site specific means and variances)...see solutions like COMBAT. Also if they have different equipment versions/manufacturers those scanners can have different SNR profiles. The other issue is that there are many processing with many ways to perform those steps. In general, researchers don't process in multiple ways and choose the way that gives them the result they want or anything nefarious like that, but it does make comparisons difficult since the effects of different preprocessing variations can be significant. To defend against this, many peer reviewers, like myself, request researchers perform the preprocessing multiple ways to assess how robust the results are to those choices. Another way the field has combatted this issue has been software like fMRIprep.
It is in fact even difficult to compare the same person on the same fMRI machine (and especially in developmental contexts).
Herting, M. M., Gautam, P., Chen, Z., Mezher, A., & Vetter, N. C. (2018). Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies. Developmental Cognitive Neuroscience, 33, 17–26. https://doi.org/10.1016/j.dcn.2017.07.001
I read that paper as suggesting that development, behavior, and fMRI are all hard.
It's not at all clear to me that teenagers' brains OR behaviours should be stable across years, especially when it involves decision-making or emotions. Their Figure 3 shows that sensory experiments are a lot more consistent, which seems reasonable.
The technical challenges (registration, motion, etc) like things that will improve and there are some practical suggestions as well (counterbalancing items, etc).
While I agree I wouldn't expect too much stability in developing brains, unfortunately there are pretty serious stability issues even in non-developing adult brains (quote below from the paper, for anyone who doesn't want to click through).
I agree it makes a lot of sense though the sensory experiments are more consistent, somatosensory and sensorimotor localization results generally seem to the be most consistent fMRI findings. I am not sure registration or motion correction is really going to help much here, I suspect the reality is just that the BOLD response is a lot less longitudinally stable than we thought (brain is changing more often and more quickly than we expected).
Or if we do get better at this, it will be more sophisticated "correction" methods (e.g. deep-learners that can predict typical longitudinal BOLD changes, and those better allow such changes to be "subtracted out", or something like that). But I am skeptical about progress here given the amount of data needed to develop any kind of corrective improvements in cases where there are such low longitudinal reliabilities.
===
> Using ICCs, recent efforts have examined test-retest reliability of task-based fMRI BOLD signal in adults. Bennett and Miller performed a meta-analysis of 13 fMRI studies between 2001 and 2009 that reported ICCs. ICC values ranged from 0.16 to 0.88, with the average reliability being 0.50 across all studies. Others have also suggested a minimal acceptable threshold of task-based fMRI ICC values of 0.4–0.5 to be considered reliable [...] Moreover, Bennett and Miller, as well as a more recent review, highlight that reliability can change on a study-by-study basis depending on several methodical considerations.
Individual fMRI is not a useful diagnostic tool for general conditions. There have been some clinics trying to push it (or SPECT) as a tool for diagnosing things like ADHD or chronic pain, but there is no scientific basis for this. The operator can basically crank up the noise and get some activity to show up, then tell the patient it’s a sign they have “ring of fire type ADHD” because they set the color pattern to reds and a circular pattern showed up at some point.
The BOLD response (oxygen-neuronal activity coupling) has been pretty much accepted in neuroscience. There have been criticisms about it (non-neuronal contributions, mysteries of negative responses/correlations) but in general it is pretty much accepted.
The measurement of the BOLD response is well-accepted, but the interpretation of it with respect to cognition is still basically mostly unclear. Most papers assuming BOLD response uniformly can be interpreted as "activation" are quite dubious.
Yes, I stupidly read the headline and said "no duh" but they are making a point about our understanding of brain activity. I was thinking about the part of the signal that is reliably filtered out, they are talking about something else. Sorry, I was wrong.
They are indeed coupled, but the coupling is complicated and may be situationally dependent.
Honestly, it's hard to imagine many aggregate measurements that aren't. For example, suppose you learn that the average worker's pay increased. Is it because a) the economy is booming or b) the economy crashed and lower-paid workers have all been laid off (and are no longer counted).
this headline is a bit misleading on the first read, since it only affects functional (f)MRI, which is controversial since a longer time. a prominent example is the activity that has been detected in a dead salmon
As the first author of the salmon paper, yes, this was exactly our point. fMRI can be an amazing tool, but if you are going to trust the results you need to have proper statistical corrections along the way.
It's not that fMRI itself is controversial, it's that it is prone to statistical abuse unless you're careful in how you analyse the data. That's what the dead salmon study showed - some voxels will appear "active" purely by statistical chance, so without correction you will get spurious activations.
This study questions the fMRI method itself, not the statistical analysis (you're right that the dead salmon study was challenging the way statistical analysis is done). Basically, this study claims that the association between the BOLD signal measured by fMRI and actual brain activity is quite weak, and they are even anti-correlated in 40% of cases.
There is no statistical analysis that can save you if your interpretation of a signal is wrong (for example, you can't get information about personality from phrenology, regardless of what statistical analysis you try to apply to the data). That's not to say that we need to just trust this study implicitly - I'm just trying to describe how serious of a problem to the field their claim is.
f is functional.
MRIs are basically huge magnets used for imaging. When you apply a strong magnetic field, different tissue types and densities will react differently, and the MRI is basically measuring how those tissues react to the magnet. It is very good for imaging soft tissues, but not so much bone. Someone figured out that you can measure blood flow using the MRI, because blood cells react in a magnetic field, then "relax" at a known rate. Since we can measure blood flow, that is correlated with increased brain activity, i.e. since more neurons are firing, they require more energy, and therefore more blood. So, fMRI is using blood flow as a proxy for brain activity.
Fmri doesn’t measure blood flow, it measures the oxygen level in the blood. Hemoglobin molecules change shape when they carry oxygen and the different shapes react differently to magnets, which is a real stroke of luck
Yep, this is why it's also called BOLD imaging, for blood-oxygenation-level-dependent fMRI. I did my PhD is BME and brain-computer interfaces, but it has been a while since I worked in the field.
I'm going to follow on a bit from what jawilson said. The idea is this - you can measure blood oxygenation by sticking your head in a big magnet that makes atoms spin really fast and measuring the radio waves that come off. This is imprecise, but reasonably repeatable.
So if I show you a picture of a cat, and you like cats, then a bit of your brain might start using more oxygen because you're thinking about cute furry things, and if I show you a picture of a car, and you like cars, a different bit of your brain lights up showing more oxygen use because you're thinking about fast shiny things.
But really we've only got the barest idea of what bits of the brain do what, and maybe it's a bit of brain that goes "hey I'm happy" that lights up in both cases because you like both cats and cars.
We can kind of see bits we think are associated with muscle movement coming to life if I show you a picture of a bike, and you like cycling, and if I show you a really cool mountain track you imagine belting down it flat out. That lights up differently if I show you something else.
However, we do not really know except in very broad terms what bits of the brain actually do what. We can't "see thoughts", we just know that some bits of brain seem to use more oxygen than others, and from that we guess "this bit of brain is for thinking about sitting in a nice cafe with a cup of coffee and a newspaper" versus "this bit of brain is for being frightened of lions".
At least when phrenology was a thing, the ceramic heads with lines painted on were inexpensive and didn't require three-phase power and huge barrels of liquid helium.
If you apply enough gain and filtering to an unknown signal, eventually you'll pull something out of it that you can convince yourself is what you're looking for.
I wonder how much variation there is between a person who does certain mental activity regularly vs a person who rarely does it.
If they were to measure a person who performs mental arithmetic on a daily basis, I'd expect his brain activity and oxygen consumption to be lower than those of a person who never does it. How much difference would that make?
I worked in an fMRI lab briefly as a grad student. I suspect you'd be correct but perhaps not exactly why you'd expect. Studies using fMRI measure a blood-oxygenation-level-dependent (BOLD) signal in the brain. This is thought to be an indirect measure of neural activity because a local increase in neural firing rate produces a local increase in the need for, and delivery of, oxygenated blood.
The question then is, do you expect a person who is really good at mental arithmetic to have less neural firing on arithmetic tasks (e.g., what is 147 x 38) than the average joe. I would hypothesize yes overall to solve each question; however, I'd also hypothesize the momentary max intensity of the expert to peak higher. Think of a bodybuilder vs. a SWE bench-pressing 100 lbs for 50 reps. The bodybuilder has way more muscle to devote to a single rep, and will likely finish the set in 20 seconds, while the SWE is going to take like 30 minutes ;)
It involved going to the lab and practicing the thing (a puzzle / maze) I would be shown during the actual MRI. I think I went in to “practice” a couple times before showing up and doing it in the machine.
IIRC the purpose of practicing was exactly that, to avoid me trying ti learn something during the scan (since that wasn’t the intention of the study).
In other words, I think you can control for that variable.
(Side note: I absolutely fell asleep during half the scan. Oops! I felt bad, but I guess that’s a risk when you recruit sleep deprived college kids!)
For task fMRI, the test-retest reliability is so poor it should probably be considered useless or bordering on pseudoscience, except for in some very limited cases like activation of the visual and/or auditory and/or motor cortex with certain kinds of clear stimuli. For resting-state fMRI (rs-fMRI), the reliabilities are a bit better, but also still generally extremely poor [1-3].
There are also two IMO major and devastating theoretical concerns re fMRI that IMO make the whole thing border on nonsense. One is the assumed relation between the BOLD signal and "activation", and two is the extremely horrible temporal resolution of fMRI.
It is typically assumed that the BOLD response (increased oxygen uptake) (1) corresponds to greater metabolic activity, and (2) increased metabolic activity corresponds to "activation" of those tissues. This trades dubiously on the meaning of "activation", often assuming "activation = excitatory", when we know in fact much metabolic activity is inhibitory. fMRI cannot distinguish between these things.
There are other deeper issues, in that it is not even clear to what extent the BOLD signal is from neurons at all (could be glia), and it is possible the BOLD signal must be interpreted differently in different brain regions, and that the usual analyses looking for a "spike" in BOLD activity are basically nonsense, since BOLD activity isn't even related to this at all, but rather the local field potential, instead. All this is reviewed in [4].
Re: temporal resolution, essentially, if you pay attention to what is going on in your mind, you know that a LOT of thought can happen in just 0.5 seconds (think of when you have a flash of insight that unifies a bunch of ideas). Or think of how quickly processing must be happening in order for us to process a movie or animation sequence where there are up to e.g. 10 cuts / shots within a single second. There is also just biological evidence that neurons take only milliseconds to spike, and that a sequence of spikes (well under 100ms) can convey meaningful information.
However, the lowest temporal resolutions (repetition times) in fMRI are only around 0.7 seconds. IMO this means that the ONLY way to analyze fMRI that makes sense is to see it as an emergent phenomenon that may be correlated with certain kinds of long-term activity reflecting cyclical BOLD patterns / low-frequency patterns of the BOLD response. I.e. rs-fMRI is the only fMRI that has ever made much sense a priori. The solution to this is maybe to combine EEG (extremely high temporal resolution, clear use in monitoring realtime brain changes like meditative states and in biofeedback training) with fMRI, as in e.g. [5]. But, it may still well be just the case fMRI remains mostly useless.
[1] Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020). What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis. Psychological Science, 31(7), 792–806. https://doi.org/10.1177/0956797620916786
[2] Herting, M. M., Gautam, P., Chen, Z., Mezher, A., & Vetter, N. C. (2018). Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies. Developmental Cognitive Neuroscience, 33, 17–26. https://doi.org/10.1016/j.dcn.2017.07.001
[3] Termenon, M., Jaillard, A., Delon-Martin, C., & Achard, S. (2016). Reliability of graph analysis of resting state fMRI using test-retest dataset from the Human Connectome Project. NeuroImage, 142, 172–187. https://doi.org/10.1016/j.neuroimage.2016.05.062
[5] Ahmad, R. F., Malik, A. S., Kamel, N., Reza, F., & Abdullah, J. M. (2016). Simultaneous EEG-fMRI for working memory of the human brain. Australasian Physical & Engineering Sciences in Medicine, 39(2), 363–378. https://doi.org/10.1007/s13246-016-0438-x
Even if neuronal activity is (obviously) faster, the (assumed) neuro-vascular coupling is slower. Typically there are several seconds till you get a BOLD response after a stimulus or task, and this has nothing to do with fMRI sampling rate (fNIRS can have much faster sampling rate, but the BOLD response it measures is equally slow, too). Think of it as that neuronal spiking happens in a range of up to some hundred milliseconds and the body changing the blood flow happens much slower than that.
The issue is that measuring the BOLD response, even in best case scenario, is a very very indirect measure of neuronal activity. This is typically lost when people referring to fMRI studies as discovering "mental representations" in the brain and other non-sense, but here we are. Criticising the validity of the BOLD response itself, though, is certainly interesting.
Right, my point is sort of that both the BOLD response and fMRI sampling rates are far too "slow" (not nearly approaching the Nyquist frequency, I guess) a priori to deeply investigate something as fast as cognition.
Depends on what you mean by cognition, but as you yourself said, BOLD may be correlated with certain kinds of long(er)-term activity, and that in itself is very useful if interpreted carefully. No one claims to detect single "thoughts" or anything of the sort, at least I haven't seen anything so shameless.
Well, a lot of task fMRI designs are pretty shameless and clearly haven't taken the temporal resolution issues seriously, at least when it comes to interpreting their findings in discussions (i.e. claiming that certain regions being involved must mean certain kind of cognition, e.g. "thoughts" must be involved too). And there have definitely been a few papers trying to show they can e.g. reconstruct the image ("thought") in a person's mind from the fMRI signal.
But I don't think we are really disagreeing on anything major here. I do think there is likely some useful potential locked away in carefully designed resting-state fMRI studies, probably especially for certain chronic and/or persistent systemic cognitive things like e.g. ADHD, autism, or, perhaps more fruitfully, it might just help with more basic understanding of things like sleep. But, I also won't be holding my breath for anything major coming out of fMRI anytime soon.
fMRI is a cool, expensive tech, like so many others in genetics and other diagnostics. These technologies create good jobs ("doing well by doing good").
But as other comments point out, and practitioners know, their usefulness for patients is more dubious.
If you actually read their paper, you will find that it's only the sign of the correlation that is being questioned. The field has generally been aware of this interpretational gap, and that's why two-sided hypothesis tests are important. Cellular neuroscience and electrophysiology are only starting to face the challenges that fMRI faced 2 decades ago.
To me this is like shitting on cars in 1925 because they kill people every now and then. Cars didn't go away, and nor will fMRI, until someone finds a better way to measure living people's brains.
TUM's press is being sloppy, from conflating fMRI with MRI to presuming this is revolutionary, and ignoring earlier empirical work against this narrative (Windkessel's, Logothetis beta/gamma coupling, etc.)
What's surprising is the desire to have a silver bullet, or not solution.
What's still amazing is fMRI can provide more visual context of what's happening in the brain, in what region, and activities that can help that improve.
There are other complementary technologies like QEEG and SPECT that can also shed a light as well.
It does seem the case that fMRI cann be more of a snapshot photo, and technologies like SPECT can provide more of a regional time lapse of activity.
As the first author on the salmon paper, yes, that was exactly our point. Researchers were capitalizing on chance in many cases as they failed to do effective corrections to the multiple comparisons problem. We argued with the dead fish that they should.
Curious what you find to be "bs" about the results of this paper? That statistical corrections are necessary when analysing fMRI scans to prevent spurious "activations" that are only there by chance?
Why did TUM let this misleading headline front the news release? Dont we have enough issues with Academia? The result just mean BOLD is an imperfect proxy.
It is especially unforgiveable that the title of on the news release itself is about "40 percent of MRI signals". What, as in all MRI, not just fMRI? Hopefully an honest typo and not just resulting from ignorance.
My previous job was at a startup doing BMI, for research. For the first time I had the chance to work with expensive neural signal measurement tools (mainly EEG for us, but some teams used fMRI). and quickly did I learn how absolute horrible the signal to noise ratio (SNR) was in this field.
And how it was almost impossible to reproduce many published and well cited result. It was both exciting and jarring to talk with the neuroscientist, because they ofc knew about this and knew how to read the papers but the one doing more funding/business side ofc didn't really spend much time putting emphasis on that.
One of the team presented a accepted paper that basically used Deep Learning (Attention) to predict images that a person was thinking of, from the fMRI signals. When I asked "but DL is proven to be able to find pattern even in random noise, so how can you be sure this is not just overfitting to artefact?" and there wasn't really any answer to that (or rather the publication didn't take that in to account, although that can be experimentally determined). Still, a month later I saw tech explore or some tech news writing an article about it, something like "AI can now read your brain" and the 1984 implications yada yada.
So this is indeed something probably most practitioners, masters and PhD, realize relatively early.
So now that someone says "you know mindfulness is proven to change your brainwaves?" I always add my story "yes, but the study was done with EEG, so I don't trust the scientific backing of it" (but anecdotally, it helps me)
There are lots of reliable science done using EEG and fMRI; I believe you learned the wrong lesson here. The important thing is to treat motion and physiological sources of noise as a first-order problem that must be taken very seriously and requires strict data quality inclusion criterion. As far as deep learning in fMRI/EEG, your response about overfitting is too sweepingly broad to apply to the entire field.
To put it succinctly, I think you have overfit your conclusions on the amount of data you have seen
I would argue in fact almost all fMRI research is unreliable, and formally so (test-retest reliabilities are in fact quite miserable: see my post below).
https://news.ycombinator.com/item?id=46289133
EDIT: The reason being, with reliabilities as bad as these, it is obvious almost all fMRI studies are massively underpowered, and you really need to have hundreds or even up to a thousand participants to detect effects with any statistical reliability. Very few fMRI studies ever have even close to these numbers (https://www.nature.com/articles/s42003-018-0073-z).
That depends immensely on the type of effect you're looking for.
Within-subject effects (this happens when one does A, but not when doing B) can be fine with small sample sizes, especially if you can repeat variations on A and B many times. This is pretty common in task-based fMRI. Indeed, I'm not sure why you need >2 participants expect to show that the principle is relatively generalizable.
Between-subject comparisons (type A people have this feature, type B people don't) are the problem because people differ in lots of ways and each contributes one measurement, so you have no real way to control for all that extra variation.
Precisely, and agreed 100%. We need far more within-subject designs.
You would still in general need many subjects to show the same basic within-subject patterns if you want to claim the pattern is "generalizable", in the sense of "may generalize to most people", but, precisely depending on what you are looking at here, and the strength of the effect, of course you may not need nearly as much participants as in strictly between-subject designs.
With the low test-retest reliability of task fMRI, in general, even in adults, this also means that strictly one-off within-subject designs are also not enough, for certain claims. One sort of has to demonstrate that even the within-subject effect is stable too. This may or may not be plausible for certain things, but it really needs to be considered more regularly and explicitly.
Yes on many of those fronts, although not all those papers support your conclusion. The field did/does too often use tasks with to few trials, with to few participants. That always frustrated me as my advisor rightly insisted we collect hundreds of participants for each study, while others would collect 20 and publish 10x faster than us.
Yes, well "almost all" is vague and needs to be qualified. Sample sizes have improved over the past decade for sure. I'm not sure if they have grown on median meaningfully, because there are still way too many low-N studies, but you do see studies now that are at least plausibly "large enough" more frequently. More open data has also helped here.
EDIT: And kudos to you and your advisor here.
EDIT2: I will also say that a lot of the research on fMRI methods is very solid and often quite reproducible. I.e. papers that pioneer new analytic methods and/or investigate pipelines and such. There is definitely a lot of fMRI research telling us a lot of interesting and likely reliable things about fMRI, but there is very little fMRI research that is telling us anything reliably generalizable about people or cognition.
The small sample sizes is rational response from scientists in the face of a) funding levels and b) unreasonable expectations from hiring/promotion committees.
cog neuro labs need to start organizing their research programs more like giant physics projects. Lots of PIs pooling funding and resources together into one big experiment rather than lots of little underpowered independent labs. But it’s difficult to set up a more institutional structure like this unless there’s a big shift in how we measure career advancement/success.
+1 to pooling funding and resources. This is desperately needed in fMRI (although site and other demographic / cultural effects make this much harder than in physics, I suspect).
I'm not an expert, but my hunch would be that a similar Big(ger) Science approach is also needed in areas like nutrition and (non-neurological) experimental psychology where (apparently) often group sizes are just too small. There are obvious drawbacks to having the choice of experiments controlled by consensus and bureaucracy, but if the experiments are otherwise not worthwhile what else is there to do?
I think the problems in nutrition are far, far deeper (we cannot properly control diet in most cases, and certainly not over long timeframes; we cannot track enough people long enough to measure most effects; we cannot trust the measurement i.e. self-report of what is consumed; industry biases are extremely strong; most nutrition effects are likely small and weak and/or interact strongly with genetics, making the sample size requirements larger still).
I'm not sure what you mean by "experimental psychology" though. There are areas like psychophysics that are arguably experimental and have robust findings, and there are some decent-ish studies in clinical psychology too. Here the group sizes are probably actually mostly not too bad.
Areas like social psychology have serious sample size problems, so might benefit, but this field also has serious measurement and reproducibility problems, weak experimental designs, and particularly strong ideological bias among the researchers. I'm not sure larger sample sizes would fix much of the research here.
> Areas like social psychology have serious sample size problems, so might benefit, but this field also has serious measurement and reproducibility problems, weak experimental designs, and particularly strong ideological bias among the researchers. I'm not sure larger sample sizes would fix much of the research here.
I can believe it; but a change doesn't have to be sufficient to be ncessary.
Agreed, it is needed regardless.
which is why the good labs follow up fMRI results and then go in with direct neurophysiological recording...
I have heard and seen good things about QEEG and fMRI as well.
But none of this (signal/noise ratio, etc) is related to the topic of the article, which claims that even with good signal, blood flow is not useful to determine brain activity.
I'm not sure I understand. Wouldn't any prediction result above statistical random (in the image mind reading study) be significant? If the study was performed correctly I don't really need to know much about fMRI to tell whether it's an interesting result or not.
The difference is that EEG can be used usefully in e.g. biofeedback training and the study of sleep phases, so there is in fact enough signal here for it to be broadly useful in some simple cases. It is not clear fMRI has enough signal for anything even as simple as these things though.
I have been told QEEG can offer an additional perspective in neurofeedback, etc as well.
fMRI's are being used in TBI/Concussion recovery that are study backed and seem to be delivering results.
Yes, there are a few medical cases where fMRI makes good simple basic sense, and TBI/Concussion sounds immediately like one of those to me. I seem also to recall them being useful in some cases prior to brain surgeries and the like.
This all makes sense because fMRI tracks metabolic activity via oxygenation changes, which is much more clearly and plausibly related to tissue health and recovery. In these cases, it is also most likely being used within-subject (i.e. longitudinally) to make comparisons to baselines, rather than in an attempt to make speculative inferences about the mind using groups of people, and likely is a simple comparison to baseline rather than bespoke statistical analyses relying on questionable assumptions about the BOLD response being related to overly-specific kinds of neural activity.
fMRI can track oxygenation changes, and indirectly where the blood flow is, or isn't, and perhaps some ideas on where to get it.
All to say, this application might not fall in the 40%.
I just find articles like these can't help but feel like they have an agenda to undermine something instead of simply acknowledge the kinds of things it is and isn't working for.
There's no doubt these researchers have found something, but the need for sensationalistic headlines is well known in academia as well.
Sometimes it's noticeable where the research is specific in scope, but the findings are more general and broad.
> fMRI's are being used in TBI/Concussion recovery
Interesting. Do you happen to have any more information on this topic? I ask because I was under the impression that concussions are a functional/metabolic injury and not a structural injury, therefore, concussions are not visible on any type of fMRI, CT Scan, etc.. Though, I haven't looked into this topic for almost half a decade, so I imagine things have likely progressed.
Well fMRI (as opposed to MRI) is used precisely because it measures things directly related to metabolism and function. Not hard to find info on this stuff: https://scholar.google.ca/scholar?hl=en&as_sdt=0%2C5&q=fMRI%...
Concussions seem to be pretty physiological - first they're a brain bleed, and blood doesn't seem to pump the same as it did before the concussion... resulting in different symptoms.
That might be what you're referring to as functional?
Metabolically, or otherwise, if the brain can't operate, other things in the body such as metabolism would be impacted for sure when it can't oversee and run as it normally can?
While I'm not sure if a concussion directly is visible or not (some have sizeable enough brain bleeds that can be visible), concussions to the extent that they are a change in blood circulation changes and issues, can be visualized on fMRI, etc, where it's not regular, those areas suffer in a brain.
Things luckily have progressed and quite exciting.
Out of convenience, I'll share one I know about (no affiliation) that lay out their therapies and the science behind it as well.
Effectively (I hope I'm getting this accurately) it seems the blood vessels in the brain also have signalling from the blood and oxygen that gets affected which affects things downstream from there.
These guys do an fMRI baseline, have you jump on a bike, fMRI again, see what's not getting blood, and then give you exercises and activites for those regions of the brain. It's pretty interesting.
https://www.cognitivefxusa.com/treatment
Some reported patient outcomes: https://www.cognitivefxusa.com/our-patients
Blog links to research: https://www.cognitivefxusa.com/blog
Independently of this I've heard QEEGs can do a similar thing of seeing where brain activity is/isn't baseline.
90% of papers I read in computer science / computer security speak of software written or AI models they trained that are nowhere to be found. Not on git nor via email to the authors.
There's fancier ML studies on EEG signal but probably not consistent enough for clinical work. For now, the one thing EEG can reliably tell is if you're having a seizure or not, if you're delirious (or in a coma) or not, or if you're asleep.
> When I asked "but DL is proven to be able to find pattern even in random noise, so how can you be sure this is not just overfitting to artefact?"
So here you say quite a mouthful. If you train it on a pattern it'll see that pattern everywhere - think about the early "Deep Dream" trippy-dogs-pictures nonsense that was pervasive about eight or nine years ago.
I repaired a couple of cameras for someone who was working with a large university hospital about 15 years ago, where they were using admittedly 2010s-era "Deep Learning" to analyse biopsy scans for signs of cancer. It worked brilliantly, at least with the training materials, incredible hit rate, not too terrible false positive rate (no biggie, you're just trying to decide if you want to investigate further), really low false negative rate (if there was cancer it would spot it, for sure, and you don't want to miss that).
But in real-world patient data it went completely mental. The sample data was real-world patient data, too, but on "uncontrolled" patients, it was detecting cancer all over the place. It also detected cancer in pictures of the Oncology department lino floor, it detected cancer in a picture of a guy's ID badge, it detected cancer in a closeup of my car tyre, and it detected cancer in a photo of a grey overcast sky.
Aw no. Now what?
Well, that's why I looked at the camera for them. They'd photographed the biopsies with one camera on site, from "real patients", but a lot of the "clear" biopsies were from other sites.
You're ahead of me now, aren't you?
The "Deep Learning" system had in fact trained itself on a speck of shit on the sensor of one of the cameras, the one used for most of the "has cancer" biopsies and most of the "real patient under test" biopsies. If that little blob of about a dozen slightly darker pixels was present, then it must be cancer because that's what the grown-ups told it. The actual picture content was largely irrelevant because the blob was consistent across all of them.
I'm not too keen on AI in healthcare, not as a definitive "go/no-go" test thing.
I remember reading a paper back in grad school where the researchers put a dead salmon in the magnet and got statistically significant brain activity readings using whatever the analysis method à la mode was. It felt like a great candidate for the Ig Nobel awards.
That was our paper! We showed that you can get false positives (significant brain activity in this case) if fMRI if you don't use the proper statistical corrections. We did win an Ig Nobel for that work in 2012 - it was a ton of fun.
I was a grad student at UCSD when Ed Vul published Voodoo Correlations in Social Neuroscience [1], which stoked a severe backlash from the fMRI syndicate resulting in a title change to Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition [2]. There is a lot of interesting commentary around this article (e.g., “Voodoo” Science in Neuroimaging: How a Controversy Transformed into a Crisis [3]). To me it was fascinating to watch Vul (an incredibly rare talent, perhaps a genius), take on an entire field during his 1st year as assistant professor.
1. http://prefrontal.org/blog/2009/01/voodoo-correlations-in-so...
2. https://journals.sagepub.com/doi/10.1111/j.1745-6924.2009.01...
3. https://www.mdpi.com/2076-0760/12/1/15
I'll get raked for this, but as someone in the field, I can say with high confidence that the majority of comments in this thread are not from imaging experts, and mostly (mis)informed by popular science articles. I do not have the time to properly respond to each issue I see. The literature is out there in any case.
> I do not have the time to properly respond to each issue I see. The literature is out there in any case.
I think your expertise would be very welcome, but this comment is entirely unhelpful as-is. Saying there are bad comments in this thread and also that there is good literature out there without providing any specifics at all is just noise.
You don't have to respond to every comment you see to contribute to the discussion. At minimum, could you provide a hint for some literature you suggest reading?
I have also published and worked for some years in this field, if that helps.
The literature is huge, and my bias is that I believe most of the only really good fMRI research is methodological research (i.e. about what fMRI actually means, and how to reliably analyze it). Many of the links I've provided here speak to this.
I don't think there is much reliable fMRI research that tells us anything about people, emotions, or cognition, beyond confirming some likely localization of function to the sensory and motor cortices, and some stuff about the Default Mode Network(s) that is of unclear importance.
A lot of the more reliable stuff involves the Human Connectome Project (HCP) fMRI data, since this was done very carefully with a lot of participants, if you want a place to start for actual human-relevant findings. But the field is still really young.
> Saying there are bad comments in this thread and also that there is good literature out there without providing any specifics at all is just noise.
Nah, it's not noise. It's a useful reminder not to take any comments too seriously and that this topic is far outside the average commenter's expertise.
It's definitely noise. Not recognizing it as noise is why phone and email scams work.
I say this as a psychologist who is advising you to ignore all claims to the contrary, because they are misinformed. It is clear from the literature.
I'll co-sign SubiculumCode's comment -- there's a lot of yelling about how bad fMRI is generally, which is not particularly fair to the fMRI research (or at least the better parts of it) or related to the argument.
The BOLD signal, the thing measured by fMRI, is a proxy for actual brain activity. The logic is that neural firing requires a lot of energy and so active neurons will being using more oxygen for their metabolism, and this oxygen comes from the blood. Thus, if you measure local changes in the oxygenation of blood, you'll know something about how active nearby neurons are. However, it's an indirect and complicated relationship. The blood flow to an area can itself change, or cells could extract more or less oxygen from the blood--the system itself is usually not running at its limits.
Direct measurements from animals, where you can measure (and manipulate) brain activity while measuring BOLD, have shown how complicated this is. Nikos Logathetis and Ralph Freeman's groups, among many others did a lot of work on this, especially c. 2000-2010. If you're interested, you could check out this news and views on Logathetis's group's 2001 Nature paper [1]. One of the conclusions of their work is that BOLD is influenced by a lot of things but largely measure the inputs to an area and the synchrony within it, rather than just the average firing rate.
In this paper, the researchers adjust the MRI sequences to compare blood oxygenation, oxygen usage, and blood flow and find that these are not perfectly related. This is a nice demonstration, but not a totally unexpected finding either. The argument in the paper is also not "abandon fMRI" but rather that you need to measure and interpret these things carefully.
In short, the whole area of neurovascular coupling is hard--it includes complicated physics (to make measurements), tricky chemistry, and messy biology, all in a system full of complicated dynamics and feedback.
[1] https://www.nature.com/articles/35084300
Seeing HN take on your speciality or topic can be brutal.
Condolences.
I hide any thread that deals with education, education funding, or teaching in general for that specific reason. It really saddens me to see that this place is full of so much misinformation and anecdotes made into data (and usually with much more self-confidence than other forums, which is interesting to me).
It's why I generally only ask questions, or ask for clarification instead of directly challenging something I think might be wrong now in threads that aren't related to something I have deeeeep personal knowledge of. I know when I'm out of my area, and don't want to add to the ignorance.
A great habit - especially when your question is an irresistible, easily-addressed homerun to a domain expert wandering through the thread looking for an entry-point.
Challenging something with a question about it, is not adding to ignorance - if a statement/study/fact/belief can't hold to up to questions, from actual opposing critics, what's the point of that position existing?
Being all "PC" and "nice" about stuff that is what it is, or isn't -- THAT adds to ignorance.
I guess maybe I didn't phrase that correctly - I ask challenging questions, but don't state the things I "know" without clarification first. I meant that I don't just pop off with "yeah, but in reality it's x, y, z" because I know that I'm probably ignorant of facts. I'll ask about x, y, or z first.
I’m sure you’re right, but given the spectrum of answers here, it’d be much more useful to point out which ones you think are wrong.
I'm not a specialist by any means, although I am a patient of an fMRI. One thing I will note is that in the eventual, resultant paperwork from the broad array of tests I had, the fMRI was not noted whatsoever, neither was it discussed with me by any of the numerous neurologists or surgeons involved in my case. I was quite curious as to why it was performed at all, but presumably it was some formality to check a box.
It makes sense they wouldn't look at it, there are very few, if any, well-validated clinical uses for it. However, they might have taken it as a baseline for later comparison, and it is definitely plausible when surgery is involved that visible abnormalities could be seen in fMRI that might not show up in MRI, either now or later.
I don't think there would be much clear guidance for them on how to interpret any such fMRI abnormality on its own, but it might still be something useful for further investigations, and this might especially be the case for surgery. It might also have been done as part of research, if you consented to anything like that?
I am NOT an expert on fMRI in medical contexts, but you can surely get a rough idea of the potential value of fMRI with a quick search: https://scholar.google.ca/scholar?q=fMRI+surgery+brain&hl=en...
There are literally less than 20 top level comments and this one is (at least for me) the 2nd or 3rd.
Instead of a nothingburger, you could have used your academic prowess to break down the top 1/2 misconceptions with expertise.
You might not have time to respond to all the comments but a couple of clarifications could have helped anyone else who doesn't comment without experience.
Just saying that next time you can be the change you want to see in HN instead of wasting text telling us how ignorant we are.
This study is validating a commonplace fMRI measure (change in blood-oxygenation-level-dependent or BOLD signal) by comparing it with a different MRI technique, one that uses a multiparametric quantitative BOLD model, a different model for BOLD derived from two separate MRI scans which measure two different kinds of signal (transverse relaxation rates), and then multiply/divide by a bunch of constants to get at a value.
I'm a software engineer in this field, and this is my layman-learns-a-bit-of-shop-talk understanding of it. Both of these techniques involve multiple layers of statistical assumptions, and multiple steps of "analysing" data, which in itself involves implicit assumptions, rules of thumb and other steps that have never sat well with me. A very basic example of this kind of multi-step data massaging is "does this signal look a bit rough? No worries, let's Gaussian-filter it".
A lot of my skepticism is due to ignorance, no doubt, and I'd probably be braver in making general claims from the image I get in the end if I was more educated in the actual biophysics of it. But my main point is that it is not at all obvious that you can simply claim "signal B shows that signal A doesn't correspond to actual brain activity", when it is quite arguable whether signal B really does measure the ground truth, or whether it is simply prone to different modelling errors.
In the paper itself, the authors say that it is limited by methodology, but because they don't have the device to get an independent measure of brain activation, they use quantitative MRI. They also say it's because of radiation exposure and blah blah, but the real reason is their uni can't afford a PET scanner for them to use.
"The gold standard for CBF and CMRO2 measurements is 15O PET; but this technique requires an on-site cyclotron, a sophisticated imaging setup and substantial experience in handling three different radiotracers (CBF, 15O-water; CBV, 15O-CO; OEF, 15O-gas) of short half-lives8,35. Furthermore, this invasive method poses certain risks to participants owing to the exposure to radioactivity and arterial sampling."
This is why I love this site. You get input from so many specialized folks! I appreciate you contributing your expertise and I also appreciate you calling out the limits to that knowledge.
Two points I'm hoping you can help clarify:
> Researchers ... found that an increased fMRI signal is associated with reduced brain activity in around 40 percent of cases.
So it's not just that they found it was uncorrelated, they found it was anticorrelated in 40% of cases?
And you are suggesting that conclusion suffers from the same potential issues as these fMRI studies in general?
Like you mention, it seems to me if we wanted to really validate the model, we'd have to run the same experiment with two, three, or maybe even more different modalities (fMRI, PET with different tracers, etc).
Most studies in non-clinical populations afaik do not use 150 PET though? Afaik this is mostly used for clinical purposes. Could be wrong though.
If you have a PET/MR system [0], you can probably do this "gold standard" comparison, and I know that one is used for research studies. I think you can piggy-back off a different study's healthy controls to write a paper like this, if that study already uses PET/MR and if adding an oxygen metabolite scan isn't a big problem. But that's speaking as someone who does not design experiments.
[0] https://www.siemens-healthineers.com/en-us/magnetic-resonanc...
Now seems like a good time to remind folks of the Stanford dead fish fMRI study: https://law.stanford.edu/2009/09/18/what-a-dead-salmon-remin...
fMRI has always had folks highlighting how shaky the science is. It's not the strongest of experimental techniques.
This study was really highlighting a statistical issue which would occur with any imaging technique with noise (which is unavoidable). If you measure enough things, you'll inevitably find some false positives. The solution is to use procedures such as Bonferroni and FDR to correct for the multiple tests, now a standard part of such imaging experiments. It's a valid critique, but it's worth highlighting that it's not specific to fMRI or evidence of shaky science unless you skip those steps (other separate factors may indicate shakiness though).
When we published the salmon paper, approximately 25-35% of published fMRI results used uncorrected statistics. For myself and my co-authors, this was evidence of shaky science. The reader of a research paper could not say with certainty which results were legitimate and which might be false positives.
Hey, I know you got a lot of flack for the article. So, I just wanted to thank you for having the courage to publish it anyways and go through all of that for all of us.
I go back to the study frequently when looking at MRI studies, and it always holds up. It always reminds me to be careful with these things and to try to have other be careful with their results too. Though to me it's a bit of a lampooning, surprisingly it has been the best reminder for me to be more careful with my work.
So thank you for putting yourself through all that. To me, it was worth it.
> a statistical issue which would occur with any imaging technique
I sounds like it goes beyond that: If a certain mistake ruins outcomes, and a lot of people are ruining outcomes and not noticing, then there's some much bigger systematic problem going on.
Why are you calling Bennett et al "the Stanford... study" ? Not one person on that team went to Stanford.
Direct link to the poster presentation: http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
Discussed here. others?
Risk of false positives in fMRI of post-mortem Atlantic salmon (2010) [pdf] - https://news.ycombinator.com/item?id=15598429 - Nov 2017 (41 comments)
Scanning dead salmon in fMRI machine (2009) - https://news.ycombinator.com/item?id=831454 - Sept 2009 (1 comment)
fMRI methods and statistics have advanced quite a lot since the dead fish days, that critique does not really hold up today.
While I would agree that the prevalence of the problem has been minimized in fMRI during the last 15 years, I disagree that our critique does not hold up. The root of our concern was that proper statistical correction(s) need to be completed in order for research results to be interpretable. I am totally biased, but I think that remains worthwhile.
I immediately thought of it too. Didn't realize it was that long ago.
In related news: ironically, Psychedelics disrupt normal link between brain’s neuronal activity and blood flow - thus casting some doubt on findings that under psychedelics more of the brain is connected (since fMRI showed elevated blood flow, suggesting higher brain activity).
https://source.washu.edu/2025/12/psychedelics-disrupt-normal...
As a caveman pondering "Stoned Ape Theory" during the rise of MRI in the 80s, having done light reading of Huxley, McKenna et. al, the claim that vascular variations were so tied to thought patterns in a purely calm and cognitive activity was fascinating. To see the brain of someone as they went through a deck of cards and paused to look at each... astounding! But frustrating also. My first question always was, was the person's hands busy going through the deck and holding up the cards, focusing on them... or were they merely shown the cards sitting still? It seemed the popsci articles often glossed over that information, and any simple "control for coordinated body movement" played second fiddle to the novelty of it all. Then I worked in a club where I was often surrounded by tripping people. I'd fetch them glasses of water and they would always drink. Do you know you can smell them, they smell like fear? The experience has every sweat gland working overtime. When I learned that I greeted this "tripping people MRIs light up indicating enhanced brain connectivity" with a grain of salt. I would not be the least bit surprised if the sweat gland thing also has the brain's vascular system in overdrive.
I didn't get my name on this, but contributed to it as an undergrad: https://ieeexplore.ieee.org/abstract/document/6903679
We sped up fMRI analysis using distributed computing (MapReduce) and GPUs back in 2014.
Funny how nothing has changes.
This isn’t entirely news to people in the field doing research, but it’s important information to keep in mind when anyone starts pushing fMRI (or SPECT) scans into popular media discussions about neurology or psychiatry.
There have been some high profile influencer doctors pushing brain imaging scans as diagnostic tools for years. Dr. Amen is one of the worst offenders with his clinics that charge thousands of dollars for SPECT scans (not the same as the fMRI in this paper but with similar interpretation issues) on patients. Insurance won’t cover them because there’s no scientific basis for using them in diagnosing or treating ADHD or chronic pain, but his clinics will push them on patients. Seeing an image of their brain with some colors overlayed and having someone confidently read it like tea leaves is highly convincing to people who want answers. Dr. Amen has made the rounds on Dr. Phil and other outlets, as well as amassing millions of followers on social media.
Dr. Mike, a rare YouTube doctor who is not peddling supplements and wares, and thus seems to be at the forefront of medical critical thinking on the platform, interviewed Dr. Amen recently[0]. I haven't finished the interview yet, but having watched some others, generally the approach is to let the interviewee make their grandiose claims, agree with whatever vague generalities and truisms they use in their rhetoric (yes it's true, doctors don't spend enough time explaining things to patients!), and then lay into them on the actual science and evidence.
[0] https://www.youtube.com/watch?v=J-SHgZ1XPXs
Dr. Mike did an incredible job in that interview. He gave Dr. Amen all the rope to hang himself with his own words. When you're hawking a diagnosis method and you're not interested in building up the foundation of evidence for it by doing a double blinded, randomized controlled study. And that the results of said study would change how your treating patients it's pretty clear who the snake oil salesman is
I also thought the rest of the interview was really worthwhile - they talked a lot about real problems in the medical industry from different perspectives. What a great and critical discussion from Dr. Mike. If Amen had conceded the point they could have moved on. There could be real findings to be had there, and some may even match his conclusions, but many likely will not, and the whole thing could also be pure fiction. We should want better answers to these questions. It's unfortunate to watch someone as seemingly intelligent and well-informed as Amen come across as shilling snake oil, and/or just being hung up on his ego, at the end of it all. Scientific literacy is so critical, because it's easy to cloak pseudoscience behind high-tech smokescreens.
Unfortunately I worry about the rebound effect, where even though the entire interview was debunking his claims this could still on average increase amen’s popularity.
I'm no expert in medicine, but I watched that entire video and your analogy about performance and rope doesn't fit well with how it came across to me.
I actually thought the interviewer was a little disingenuous. He said things like "We're on the same team" and "I'm not trying to trap you", then proceeded to lob his guest with criticisms from the other team and questions aimed to maneuver him into a contradiction. There's nothing inherently wrong with that, but if you're going to do it, be forthright you're engaging in a debate.
Earlier in the interview he could have put his cards on the table and plainly stated "Myself and others in the medical community are skeptical of the efficacy of imaging on outcomes, and a rigorous, double-blind study would lend dramatic support for us to adopt what you're touting."
Then they could have had the conversation he was clearly after, focused on that issue.
Instead it felt like I was watching for ages as he took a winding route to get there, then the interview cut off abruptly when they finally really did.
The overlays applied in editing while helpful and fair in some cases, at other times came across as one-sided. It's a shame we can't see a follow-up where the interviewee has an opportunity to respond (or squirm) in light of them.
For the record I would very much love to see additional research and gold-standard, double-blind studies. In the meantime I'll treat this as "Hey, we've got this interesting thing we can measure, we're seeing some good results in our practice" without over-emphasizing the confidence in this one diagnostic.
I did find the bit interesting about how having a gauge you can viscerally see impacted patients' engagement in care. Both agreed on the potential usefulness of that aspect, and conceded the difference in profiles between patients coming to Dr. Amen vs. ordinary front-line family physicians.
Back in 2009 I remember reading about how dead salmon apparently turns up brain activity in fMRI without proper statistical methods. fMRI studies are something frequently invoked unscientifically and out of context.
https://www.wired.com/2009/09/fmrisalmon/
I think technically there's some statistical correction you apply to the voxels to avoid this. But yea...most hypotheses from fMRI are considered hypotheses until there's some other modality, i.e. electrical recordings, etc that confirm it.
i.e. the well regarded studies, i.e. Kanwisher and the visual processing areas, have follow up studies on primates and surgical volunteers w/ actual electrical activity correlating w/ visual stimuli etc
I thought I was being clever by coining the term "non-invasive phrenology" but it appears people are already using it non-ironically.
In many ways old-school bump measurement is actually less invasive
("wallet biopsy" is another fun term if you haven't encountered it)
Cashectomy.
I saw Parvizi say this in a talk back in 2019!
>> Seeing an image of their brain with some colors overlayed ... is highly convincing
Indeed, there's been quite a few studies [1] that find just including any old image of a brain with stuff highlighted will cause a paper to be perceived as more scientifically credible.
[1] https://pubmed.ncbi.nlm.nih.gov/17803985/
I saw a clinical report of his on a patient, he puts a graphic in their report of their "brain scan" but it's basically a vector graphic of the brain w/ a multicolor MS Paint gradient...
Dr. Amen is more of a marketing/sales guy than a medical expert.
thanks for this comment. it was really insightful thank you.
Pop science guru-ing is a giant flashing red sign for me. I am never even a little surprised when the latest “sense maker” or pop science guru comes out as a complete loon or is consumed by some kind of scandal.
Influencers in general are always suspect. The things that get you an audience fast are trolling or tabloid-ish tactics like conspiracism.
There are good ones but you have to be discerning.
I might be oversimplifying, but isn't a lot of our neurological understanding about ADHD based on "fMRI shows decreased activity in the frontal cortex"? Or for that matter, our neurological understanding of a lot of mental health conditions.
I know the actual diagnosis is several times more layered than this attempt at an explanation, but I always felt that trying to explain the brain by peering at it from outwards is like trying to debug code by looking at a motherboard through a bad microscope.
I do not think there is much neurological understanding about ADHD at all from current fMRI research, there are far too many quality and reliability issues here, not just on the fMRI end or limited amount of data overall, but in the measurement and diagnosis of ADHD itself (i.e. ADHD subtypes, and of course ADHD is a complicated diagnosis with many components manifesting to different degrees in different individuals, which makes it very hard to cleanly link to messy fMRI signals).
Or, as I have commented elsewhere here, the idea that statements like "fMRI shows decreased activity" are ever valid is just fundamentally suspect (lower BOLD response could mean less inhibition or less excitation, and this is a rather crucial difference that fMRI simply can't distinguish). EDIT: Or to be more precise: it may well be that fMRI research suggests less metabolic activity in certain regions, but this could mean the region is actually firing more than normal, less than normal, is more efficient than normal, etc., and interpreting anything about what is functioning differently in ADHD, given this uncertainty, is what is going to be suspect.
Your analogy is largely correct IMO.
Yes this is true, but we actually have a lot more data to back this on than exclusively fMRI analysis - for example the ADHD medication guanfacine works only because the alpha 2 receptors happen to be wired differently in the prefrontal cortex than it is in other areas of the brain (a2 is inhibitory for most brain regions, but in the PFC they're positioned to amplify connections between neurons) , so by stimulating alpha 2 we allow for a more “top down” control from the prefrontal cortex than we do without, which improves executive function.
So that is one extremely robust way to understand neurological conditions like ADHD or Parkinson’s
As someone who used to work at the Cognitive Neurophysiology Lab in the Scripts Institute-- doing some work on functional brain image-- I can confirm this was not news even thirty years ago. I guess this is trying to make some point to lay people?
Are there proposed reasons for increased blood flow to brain regions other than neural activity? Are neurons flushing waste products or something when less active?
Many reasons, and yes, basically, that is one of them.
Ekstrom, A. (2010). How and when the fMRI BOLD signal relates to underlying neural activity: The danger in dissociation. Brain Research Reviews, 62(2), 233–244. https://doi.org/10.1016/j.brainresrev.2009.12.004, https://scholar.google.ca/scholar?cluster=642045057386053841...
The glymphatic system, sure.
fMRI has been abused by a lot of researchers, doctors, and authors over the years even though experts in the field knew the reality. It’s worth repeating the challenges of interpreting fMRI data to a wider audience.
The way I understood it is that while individual fMRI studies can be amazing, it is borderline impossible to compare them when made using different people or even different MRI machines. So reproducibility is a big issue, even though the tech itself is extremely promising.
The article is pointing out that one of the base assumptions behind fMRI, that increased blood flow (which is what the machine can image) is strongly correlated to increased brain activity (which is what you want to measure) is not true in many situations. This means that the whole approach is suspect if you can't tell which situation you're in.
fMRI ususally measures BOLD, changes in blood oxygenation (well, deoxygenation). The point of the paper is that you can get relative changes like that in lots of ways: you could have more or less blood, or take out more/less oxygen from the same blood.
These can be measured themselves separately (that's exactly what they did here!) and if there's a spatial component, which the figures sort of suggest, you can also look at what a particular spot tends to do. It may also be interesting/important to understand why different parts of the brain seem to use different strategies to meet that demand.
This isn't really true. The issue is that when you combine data across multiple MRI scanners (sites), you need to account for random effects (e.g. site specific means and variances)...see solutions like COMBAT. Also if they have different equipment versions/manufacturers those scanners can have different SNR profiles. The other issue is that there are many processing with many ways to perform those steps. In general, researchers don't process in multiple ways and choose the way that gives them the result they want or anything nefarious like that, but it does make comparisons difficult since the effects of different preprocessing variations can be significant. To defend against this, many peer reviewers, like myself, request researchers perform the preprocessing multiple ways to assess how robust the results are to those choices. Another way the field has combatted this issue has been software like fMRIprep.
It is in fact even difficult to compare the same person on the same fMRI machine (and especially in developmental contexts).
Herting, M. M., Gautam, P., Chen, Z., Mezher, A., & Vetter, N. C. (2018). Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies. Developmental Cognitive Neuroscience, 33, 17–26. https://doi.org/10.1016/j.dcn.2017.07.001
I read that paper as suggesting that development, behavior, and fMRI are all hard.
It's not at all clear to me that teenagers' brains OR behaviours should be stable across years, especially when it involves decision-making or emotions. Their Figure 3 shows that sensory experiments are a lot more consistent, which seems reasonable.
The technical challenges (registration, motion, etc) like things that will improve and there are some practical suggestions as well (counterbalancing items, etc).
While I agree I wouldn't expect too much stability in developing brains, unfortunately there are pretty serious stability issues even in non-developing adult brains (quote below from the paper, for anyone who doesn't want to click through).
I agree it makes a lot of sense though the sensory experiments are more consistent, somatosensory and sensorimotor localization results generally seem to the be most consistent fMRI findings. I am not sure registration or motion correction is really going to help much here, I suspect the reality is just that the BOLD response is a lot less longitudinally stable than we thought (brain is changing more often and more quickly than we expected).
Or if we do get better at this, it will be more sophisticated "correction" methods (e.g. deep-learners that can predict typical longitudinal BOLD changes, and those better allow such changes to be "subtracted out", or something like that). But I am skeptical about progress here given the amount of data needed to develop any kind of corrective improvements in cases where there are such low longitudinal reliabilities.
===
> Using ICCs, recent efforts have examined test-retest reliability of task-based fMRI BOLD signal in adults. Bennett and Miller performed a meta-analysis of 13 fMRI studies between 2001 and 2009 that reported ICCs. ICC values ranged from 0.16 to 0.88, with the average reliability being 0.50 across all studies. Others have also suggested a minimal acceptable threshold of task-based fMRI ICC values of 0.4–0.5 to be considered reliable [...] Moreover, Bennett and Miller, as well as a more recent review, highlight that reliability can change on a study-by-study basis depending on several methodical considerations.
Individual fMRI is not a useful diagnostic tool for general conditions. There have been some clinics trying to push it (or SPECT) as a tool for diagnosing things like ADHD or chronic pain, but there is no scientific basis for this. The operator can basically crank up the noise and get some activity to show up, then tell the patient it’s a sign they have “ring of fire type ADHD” because they set the color pattern to reds and a circular pattern showed up at some point.
The BOLD response (oxygen-neuronal activity coupling) has been pretty much accepted in neuroscience. There have been criticisms about it (non-neuronal contributions, mysteries of negative responses/correlations) but in general it is pretty much accepted.
The measurement of the BOLD response is well-accepted, but the interpretation of it with respect to cognition is still basically mostly unclear. Most papers assuming BOLD response uniformly can be interpreted as "activation" are quite dubious.
Yes, I stupidly read the headline and said "no duh" but they are making a point about our understanding of brain activity. I was thinking about the part of the signal that is reliably filtered out, they are talking about something else. Sorry, I was wrong.
Good for you George E Collins.
Really? This was known: "there is no generally valid coupling between the oxygen content measured by MRI and neuronal activity"?
"generally valid" is a bit of a loaded phrase.
They are indeed coupled, but the coupling is complicated and may be situationally dependent.
Honestly, it's hard to imagine many aggregate measurements that aren't. For example, suppose you learn that the average worker's pay increased. Is it because a) the economy is booming or b) the economy crashed and lower-paid workers have all been laid off (and are no longer counted).
The coupling was always debated, but you are right, that wasn't known or at least decided. I made a mistake and you are right.
Hasty post. I apologize.
this headline is a bit misleading on the first read, since it only affects functional (f)MRI, which is controversial since a longer time. a prominent example is the activity that has been detected in a dead salmon
The dead salmon was just a lesson in failing to correct for multiple comparisons.
As the first author of the salmon paper, yes, this was exactly our point. fMRI can be an amazing tool, but if you are going to trust the results you need to have proper statistical corrections along the way.
Cheers!
It's not that fMRI itself is controversial, it's that it is prone to statistical abuse unless you're careful in how you analyse the data. That's what the dead salmon study showed - some voxels will appear "active" purely by statistical chance, so without correction you will get spurious activations.
This study questions the fMRI method itself, not the statistical analysis (you're right that the dead salmon study was challenging the way statistical analysis is done). Basically, this study claims that the association between the BOLD signal measured by fMRI and actual brain activity is quite weak, and they are even anti-correlated in 40% of cases.
There is no statistical analysis that can save you if your interpretation of a signal is wrong (for example, you can't get information about personality from phrenology, regardless of what statistical analysis you try to apply to the data). That's not to say that we need to just trust this study implicitly - I'm just trying to describe how serious of a problem to the field their claim is.
So, is fMRI like "fast" MRI? Can someone fill the rest of us mortals in on this? :)
f is functional. MRIs are basically huge magnets used for imaging. When you apply a strong magnetic field, different tissue types and densities will react differently, and the MRI is basically measuring how those tissues react to the magnet. It is very good for imaging soft tissues, but not so much bone. Someone figured out that you can measure blood flow using the MRI, because blood cells react in a magnetic field, then "relax" at a known rate. Since we can measure blood flow, that is correlated with increased brain activity, i.e. since more neurons are firing, they require more energy, and therefore more blood. So, fMRI is using blood flow as a proxy for brain activity.
Fmri doesn’t measure blood flow, it measures the oxygen level in the blood. Hemoglobin molecules change shape when they carry oxygen and the different shapes react differently to magnets, which is a real stroke of luck
Yep, this is why it's also called BOLD imaging, for blood-oxygenation-level-dependent fMRI. I did my PhD is BME and brain-computer interfaces, but it has been a while since I worked in the field.
I'm going to follow on a bit from what jawilson said. The idea is this - you can measure blood oxygenation by sticking your head in a big magnet that makes atoms spin really fast and measuring the radio waves that come off. This is imprecise, but reasonably repeatable.
So if I show you a picture of a cat, and you like cats, then a bit of your brain might start using more oxygen because you're thinking about cute furry things, and if I show you a picture of a car, and you like cars, a different bit of your brain lights up showing more oxygen use because you're thinking about fast shiny things.
But really we've only got the barest idea of what bits of the brain do what, and maybe it's a bit of brain that goes "hey I'm happy" that lights up in both cases because you like both cats and cars.
We can kind of see bits we think are associated with muscle movement coming to life if I show you a picture of a bike, and you like cycling, and if I show you a really cool mountain track you imagine belting down it flat out. That lights up differently if I show you something else.
However, we do not really know except in very broad terms what bits of the brain actually do what. We can't "see thoughts", we just know that some bits of brain seem to use more oxygen than others, and from that we guess "this bit of brain is for thinking about sitting in a nice cafe with a cup of coffee and a newspaper" versus "this bit of brain is for being frightened of lions".
At least when phrenology was a thing, the ceramic heads with lines painted on were inexpensive and didn't require three-phase power and huge barrels of liquid helium.
If you apply enough gain and filtering to an unknown signal, eventually you'll pull something out of it that you can convince yourself is what you're looking for.
wondering why you are downvoted. You are right, though it's kind of inferred that the author means fMRI as the title focuses on brain activity only.
Structural MRI does not record brain activity, because it is, like, structural, not functional.
Structural MRI is even more abused, where people find "differences" between 2 groups with ridiculously small sample sizes.
I wonder how much variation there is between a person who does certain mental activity regularly vs a person who rarely does it.
If they were to measure a person who performs mental arithmetic on a daily basis, I'd expect his brain activity and oxygen consumption to be lower than those of a person who never does it. How much difference would that make?
I worked in an fMRI lab briefly as a grad student. I suspect you'd be correct but perhaps not exactly why you'd expect. Studies using fMRI measure a blood-oxygenation-level-dependent (BOLD) signal in the brain. This is thought to be an indirect measure of neural activity because a local increase in neural firing rate produces a local increase in the need for, and delivery of, oxygenated blood.
The question then is, do you expect a person who is really good at mental arithmetic to have less neural firing on arithmetic tasks (e.g., what is 147 x 38) than the average joe. I would hypothesize yes overall to solve each question; however, I'd also hypothesize the momentary max intensity of the expert to peak higher. Think of a bodybuilder vs. a SWE bench-pressing 100 lbs for 50 reps. The bodybuilder has way more muscle to devote to a single rep, and will likely finish the set in 20 seconds, while the SWE is going to take like 30 minutes ;)
I did a fMRI study as a volunteer in college.
It involved going to the lab and practicing the thing (a puzzle / maze) I would be shown during the actual MRI. I think I went in to “practice” a couple times before showing up and doing it in the machine.
IIRC the purpose of practicing was exactly that, to avoid me trying ti learn something during the scan (since that wasn’t the intention of the study).
In other words, I think you can control for that variable.
(Side note: I absolutely fell asleep during half the scan. Oops! I felt bad, but I guess that’s a risk when you recruit sleep deprived college kids!)
It's so much worse than this.
For task fMRI, the test-retest reliability is so poor it should probably be considered useless or bordering on pseudoscience, except for in some very limited cases like activation of the visual and/or auditory and/or motor cortex with certain kinds of clear stimuli. For resting-state fMRI (rs-fMRI), the reliabilities are a bit better, but also still generally extremely poor [1-3].
There are also two IMO major and devastating theoretical concerns re fMRI that IMO make the whole thing border on nonsense. One is the assumed relation between the BOLD signal and "activation", and two is the extremely horrible temporal resolution of fMRI.
It is typically assumed that the BOLD response (increased oxygen uptake) (1) corresponds to greater metabolic activity, and (2) increased metabolic activity corresponds to "activation" of those tissues. This trades dubiously on the meaning of "activation", often assuming "activation = excitatory", when we know in fact much metabolic activity is inhibitory. fMRI cannot distinguish between these things.
There are other deeper issues, in that it is not even clear to what extent the BOLD signal is from neurons at all (could be glia), and it is possible the BOLD signal must be interpreted differently in different brain regions, and that the usual analyses looking for a "spike" in BOLD activity are basically nonsense, since BOLD activity isn't even related to this at all, but rather the local field potential, instead. All this is reviewed in [4].
Re: temporal resolution, essentially, if you pay attention to what is going on in your mind, you know that a LOT of thought can happen in just 0.5 seconds (think of when you have a flash of insight that unifies a bunch of ideas). Or think of how quickly processing must be happening in order for us to process a movie or animation sequence where there are up to e.g. 10 cuts / shots within a single second. There is also just biological evidence that neurons take only milliseconds to spike, and that a sequence of spikes (well under 100ms) can convey meaningful information.
However, the lowest temporal resolutions (repetition times) in fMRI are only around 0.7 seconds. IMO this means that the ONLY way to analyze fMRI that makes sense is to see it as an emergent phenomenon that may be correlated with certain kinds of long-term activity reflecting cyclical BOLD patterns / low-frequency patterns of the BOLD response. I.e. rs-fMRI is the only fMRI that has ever made much sense a priori. The solution to this is maybe to combine EEG (extremely high temporal resolution, clear use in monitoring realtime brain changes like meditative states and in biofeedback training) with fMRI, as in e.g. [5]. But, it may still well be just the case fMRI remains mostly useless.
[1] Elliott, M. L., Knodt, A. R., Ireland, D., Morris, M. L., Poulton, R., Ramrakha, S., Sison, M. L., Moffitt, T. E., Caspi, A., & Hariri, A. R. (2020). What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis. Psychological Science, 31(7), 792–806. https://doi.org/10.1177/0956797620916786
[2] Herting, M. M., Gautam, P., Chen, Z., Mezher, A., & Vetter, N. C. (2018). Test-retest reliability of longitudinal task-based fMRI: Implications for developmental studies. Developmental Cognitive Neuroscience, 33, 17–26. https://doi.org/10.1016/j.dcn.2017.07.001
[3] Termenon, M., Jaillard, A., Delon-Martin, C., & Achard, S. (2016). Reliability of graph analysis of resting state fMRI using test-retest dataset from the Human Connectome Project. NeuroImage, 142, 172–187. https://doi.org/10.1016/j.neuroimage.2016.05.062
[4] Ekstrom, A. (2010). How and when the fMRI BOLD signal relates to underlying neural activity: The danger in dissociation. Brain Research Reviews, 62(2), 233–244. https://doi.org/10.1016/j.brainresrev.2009.12.004, https://scholar.google.ca/scholar?cluster=642045057386053841...
[5] Ahmad, R. F., Malik, A. S., Kamel, N., Reza, F., & Abdullah, J. M. (2016). Simultaneous EEG-fMRI for working memory of the human brain. Australasian Physical & Engineering Sciences in Medicine, 39(2), 363–378. https://doi.org/10.1007/s13246-016-0438-x
Re: temporal resolution
Even if neuronal activity is (obviously) faster, the (assumed) neuro-vascular coupling is slower. Typically there are several seconds till you get a BOLD response after a stimulus or task, and this has nothing to do with fMRI sampling rate (fNIRS can have much faster sampling rate, but the BOLD response it measures is equally slow, too). Think of it as that neuronal spiking happens in a range of up to some hundred milliseconds and the body changing the blood flow happens much slower than that.
The issue is that measuring the BOLD response, even in best case scenario, is a very very indirect measure of neuronal activity. This is typically lost when people referring to fMRI studies as discovering "mental representations" in the brain and other non-sense, but here we are. Criticising the validity of the BOLD response itself, though, is certainly interesting.
Right, my point is sort of that both the BOLD response and fMRI sampling rates are far too "slow" (not nearly approaching the Nyquist frequency, I guess) a priori to deeply investigate something as fast as cognition.
Depends on what you mean by cognition, but as you yourself said, BOLD may be correlated with certain kinds of long(er)-term activity, and that in itself is very useful if interpreted carefully. No one claims to detect single "thoughts" or anything of the sort, at least I haven't seen anything so shameless.
Well, a lot of task fMRI designs are pretty shameless and clearly haven't taken the temporal resolution issues seriously, at least when it comes to interpreting their findings in discussions (i.e. claiming that certain regions being involved must mean certain kind of cognition, e.g. "thoughts" must be involved too). And there have definitely been a few papers trying to show they can e.g. reconstruct the image ("thought") in a person's mind from the fMRI signal.
But I don't think we are really disagreeing on anything major here. I do think there is likely some useful potential locked away in carefully designed resting-state fMRI studies, probably especially for certain chronic and/or persistent systemic cognitive things like e.g. ADHD, autism, or, perhaps more fruitfully, it might just help with more basic understanding of things like sleep. But, I also won't be holding my breath for anything major coming out of fMRI anytime soon.
Biotech industrial complex
fMRI is a cool, expensive tech, like so many others in genetics and other diagnostics. These technologies create good jobs ("doing well by doing good").
But as other comments point out, and practitioners know, their usefulness for patients is more dubious.
The researchers found that “40% of increased fMRI signal correspond to a decrease in neuronal activity”, so it’s even worse than the headline.
It is the microbiome, stupid.
[just kidding]
If you actually read their paper, you will find that it's only the sign of the correlation that is being questioned. The field has generally been aware of this interpretational gap, and that's why two-sided hypothesis tests are important. Cellular neuroscience and electrophysiology are only starting to face the challenges that fMRI faced 2 decades ago.
To me this is like shitting on cars in 1925 because they kill people every now and then. Cars didn't go away, and nor will fMRI, until someone finds a better way to measure living people's brains.
TUM's press is being sloppy, from conflating fMRI with MRI to presuming this is revolutionary, and ignoring earlier empirical work against this narrative (Windkessel's, Logothetis beta/gamma coupling, etc.)
What's surprising is the desire to have a silver bullet, or not solution.
What's still amazing is fMRI can provide more visual context of what's happening in the brain, in what region, and activities that can help that improve.
There are other complementary technologies like QEEG and SPECT that can also shed a light as well.
It does seem the case that fMRI cann be more of a snapshot photo, and technologies like SPECT can provide more of a regional time lapse of activity.
you're telling me the results of this paper were likely bs? --- https://www.sciencedirect.com/science/article/abs/pii/S10538...
The point of the salmon paper is to demonstrate to people “if you do your stats wrong, you’re going to think noise is real” and not “fmri is bs”
As the first author on the salmon paper, yes, that was exactly our point. Researchers were capitalizing on chance in many cases as they failed to do effective corrections to the multiple comparisons problem. We argued with the dead fish that they should.
> We argued with the dead fish that they should.
Arguing with a dead fish may be a sign you're working too hard :)
Yeah, it did prove to be a rather one-sided conversation... ;)
Did you try tuning it? https://youtu.be/F2y92obnsc0
Nothing to add to this conversation in particular, but just wanted to say - truly amazing paper. Well done!
Many thanks! It was a ton of fun. Hard to beleive that we are coming up on 20 years since the data for the salmon was first collected...
Curious what you find to be "bs" about the results of this paper? That statistical corrections are necessary when analysing fMRI scans to prevent spurious "activations" that are only there by chance?
They were being sarcastic.
Oh man you stole my thunder. I hoped to be the first to bring up the dead salmon.
>which are known to produce predictable fMRI signal changes in distributed brain regions.
Wondering how they created that baseline. Was it with fMRI data (which has deviance from actual data, as pointed out)? Or was it through other means?
Why did TUM let this misleading headline front the news release? Dont we have enough issues with Academia? The result just mean BOLD is an imperfect proxy.
It is especially unforgiveable that the title of on the news release itself is about "40 percent of MRI signals". What, as in all MRI, not just fMRI? Hopefully an honest typo and not just resulting from ignorance.
let me write the correct title for you: "new evidence that fMRI data should be processed and interpreted only in the presence of an adult"
Can the OP change the HN item title so scrollers don't think there is a problem with MRI? Isn't fMRI being questioned?
Looks like a mod got to it!