Published online before print September 26, 2013, doi: 10.3174/ajnr.A3750
AJNR 2013 34: E114-E115

H.R. Pardoe^a
aNew York University School of Medicine
New York, New York

G.D. Jackson^b
bThe Florey Institute of Neuroscience and Mental Health
Heidelberg, Victoria, Australia

We read with interest “3T MRI Quantification of Hippocampal Volume and Signal in Mesial Temporal Lobe Epilepsy Improves Detection of Hippocampal Sclerosis,”1 in which Coan et al presented convincing evidence that quantitative assessment of hippocampal volume and T2 improves detection of hippocampal sclerosis vs visual inspection. Although we agree with the principal findings of the study, we disagree with the statement “whether manual or automatic analysis [of hippocampal volume] has higher sensitivity and specificity is still debatable” (text in square brackets added for clarity). We assert that current methods of automated hippocampal segmentation have poorer sensitivity and specificity than manual hippocampal segmentation for the detection of hippocampal sclerosis. To provide evidence supporting this assertion, we measured left hippocampal volumes in 22 patients with epilepsy with left-lateralized hippocampal sclerosis and 22 age-matched healthy control participants 1) manually and 2) automatically (using FreeSurfer version 5.0; http://surfer.nmr.mgh.harvard.edu), and compared the sensitivity and specificity of the 2 techniques. A subset of the MR imaging scans used for this analysis were used in a prior study.2

The sensitivity and specificity are both dependent on the hippocampal volume threshold used to classify participants. The commonly used method of measuring the area under the receiver operating characteristic (ROC) curve was used to assess which method (manual or automated) is a superior detector of hippocampal sclerosis. The ROC curves for manual and automated hippocampal segmentation are provided in the Figure. The area under the ROC curve for manual hippocampal segmentation is higher than automated hippocampal segmentation, indicating that manual segmentation is a superior method for the discrimination of hippocampal sclerosis (Table). Sensitivity and specificity are greater for manual segmentation than automated when each other’s measure is set at 0.95. In our opinion, the superiority of manual segmentation vs current automated methods, particularly in pathologic conditions such as hippocampal sclerosis, is widely accepted by the image-processing community and is not particularly controversial.

The most important caveat to attach to this analysis is that the discriminative ability of automated, manual, and visual-based methods is likely to improve as MR imaging acquisitions improve. Nevertheless, we believe that manual hippocampal segmentation would likely improve the ability of quantitative methods to detect hippocampal sclerosis above the 28% improvement presented by Coan et al.1

In summary, we have presented evidence that manual hippocampal segmentation has higher sensitivity and specificity and is a better detector of hippocampal sclerosis than current automated hippocampal segmentation methods.

References

1. Coan AC, Kubota B, Bergo FP, et al. 3T MRI quantification of hippocampal volume and signal in mesial temporal lobe epilepsy improves detection of hippocampal sclerosis. AJNR Am J Neuroradiol 2013 Jul 18. [Epub ahead of print] » Search Google Scholar
2. Pardoe HR, Pell GS, Abbott DF, et al. Hippocampal volume assessment in temporal lobe epilepsy: How good is automated segmentation? Epilepsia 2009;50:2586–92 » CrossRef » Medline

Reply

Published online before print September 26, 2013, doi: 10.3174/ajnr.A3751
AJNR 2013 34: E116

A.C. Coan^a and F. Cendes^a
aNeuroimaging Laboratory, Department of Neurology
State University of Campinas
Campinas, São Paulo, Brazil

Pardoe and Jackson have commented on our recent article1 in which we demonstrated that automated hippocampal volumetry in 3T MR images can improve the detection of signs of hippocampal sclerosis. They discuss in their letter that consistent data in the literature have demonstrated that manual hippocampal segmentation has a higher sensitivity and specificity than the current automated methods.2 Our study did not address this issue directly because we did not compare the results with manual volumetry. Nevertheless, we believe this is a very important point that needs further discussion.

Taking this issue on a purely conceptual basis, we believe it is logical to assume that automatic MR imaging segmentation will never be superior to evaluation by expert professionals for clinical diagnosis. Therefore, any automatic segmentation would be an additional tool for the imaging diagnosis and never a substitute. As for the sensitivity and specificity of manual vs automatic segmentation, the question is more complex, simply because it will depend on 3 main factors: 1) level of expertise of the professional drawing the boundaries of the hippocampi, 2) the program used for automatic segmentation, and 3) the quality of the MR imaging scan. Because these 3 factors vary significantly among centers, and also because of the constant improvement in software and hardware for imaging acquisition and postprocessing, it is hard to compare different studies.

Our group has extensive experience with manual hippocampal volumetry, with a personal contribution to the validation of the volumetry protocol that is still commonly used.3,4 An important earlier study showed that qualitative visual analysis correctly identified hippocampal sclerosis in most patients (41/44; 93%) and that hippocampal volumetry provided localization in an additional small number of patients (43/44; 97%).5 Since then, imaging scanners and protocols have improved significantly and, thus, the accuracy of visual analyses.

However, although it has been known for more than 20 years that manual hippocampal volumetry can improve visual analysis in the detection of hippocampal sclerosis, why is it still not widely used as a clinical tool? In our opinion, there are 2 main reasons for this: 1) manual volumetry is very time consuming and not practical for radiology clinics, and 2) it is operator dependent. For experienced operators, manual hippocampal volumetry shows very reliable results, but for those without considerable experience, the results vary and the interrater agreement may be very poor.

Another important point is the possible difference in accuracy between the manual vs the automated method to detect hippocampal abnormalities in severe vs subtle hippocampal sclerosis. Although visual analyses with an adequate MR imaging protocol correctly identify signs of hippocampal sclerosis in most patients,1,5 the accuracy of the volumetry tool should also be measured to detect subtle cases in the clinical context, where it would be most useful, and not necessarily in cases where one can detect easily gross asymmetries by visual analysis by using different (T1, T2, FLAIR) images with appropriate acquisition (high-resolution, thin coronal cuts perpendicular to the hippocampi). Clinicians need a validated practical tool for these difficult cases, not for those with clear-cut MRI finding of hippocampal sclerosis (we are not talking here about segmentation for research purposes). This issue has not yet been properly addressed.

Neuroimaging has transformed evaluation of epilepsies that are resistant to antiepileptic drugs. A relevant challenge is to be able to transform a negative MR image into an MR image with evidence of focal abnormality, which can substantially increase the odds of successful surgical treatment of seizures in a given patient. In our study, we demonstrated that hippocampal volumetry and signal quantification are still useful for increasing the detection of MR imaging–positive temporal lobe epilepsy, even in patients who had normal MR imaging results by visual analyses.1 We strongly believe that hippocampal volumetry should be used in the presurgical evaluation of patients with antiepileptic drug–resistant epilepsy. But for that, volumetry must be practical, not time consuming, and free of human bias. More efforts should be made to improve automatic techniques, including improved sensitivity and specificity, to make this technique clinically widely available.

References

1. Coan AC, Kubota B, Bergo FP, et al. 3T MRI quantification of hippocampal volume and signal in mesial temporal lobe epilepsy improves detection of hippocampal sclerosis. AJNR Am J Neuroradiol 2013 Jul 18 [Epub ahead of print] » Search Google Scholar
2. Pardoe HR, Pell GS, Abbott DF, et al. Hippocampal volume assessment in temporal lobe epilepsy: how good is automated segmentation? Epilepsia 2009;50:2586–92 » CrossRef » Medline
3. Watson C, Cendes F, Fuerst D, et al. Specificity of volumetric magnetic resonance imaging in detecting hippocampal sclerosis. Arch Neurol 1997;54:67–73 » CrossRef » Medline
4. Cendes F, Andermann F, Gloor P, et al. MRI volumetric measurement of amygdala and hippocampus in temporal lobe epilepsy. Neurology 1993;43:719–25 » CrossRef
5. Kuzniecky RI, Bilir E, Gilliam F, et al. Multimodality MRI in mesial temporal sclerosis: relative sensitivity and specificity. Neurology 1997;49:774–78 » CrossRef