|
 |
| ORIGINAL ARTICLE |
|
| Year : 2014 | Volume
: 55
| Issue : 2 | Page : 144-147 |
|
|
A comparison of Simplified Acute Physiology Score II, Acute Physiology and Chronic Health Evaluation II and Acute Physiology and Chronic Health Evaluation III scoring system in predicting mortality and length of stay at surgical intensive care unit
Mahryar Taghavi Gilani, Majid Razavi, Azadeh Mokhtari Azad
Department of Anesthesia, Cardiac Anesthesia Research Center, Imam-Reza Hospital, School of Medicine, Mashhad University of Medical Science, Mashhad, Iran
| Date of Web Publication | 31-Mar-2014 |
Correspondence Address:
Mahryar Taghavi Gilani
Department of Anesthesia, Cardiac Anesthesia Research Center, Imam-Reza Hospital, School of Medicine, Mashhad University of Medical Science, Mashhad
Iran
 Source of Support: None, Conflict of Interest: None  | Check |
DOI: 10.4103/0300-1652.129651
 Abstract | | |
Background: In critically ill patients, several scoring systems have been developed over the last three decades. The Acute Physiology and Chronic Health Evaluation (APACHE) and the Simplified Acute Physiology Score (SAPS) are the most widely used scoring systems in the intensive care unit (ICU). The aim of this study was to assess the prognostic accuracy of SAPS II and APACHE II and APACHE III scoring systems in predicting short-term hospital mortality of surgical ICU patients. Materials and Methods: Prospectively collected data from 202 patients admitted to Mashhad University Hospital postoperative ICU were analyzed. Calibration was estimated using the Hosmer-Lemeshow goodness-of-fit test. Discrimination was evaluated by using the receiver operating characteristic (ROC) curves and area under a ROC curve (AUC). Result: Two hundred and two patients admitted on post-surgical ICU were evaluated. The mean SAPS II, APACHE II, and APACHE III scores for survivors were found to be significantly lower than of non-survivors. The calibration was best for APACHE II score. Discrimination was excellent for APACHE II (AUC: 0.828) score and acceptable for APACHE III (AUC: 0.782) and SAPS II (AUC: 0.778) scores. Conclusion: APACHE II provided better discrimination than APACHE III and SAPS II calibration was good at APACHE II and poor at APACHE III and SAPS II. Use of APACHE II was excellent in this post-surgical ICU. Keywords: APACHE II, APACHE III, ICU mortality, SAPS
How to cite this article:
Gilani MT, Razavi M, Azad AM. A comparison of Simplified Acute Physiology Score II, Acute Physiology and Chronic Health Evaluation II and Acute Physiology and Chronic Health Evaluation III scoring system in predicting mortality and length of stay at surgical intensive care unit. Niger Med J 2014;55:144-7 |
How to cite this URL:
Gilani MT, Razavi M, Azad AM. A comparison of Simplified Acute Physiology Score II, Acute Physiology and Chronic Health Evaluation II and Acute Physiology and Chronic Health Evaluation III scoring system in predicting mortality and length of stay at surgical intensive care unit. Niger Med J [serial online] 2014 [cited 2023 Dec 8];55:144-7. Available from: https://www.nigeriamedj.com/text.asp?2014/55/2/144/129651 |
| Introduction | |  |
The prognostic and general severity scoring systems that are used in the intensive care unit (ICU) are beneficial in predicting risk of mortality. Mortality prediction is important for patient or family information and consent, comparison of ICU results, monitoring quality of ICU care and can be used to stratify patients for clinical research.
Several criteria should be taken into consideration when judging the value of any scoring system in clinical practice. Validity and reliability are important issues that allow confident use of a scoring system in ICU patients with different disease and baseline characteristics.
In critically ill patients, several scoring systems have been developed over the last three decades. Acute Physiology and Chronic Health Evaluation (APACHE) II and III scores were developed by Knause et al. [1] , in 1985 and 1991, respectively, and Simplified Acute Physiology Score (SAPS) II was developed by Le Gall et al. [2] , in 1993. These are the most widely used scoring system in the ICUs. Nevertheless, there are still conflicting data concerning which of them is the best predictor tool.
The aim of this study was, therefore, to compare and evaluate the performance of APACHE II, APACHE III and SAPS II as scores in predicting the mortality and morbidity of surgical ICU patients.
| Materials and Methods | | |
This prospective study included 202 consecutive patients admitted to the surgical ICU of university hospital of Imam Reza-Mashhad-Iran, during the 6 months, from April 2010 through September 2010.
For the purpose of the study, each admission (elective or urgent) was considered as one patient. Patients with ICU Length of Study (LOS) less than 24 hours were excluded from the analysis as SAPS II and APACHE II and III cannot be calculated in these patients.
To calculate the APACHE II score, twelve common physiological and laboratory values are marked and calculated with APACHE II software. The sum of these values is added to a mark adjusting for chronic health problems (severe organ insufficiency or immune-compromised patients) and a mark adjusting for patient age to achieve the APPACHE II score.
APACHE III scores are derived from marks for the extent of abnormality of 17 physiological measurements, adjusts for seven comorbidities that reduce immune function and influence hospital survival, and adjusts for age, and range from 0 to 299.
Clinical and laboratory data necessary for the SAPS II and APACHE II and III systems were recorded on the first day of admission for all patients. Physiological data were recorded 3-hourly during the first day. The calculation of APACHE II and III and SAPS II scores was based on the worst values taken during the first 24 hours after admission.
Statistical analysis
Analysis of Data and Results was done with SPSS V.18 software. Individual relationship of each score (SAPS II, APACHE III and II) and length of admission to the risk of death and comparison of score was assessed by t-test and ANOVA, P-value less than 0.05 was significant statistically. Discrimination was tested using the receiver operating characteristic (ROC) curves and by comparing areas under the curve (AUC). AUCs more than 0.8 were excellent and 0.6-0.8 were acceptable.
The calibration of the systems (prognostic accuracy at different levels of risk) was studied using Youden index and Hosmer-Lemeshow goodness of fit statistics which divides subjects into deciles based on predicted probabilities of death and then computes a Chi-square from observed and expected frequencies. Lower Chi-square values and higher P values (P > 0.5) are associated with a better fit. For the different scoring systems tested, the sensitivity, specificity, positive and negative predictive values were calculated, and the cutoff point giving the best Youden index was determined. This cutoff point was also used to calculate the predicted and observed outcome for patients.
| Results | | |
During the study period, 202 patients were admitted to the ICU which 118 (58.8%) were men and 84 (41.8%) were women. The mean age was 53.1 ± 20.3 years (range 14-85 years). Elective surgery was performed before admission to the ICU in 195 patients and emergency surgery in seven patients.
[Table 1] reports predictive values of the various scoring systems calculated at the cutoff point giving the best Youden index, sensitivity, specificity, positive and negative predictive value and overall success rate.
The mean ±SD SAPS II, APACHE II and APACHEIII score, calculated within 24 h of admission to the ICU, were 13.42 ± 6.65, 18.56 ± 7.32 and 23.66 ± 11.50, respectively [Table 2]. [Table 3] shows relationship of mortality with scores and there were significant differences in SAPS II score, APACHE II score and APACHE III score between survivors and non-survivors (P < 0.001 at all). | Table 3: Comparison of three scoring systems with survivor and non-survivor Mean (st deviation)
Click here to view |
[Table 4] shows that admission duration correlated with SAPS II, APACHE II and III scores and length of admission in ICU increased significantly with higher SAPS II, APACHE II and APACHE III scores (P = 0.035, 0.017 and 0.049, respectively). | Table 4: Comparison of three scoring systems with length of admission (days). Mean (st deviation)
Click here to view |
Calibration measured with Hosmer-Lemeshow goodness-of-fit tests are shown in [Table 5]. The Hosmer-Lemeshow statistic was best for APACHE II score (P = 0.71). However, for the APACHE III and SAPS II scores, calibration was poor (P value = 0.392 and 0.379, respectively).
Discrimination power evaluated with ROC curve and area under curve (AUC). ROC curves are shown in [Figure 1]. AUC of APACHE II was 0.828 and excellent, while that of APACHE III (0.782) and SAPS II (0.778) was acceptable. | Figure 1: ROC curves for SAPS II, APACHE II and PACHE III scoring systems
Click here to view |
 | Table 5: Hosmer-lemeshow goodness of fit tests for three scoring systems
Click here to view |
| Discussion | | |
The performance of the prognostic models is evaluated by tow objective measures: Calibration and discrimination. Calibration refers to how closely the estimated probabilities of mortality correlate with the observed mortality over the entire range of probabilities and can be tested using Hosmer-Lemeshow goodness-of-fit statistic. Discrimination refers to the ability of a prognostic score to classify patients correctly as survivors or non-survivors and is measured by AUC. From the individual patient's point of view, it would be interesting to have perfect discrimination; however, for clinical trials or comparison of care between ICUs better calibration is needed.
In our study, the discriminative ability of APPACHE II is excellent. Moreover, it has greater discriminative power than APACHE III or SAPS II in our critically ill patient. APACHE II also has a better, more appropriate calibration than APACHE III or SAPS II, so only APACHE II properly predicts mortality risk in our ICU.
Although ICU admission policies generally are unknown, they probably also influence outcome. The APACHE model differs in risk assessment of medical or surgical patients. Nevertheless, APACHE II prediction has been more consistent across a wide range of mortality risks than APACHE III or SAPS II. [2],[3]
Our results are in agreement with other reports on the performance of the APACHE scoring system in UK. [4],[5],[6] The same pattern was observed in the external validation of the SAPS II, APACHE II and APACHE III models in Scottish intensive care patients. [7] One study reported good calibration for the APACHE II model, but again imperfect calibration for the two other score tested. [8],[9]
In one study, Beck and colleagues validated the SAPS II and APACHE II and III prognostic models in 16,646 adult intensive care patients in Southern UK. The external validation showed a similar pattern for all three models tested: Good discrimination, but imperfect calibration. [10] Differences in the performance of scoring systems reinforce the need to validate them using data of independent samples from different ICUs in different countries, due to variation in case mix, structure and organization of acute medical care, lifestyles and genetic makeup between populations. [11] Adequate discrimination by APACHE II previously has been described with an AUROC of 0.91 in Thailand, 0.88 in Hong Kong, 0.83 in Greece and Saudi Arabia and 0.79 in Portugal. [12] Its calibration, however, always has been poor, as evidenced by recent studies, primarily due to differences in case mix, data collection and lead-time bias. [2],[12]
The present study has some limitations. First, as a single-centre study, there may be bias with regard to case mix, quality of ICU care and ICU policy. Second, our relatively small sample size is a limiting factor in stratified analysis of calibration. Third, APACHE II is based on retrospective data that is available within 24 h of ICU admission; consequently, the sampling rate that is used can influence mortality estimation. A multi-centre study would mitigate the concerns over case mix and benefit from a larger sample size.
| Conclusions | | |
We found a better calibration of APACHE II than APACHE III or SAPS II such that APACHE II improves the ability to predict hospital mortality in comparison with APACHE III or SAPS II. The discrimination of APACHE II is excellent, but of APACHE III and SAPS II is acceptable.
| References | | |
| 1. | Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APPACHE II: A severity of disease classification system. Crit Care Med 1985;13:818-29. 
[PUBMED] |
| 2. | Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Phusiology Score (SAPS II) based on a European/North American Multicenter study. JAMA 1993;270:2957-63.
|
| 3. | Markgraf R, Deutschinoff G, Pientka L, Scholten T. Comparison of Acute Physiology and Chronic Health Evaluations II and III and Simplified acute physiology Score II: A prospective cohort study evaluating these method to predict outcome in a German interdisciplinary intensive care unit. Crit Care Med 2000;28:26-33.
|
| 4. | Rowan KM, Kerr JH, Major E, McPhersonk, Short A, Vessey MP. Intensive Care Society's APACHE II study in Britain and Ireland II: Outcome comparison of intensive care units after adjustment for case mix by the American APACHE II method. BMJ 1993;307:977-810.
|
| 5. | Pappachan JV, Millar B, Bennett ED, Smith GB. Comparison of outcome from intensive care admission after adjustment for case mix by the APACHE III prognostic system. Chest 1999;115:802-10.
|
| 6. | Apolone G, Bertolini G, D'Amico R, Iapichino G, Cattaneo A, De Salvo G, et al. The performance of SAPS II in a cohort of patients admitted to 99 Italian ICUs. Intensive Care Med 1996;22:1368-78.
|
| 7. | Livingston BM, MaKirdy FN, Howic JC, Jones R, Norrie JD. Assessment of the performance of five intensive care scoring models within a larger Scottish database. Crit Care Med 2000;28:1820-70.
|
| 8. | Markgraf R, Deutschinoff G, Pientka L, Scholten T. camparison of APACHE III and SAPS II in a German interdisciplinary ICU. Crit Care Med 2001;28:33-5.
|
| 9. | Bosman RJ, Oudemane Van Straaten HM, Zandstra DF. The use of intensive care information systems alters outcome prediction. Intensive Care Med 1998;24:953-8.
|
| 10. | Beck DH, Smith GB, Pappachan JV, Millar B. External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: A multicenter study. Intensive Care Med 2003;29:249-56.
|
| 11. | Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAP 3 Investigators. SAPS 3- from evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 2005;31:1345-55.
|
| 12. | Khwannimit B, Geater A. A comparison of APACHE II and SAPS II Scoring systems in predicting hospital mortality in Thai and intensive care units. J Med Assoc Thai 2007;90:643-52.
|
[Figure 1]
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5]
| This article has been cited by | | 1 |
Association between neutrophil percentage-to-albumin ratio and 28-day mortality in Chinese patients with sepsis |
|
| Chunying Hu, Yinyan He, Jinfeng Li, Cuicui Zhang, Qinghe Hu, Wei Li, Cuiping Hao | | Journal of International Medical Research. 2023; 51(6) | | [Pubmed] | [DOI] | | | 2 |
Internal Validation of the Predictive Performance of Models Based on Three ED and ICU Scoring Systems to Predict Inhospital Mortality for Intensive Care Patients Referred from the Emergency Department |
|
| Zahra Rahmatinejad, Benyamin Hoseini, Fatemeh Rahmatinejad, Ameen Abu-Hanna, Robert Bergquist, Ali Pourmand, MirMohammad Miri, Saeid Eslami, Yeong Shiong Chiew | | BioMed Research International. 2022; 2022: 1 | | [Pubmed] | [DOI] | | | 3 |
Neutrophil-to-Albumin Ratio as A Novel Predictor of Mortality in Patients with Sepsis |
|
| Naser Gharebaghi, Mohammad Amin Valizade Hasanloei, Mohammad Fromandi, Mohammad Reza Pashaei | | Avicenna Journal of Clinical Medicine. 2022; 29(1): 12 | | [Pubmed] | [DOI] | | | 4 |
What do we know about experiencing end-of-life in burn intensive care units? A scoping review |
|
| André Filipe Ribeiro, Sandra Martins Pereira, Rui Nunes, Pablo Hernández-Marrero | | Palliative and Supportive Care. 2022; : 1 | | [Pubmed] | [DOI] | | | 5 |
A machine learning approach to predict healthcare-associated infections at intensive care unit admission: findings from the SPIN-UTI project |
|
| M. Barchitta,A. Maugeri,G. Favara,P.M. Riela,G. Gallo,I. Mura,A. Agodi | | Journal of Hospital Infection. 2021; 112: 77 | | [Pubmed] | [DOI] | | | 6 |
Early Prediction of Seven-Day Mortality in Intensive Care Unit Using a Machine Learning Model: Results from the SPIN-UTI Project |
|
| Martina Barchitta,Andrea Maugeri,Giuliana Favara,Paolo Riela,Giovanni Gallo,Ida Mura,Antonella Agodi | | Journal of Clinical Medicine. 2021; 10(5): 992 | | [Pubmed] | [DOI] | | | 7 |
A PROSPECTIVE STUDY TO DETERMINE POST
OPERATIE MORBIDITY AND MORTALITY
USING SURGICAL APGAR SCORE |
|
| Suraj Girish,P S Saravanan,K Suresh babu | | PARIPEX INDIAN JOURNAL OF RESEARCH. 2021; : 1 | | [Pubmed] | [DOI] | | | 8 |
Prognostic Value of Blood Urea Nitrogen/Creatinine Ratio for Septic Shock: An Analysis of the MIMIC-III Clinical Database |
|
| Didi Han,Luming Zhang,Shuai Zheng,Fengshuo Xu,Chengzhuo Li,Rui Yang,Wen Ma,Haiyan Yin,Jun Lyu,Despina Deligianni | | BioMed Research International. 2021; 2021: 1 | | [Pubmed] | [DOI] | | | 9 |
Association between lactate/albumin ratio and all-cause mortality in patients with acute respiratory failure: A retrospective analysis |
|
| Yan Lu,Haoyang Guo,Xuya Chen,Qiaohong Zhang,Aleksandar R. Zivkovic | | PLOS ONE. 2021; 16(8): e0255744 | | [Pubmed] | [DOI] | | | 10 |
Prognostic Value of Procalcitonin, C-Reactive Protein, and Lactate Levels in Emergency Evaluation of Cancer Patients with Suspected Infection |
|
| Patrick Chaftari,Aiham Qdaisat,Anne-Marie Chaftari,Julian Maamari,Ziyi Li,Florea Lupu,Issam Raad,Ray Hachem,George Calin,Sai-Ching Jim Yeung | | Cancers. 2021; 13(16): 4087 | | [Pubmed] | [DOI] | | | 11 |
Comparison of APACHE II scores and mortality with CRP/albumin, neutrophil/lymphocyte and thrombocyte/lymphocyte ratios in patients admitted to internal medicine and anesthesia reanimation intensive care unit |
|
| Ahmet Usta, Emin Gemcioglu, Salih Baser, Osman Ersoy, Yunus Halil Polat, Abdülsamet Erden, Seval Izdes | | Journal of Laboratory Medicine. 2021; 0(0) | | [Pubmed] | [DOI] | | | 12 |
Assessment of severity of acute pancreatitis over time |
|
| Florina-Alexandra Cofaru,Silvia Nica,Carmen Fierbin?eanu-Braticevici | | Romanian Journal of Internal Medicine. 2020; 58(2): 47 | | [Pubmed] | [DOI] | | | 13 |
Performance of three prognostic models in critically ill patients with cancer: a prospective study |
|
| Frank D. Martos-Benítez,Hilev Larrondo-Muguercia,David León-Pérez,Juan C. Rivero-López,Versis Orama-Requejo,Jorge L. Martínez-Alfonso | | International Journal of Clinical Oncology. 2020; | | [Pubmed] | [DOI] | | | 14 |
What factors predict length of stay in the intensive care unit? Systematic review and meta-analysis |
|
| Igor Tona Peres,Silvio Hamacher,Fernando Luiz Cyrino Oliveira,Antônio Márcio Tavares Thomé,Fernando Augusto Bozza | | Journal of Critical Care. 2020; 60: 183 | | [Pubmed] | [DOI] | | | 15 |
Predicting hospital mortality for intensive care unit patients: Time-series analysis |
|
| Aya Awad,Mohamed Bader-El-Den,James McNicholas,Jim Briggs,Yasser El-Sonbaty | | Health Informatics Journal. 2020; 26(2): 1043 | | [Pubmed] | [DOI] | | | 16 |
Validation of APACHE II, APACHE III and SAPS II scores in in-hospital and one year mortality prediction in a mixed intensive care unit in Poland: a cohort study |
|
| Szymon Czajka,Katarzyna Ziebinska,Konstanty Marczenko,Barbara Posmyk,Anna J. Szczepanska,Lukasz J. Krzych | | BMC Anesthesiology. 2020; 20(1) | | [Pubmed] | [DOI] | | | 17 |
Critically ill patientsæ experiences of nursing care in the intensive care unit |
|
| Leah C Kisorio,Gayle C Langley | | Nursing in Critical Care. 2019; | | [Pubmed] | [DOI] | | | 18 |
The Pros and Cons of the Prediction Game: The Never-ending Debate of Mortality in the Intensive Care Unit |
|
| Piotr A. Fuchs,Iwona J. Czech,Lukasz J. Krzych | | International Journal of Environmental Research and Public Health. 2019; 16(18): 3394 | | [Pubmed] | [DOI] | | | 19 |
New model averaging approach in predicting mortality rate of intensive care unit patients |
|
| Siti Aisyah Mohd Padzil,Khuneswari Gopal Pillay,Mohd Saifullah Rusiman,Rohayu Mohd Salleh | | Journal of Physics: Conference Series. 2019; 1366: 012123 | | [Pubmed] | [DOI] | | | 20 |
Escala APACHE II para pacientes críticos con cáncer sólido. Estudio de reclasificación |
|
| F.D. Martos-Benítez,I. Cordero-Escobar,A. Soto-García,I. Betancourt-Plaza,I. González-Martínez | | Revista Española de Anestesiología y Reanimación. 2018; | | [Pubmed] | [DOI] | | | 21 |
APACHE II score for critically ill patients with a solid tumor: A reclassification study |
|
| F.D. Martos-Benítez,I. Cordero-Escobar,A. Soto-García,I. Betancourt-Plaza,I. González-Martínez | | Revista Española de Anestesiología y Reanimación (English Edition). 2018; | | [Pubmed] | [DOI] | | | 22 |
Outcome and prognostic factors of patients treated in the intensive care unit for carbon monoxide poisoning |
|
| Wei-Chih Liao,Wen-Chien Cheng,Biing-Ru Wu,Wei-Chun Chen,Chih-Yu Chen,Chia-Hung Chen,Chih-Yen Tu,Te-Chun Hsia | | Journal of the Formosan Medical Association. 2018; | | [Pubmed] | [DOI] | | | 23 |
Performance of critical care prognostic scoring systems in low and middle-income countries: a systematic review |
|
| Rashan Haniffa,Ilhaam Isaam,A. Pubudu De Silva,Arjen M. Dondorp,Nicolette F. De Keizer | | Critical Care. 2018; 22(1) | | [Pubmed] | [DOI] | | | 24 |
Development of a national core dataset for the Iranian ICU patients outcome prediction: a comprehensive approach |
|
| Alireza Atashi,Leila Ahmadian,Zahra Rahmatinezhad,Mirmohammad Miri,Najmeh Nazeri,Saeid Eslami | | BMJ Health & Care Informatics. 2018; 25(2): 71 | | [Pubmed] | [DOI] | | | 25 |
Patient length of stay and mortality prediction: A survey |
|
| Aya Awad,Mohamed Bader–El–Den,James McNicholas | | Health Services Management Research. 2017; 30(2): 105 | | [Pubmed] | [DOI] | | | 26 |
The Ability of the Acute Physiology and Chronic Health Evaluation (APACHE) IV Score to Predict Mortality in a Single Tertiary Hospital |
|
| Jae Woo Choi,Young Sun Park,Young Seok Lee,Yeon Hee Park,Chaeuk Chung,Dong Il Park,In Sun Kwon,Ju Sang Lee,Na Eun Min,Jeong Eun Park,Sang Hoon Yoo,Gyu Rak Chon,Young Hoon Sul,Jae Young Moon | | The Korean Journal of Critical Care Medicine. 2017; 32(3): 275 | | [Pubmed] | [DOI] | | | 27 |
Early Hospital Mortality Prediction of Intensive Care Unit Patients Using an Ensemble Learning Approach |
|
| Aya Awad,Mohamed Bader-El-Den,James McNicholas,Jim Briggs | | International Journal of Medical Informatics. 2017; | | [Pubmed] | [DOI] | | | 28 |
Clinical study of a new Modified Early Warning System scoring system for rapidly evaluating shock in adults |
|
| Qin Qin,Yiqin Xia,Yu Cao | | Journal of Critical Care. 2017; 37: 50 | | [Pubmed] | [DOI] | | | 29 |
Comparison of Risk Scoring Systems to Predict the Outcome in ASA-PS V Patients Undergoing Surgery |
|
| Derya Arslan Yurtlu,Murat Aksun,Pnar Ayvat,Nagihan Karahan,Lale Koroglu,Gülcin Önder Aran | | Medicine. 2016; 95(13): e3238 | | [Pubmed] | [DOI] | | | 30 |
Performance of Three Prognostic Models in Patients with Cancer in Need of Intensive Care in a Medical Center in China |
|
| XueZhong Xing,Yong Gao,HaiJun Wang,ChuLin Huang,ShiNing Qu,Hao Zhang,Hao Wang,KeLin Sun,Olga Y Gorlova | | PLOS ONE. 2015; 10(6): e0131329 | | [Pubmed] | [DOI] | |
|
 |
|
|
|
