P044 Uncertainty-Calibrated Network Initialization via Pretraining with Random Noise
Jeonghwan Cheon*1, Se-Bum Paik1
1Department of Brain and Cognitive Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
*Email: jeonghwan518@kaist.ac.kr
Uncertainty calibration — the ability to estimate predictive confidence that reflects the actual accuracy — is essential to real-world decision-making. Human cognition involves metacognitive processes, allowing us to assess uncertainty and distinguish between what we know and what we do not know. In contrast, current machine learning models often struggle to properly calibrate their confidence, even though they have achieved high accuracy in various task domains [1]. This miscalibration presents a significant challenge in real-world applications, such as autonomous driving or medical diagnosis, where incorrect decisions can have critical consequences. Although post-processing techniques have been used to address calibration issues, they require additional computational steps to obtain reliable confidence estimates. In this study, we show that random initialization — a common practice in deep learning — is a fundamental cause of miscalibration. We found that randomly initialized, untrained networks exhibit excessively high confidence despite lacking meaningful knowledge. This miscalibration at the initial stage prevents the alignment of confidence with actual accuracy as the network learns from data. To address this issue, we draw inspiration from the developmental brain, which is initialized through spontaneous neural activity even before receiving sensory inputs [2]. By mimicking this process, we pretrain neural networks with random noise [3] and demonstrate that this simple approach resolves the overconfidence issue, bringing initial confidence levels to near chance. This pre-calibration through random noise pretraining enables optimal calibration by aligning confidence levels with actual accuracy during subsequent data training. As a result, networks pretrained with random noise achieve significantly lower calibration errors compared to those trained solely with data. We also confirmed that this method generalizes well across different conditions, regardless of dataset size or network complexity. Notably, these pre-calibrated networks consistently identify “unknown data” by showing low confidence for outlier inputs. Our findings present a key solution for calibrating uncertainty in both in-distribution and out-of-distribution scenarios without the need for post-processing. This provides a fundamental approach to addressing miscalibration issues in artificial intelligence and may offer insights into the biological development of metacognition.
Acknowledgements
This work was supported by the National Research Foundation of Korea (NRF) grants (NRF-2022R1A2C3008991 to S.P.) and by the Singularity Professor Research Project of KAIST (to S.P.).
References
● Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. InInternational Conference on Machine Learning(pp. 1321-1330). PMLR.
● Martini, F. J., Guillamón-Vivancos, T., Moreno-Juan, V., Valdeolmillos, M., & López-Bendito, G. (2021). Spontaneous activity in developing thalamic and cortical sensory networks.Neuron,109(16), 2519-2534.
● Cheon, J., Lee, S. W., & Paik, S. B. (2024). Pretraining with random noise for fast and robust learning without weight transport.Advances in Neural Information Processing Systems,37, 13748-13768.
