모델 폐기 시점 및 기준

모델 폐기는 단순히 성능 저하를 넘어, 비즈니스 가치 유지, 운영 효율성, 규제 준수, 기술 진부화 등을 종합적으로 고려하여 결정해야 합니다. 아래에 대표적인 모델 폐기(퇴출) 기준을 정리합니다.

1. 성능 모니터링 기반 기준

• 지속적 성능 저하:
• 정확도·정밀도·재현율 등 핵심 지표가 사전에 정의한 임계치 이하로 떨어지고, 일정 기간(예: 7일, 30일) 동안 회복되지 않는 경우[1].
• 드리프트(Drift) 탐지:
• 입력 데이터 분포 변화(데이터 드리프트)나 입력-타겟 관계 변화(컨셉 드리프트)를 식별하는 통계 검정(KS 테스트, PSI 등)에서 설정한 임계치를 초과하면 즉시 경고 및 폐기 검토[1].

2. 비즈니스 영향 및 비용 분석

• 비용 대비 이득(Cost-Benefit) 불균형:
• 모델 유지·재학습 비용이 모델 활용으로 얻는 비즈니스 가치(수익 증대, 비용 절감 등)보다 클 때[2].
• ‘모델 노후화 비용(staleness cost)’과 ‘재학습 비용(retraining cost)’을 비교하여, 재학습 없이 유지할 때의 성능 손실이 더 클 경우 폐기 또는 재학습 결정[2].
• 비즈니스 KPI 저하:
• 모델 예측 결과가 사용자 전환율, 매출, 운영 효율성 등 핵심 성과 지표에 미치는 영향이 기준 이하로 하락하면 폐기 대상이 됨.

3. 기술·운영 환경 변화

• 신규 모델 대체:
• 더 높은 성능의 신규 알고리즘·아키텍처가 도입되어 운영 중인 버전 대비 우월할 때, 단계적 마이그레이션 후 기존 모델 폐기[3].
• 플랫폼·라이브러리 지원 종료:
• 사용 중인 프레임워크나 라이브러리가 더 이상 보안 패치·기능 업데이트를 제공하지 않는 경우, 해당 버전 모델을 안전한 버전으로 전환 후 폐기[3].

4. 규제·컴플라이언스 요건

• 데이터·보안 정책 변화:
• 개인정보 보호법(GDPR·CCPA 등) 또는 업계 가이드라인이 변경되어, 모델의 데이터 활용 방식이 불법·비윤리적으로 판단될 때 즉시 폐기 및 재설계 필요[3].
• 감사(audit)·거버넌스 요건:
• 내부·외부 감사에서 리스크·컴플라이언스 위반 사항이 보고된 모델은 운영 중단 후 폐기 검토[3].

5. 벤더 또는 오픈소스 정책

• 라이프사이클(Deprecation & Retirement) 일정:
• 클라우드 서비스(예: Azure OpenAI, Databricks Foundation Models)에서 제공하는 모델 지원 종료(Deprecation) 및 폐기(Retirement) 일정에 따라, 안내된 유예 기간 이후 자동 폐기[4][5].

위 기준들은 단일 지표가 아닌 다양한 관점(성능, 비용·가치, 기술, 규제, 공급자 정책)을 통합하여 종합적으로 평가해야 하며, 폐기 결정 전에는 “Graceful Model Retirement” 관점에서 사전 안내·이전 계획·이력 보존을 수행해야 합니다[3].

출처
[1] Monitoring and Maintaining Model Performance Over Time – LinkedIn https://www.linkedin.com/pulse/evaluating-drift-monitoring-maintaining-model-over-time-brown-ph-d–jvlae
[2] Cost-aware retraining for machine learning – ScienceDirect.com https://www.sciencedirect.com/science/article/pii/S0950705124002454
[3] Pro tips to avoid five critical pitfalls in MLOps – Nagarro https://www.nagarro.com/en/blog/pro-tips-avoid-five-critical-pitfalls-mlops
[4] MLOps and Data Drift Detection: Ensuring Accurate ML Model … https://dataheroes.ai/blog/mlops-and-data-drift-detection-ensuring-accurate-ml-model-performance/
[5] Generative AI models maintenance policy – Databricks Documentation https://docs.databricks.com/aws/en/machine-learning/retired-models-policy
[6] [PDF] scoring and prediction of early retirement using machine learning … https://www.actuarios.org/wp-content/uploads/2019/12/Art6-Anales2019.pdf
[7] APPENDIX FOR ONLINE PUBLICATION https://www.aeaweb.org/content/file?id=2622
[8] The Evolution of MLOps for https://www.cognilytica.com/wp-content/uploads/2020/11/PacteraEDGE_MLOps_Evolution.pdf
[9] Three Models of Retirement: Computational Complexity versus Predictive Validity https://www.nber.org/system/files/chapters/c7097/c7097.pdf
[10] ML Models and Lifecycle: A Starter Guide – LinkedIn https://www.linkedin.com/pulse/ml-models-lifecycle-starter-guide-chathuranga-bandara-abeyarathna-50pzf
[11] Can Machine Learning Algorithms Completely Run Your Pension … https://www.linkedin.com/pulse/can-machine-learning-algorithms-completely-run-your-pension-glah-v5aue
[12] RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budget https://arxiv.org/pdf/2505.24149v1.pdf
[13] [PDF] 10 Good Practice Principles for Retirement Phase Modelling in … https://www.actuaries.asn.au/Library/Events/FSF/2016/HenningtonLangtonRetirement.pdf
[14] Transition to retirement https://www.ato.gov.au/individuals-and-families/jobs-and-employment-types/working-as-an-employee/leaving-the-workforce/transition-to-retirement
[15] MLOps explained end-to-end | Theory https://campus.datacamp.com/courses/mlops-for-business/the-mlops-life-cycle?ex=1
[16] Good Practice Principles: Superannuation & Retirement Models https://www.linkedin.com/pulse/good-practice-principles-superannuation-retirement-jim-hennington
[17] Retirement Concepts as Predictors of Preretirement Consideration and Planning https://journals.sagepub.com/doi/pdf/10.1177/19394225241280099?download=true
[18] Navigating MLOps: Insights into Maturity, https://arxiv.org/pdf/2503.15577.pdf
[19] Best Practices for Product Retirement – Number Analytics https://www.numberanalytics.com/blog/best-practices-product-retirement
[20] Retirement Concepts as Predictors of Preretirement Consideration and Planning – Viera Bačová, Michal Kohút, Peter Halama, 2024 https://journals.sagepub.com/doi/10.1177/19394225241280099?icid=int.sj-full-text.citing-articles.1
[21] 91% of ML Models degrade in time | MIT Paper Review – NannyML https://www.nannyml.com/blog/91-of-ml-perfomance-degrade-in-time
[22] Predicting Patient Mortality for Earlier Palliative Care Identification in … https://pmc.ncbi.nlm.nih.gov/articles/PMC11041411/
[23] What Is Model Drift? | IBM https://www.ibm.com/think/topics/model-drift
[24] Why Do Machine Learning Models Die In Silence? – KDnuggets https://www.kdnuggets.com/2022/01/machine-learning-models-die-silence.html
[25] Machine learning models for 180-day mortality prediction of patients … https://pmc.ncbi.nlm.nih.gov/articles/PMC9992030/
[26] [Model Drift] Model Drift에 대한 A to Z # 2. Detection 방법과 Handling … https://calmmimiforest.tistory.com/120
[27] 💡⚡️ When AI Models Expire #ML #Technology #Retraining #Adaptability #Performance #Updates Part 1 https://www.youtube.com/watch?v=dP5lZQYI8V4
[28] Understanding MLops Lifecycle: From Data to Deployment https://www.projectpro.io/article/mlops-lifecycle/885
[29] [PDF] AN OVERVIEW OF – MLOps – VIANOPS https://vianops.ai/wp-content/uploads/2023/03/VIANOPS_Whitepaper-3-16.pdf
[30] Improving palliative and end-of-life care with machine learning and routine a rapid review – PubMed https://pubmed.ncbi.nlm.nih.gov/32002512/
[31] A Comprehensive Guide to Understanding and Implementing MLOPs https://www.digital-alpha.com/a-comprehensive-guide-to-understanding-and-implementing-mlops/
[32] Use of Machine Learning to Optimize Referral for Early Palliative Care https://ascopubs.org/doi/10.1200/JCO.24.00024
[33] Artificial Intelligence Decision Support Tools for End-of-Life Care … https://www.ncbi.nlm.nih.gov/books/NBK599854/
[34] MLOps Maturity Model · Azure ML-Ops (Accelerator) https://microsoft.github.io/azureml-ops-accelerator/1-MLOpsFoundation/1-MLOpsOverview/2-MLOpsMaturityModel.html
[35] Developing Performance Standards – OPM https://www.opm.gov/policy-data-oversight/performance-management/performance-management-cycle/planning/developing-performance-standards/
[36] Summary https://www.ebri.org/retirement/retirement-security-projection-model/content/the-ebri-retirement-readiness-rating-retirement-income-preparation-and-future-prospects-4593
[37] 2.5 Vesting conditions for stock-based compensation awards https://viewpoint.pwc.com/dt/us/en/pwc/accounting_guides/stockbased_compensat/stockbased_compensat__3_US/chapter_2_measuremen_US/25_vesting_condition_US.html
[38] Institute of Actuaries of Australia https://www.actuaries.asn.au/Library/Events/FSF/2018/NickCallilPaper.pdf
[39] Optimal retirement age, leisure and consumption – ScienceDirect https://www.sciencedirect.com/science/article/abs/pii/S0264999314003290
[40] Measuring the Adequacy of Retirement Income: A Primer https://www.cbo.gov/system/files/115th-congress-2017-2018/reports/53191-retirementadequacy.pdf
[41] [PDF] Good Practice Principles for Retirement Modelling Exposure Draft https://www.actuaries.asn.au/Library/Standards/SuperannuationEmployeeBenefits/2023/TPAUGUST2023.pdf
[42] Key Performance Indicators in Gauging the Health of Your Retirement Plan https://annuity.com/retirement-planning/key-performance-indicators-in-gauging-the-health-of-your-retirement-plan/
[43] Why drift in machine learning can deteriorate your model’s … – Agilytic https://www.agilytic.com/blog/why-drift-in-machine-learning-can-deteriorate-your-model-s-performance-and-what-you-can-do-about-it
[44] Model Drift: Best Practices to Improve ML Model Performance – Encord https://encord.com/blog/model-drift-best-practices/
[45] How to Automate Data Drift Thresholding in Machine Learning https://www.deepchecks.com/how-to-automate-data-drift-thresholding-in-machine-learning/
[46] D06. Monitoring and Feedback Loop – Deep Learning Bible – 위키독스 https://wikidocs.net/185358
[47] Automatic retraining for machine learning models – Building Nubank https://building.nubank.com/automatic-retraining-for-machine-learning-models/
[48] Evolving Strategies in Machine Learning: A Systematic Review of … https://www.mdpi.com/2078-2489/15/12/786
[49] Assessing the effects of data drift on the performance of machine … https://pmc.ncbi.nlm.nih.gov/articles/PMC9196120/
[50] External validation of a proprietary risk model for 1-year mortality in community-dwelling adults aged 65 years or older https://academic.oup.com/jamia/article/32/7/1110/8121806
[51] Lessons Learned from Model Lifecycle Management | Principles and Practices of the Generative AI Life Cycle https://youaccel.com/lesson/lessons-learned-from-model-lifecycle-management/premium
[52] Machine Learning operations maturity model – Azure Architecture Center https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model
[53] Comparison of 1-year mortality predictions from vendor-supplied versus academic model for cancer patients – PubMed https://pubmed.ncbi.nlm.nih.gov/39959833/?fc=20210219060646&ff=20250217120209&v=2.18.0.post9+e462414
[54] [PDF] Dimensionality reduction for multi-criteria problems – -ORCA https://orca.cardiff.ac.uk/id/eprint/128967/1/Edilson_Dimensionality%20reduction_ESWA2020.pdf
[55] 1 https://papers.phmsociety.org/index.php/phmconf/article/download/3007/1860
[56] Documenting the Decommissioning Process | Principles and Practices of the Generative AI Life Cycle https://youaccel.com/lesson/documenting-the-decommissioning-process/premium
[57] Automating Data Drift Thresholding in Machine Learning Systems https://www.arthur.ai/blog/automating-data-drift-thresholding-in-machine-learning-systems
[58] azure-ai-docs/articles/ai-services/openai/concepts/model-retirements.md at main · MicrosoftDocs/azure-ai-docs https://github.com/MicrosoftDocs/azure-ai-docs/blob/main/articles/ai-services/openai/concepts/model-retirements.md
[59] When to stop training your model https://docs.uipath.com/communications-mining/automation-cloud/latest/user-guide/when-to-stop-training-your-model
[60] Why do we need MLOps? https://docs.rafay.co/aiml/mlops-kubeflow/mlops/
[61] When Should a Machine Learning Model Be Retrained? – Valohai https://valohai.com/blog/when-should-a-machine-learning-model-be-retrained/
[62] From concept drift to model degradation – ScienceDirect.com https://www.sciencedirect.com/science/article/pii/S0950705122002854
[63] MLOps Principles https://ml-ops.org/content/mlops-principles
[64] [PDF] Making AI Forget You: Data Deletion in Machine Learning http://papers.neurips.cc/paper/8611-making-ai-forget-you-data-deletion-in-machine-learning.pdf
[65] Ultimate Guide to MLOps: Process, Maturity Path and Best Practices https://coralogix.com/ai-blog/ultimate-guide-to-mlops-process-maturity-path-and-best-practices/
[66] Decommissioning AI Systems: Best Practices and Guidelines for Off-boarding Large Language Models and Infrastructure https://www.amtechconsulting.org/post/decommissioning-ai-systems-best-practices-and-guidelines-for-off-boarding-large-language-models-and
[67] Ethical Considerations in Model Decommissioning | Principles and Practices of the Generative AI Life Cycle https://youaccel.com/lesson/ethical-considerations-in-model-decommissioning/premium
[68] 5.1 Decommission your assets appropriately – NCSC.GOV.UK https://www.ncsc.gov.uk/collection/machine-learning-principles/end-of-life/decommission-assets
[69] Machine unlearning | European Data Protection Supervisor https://www.edps.europa.eu/data-protection/technology-monitoring/techsonar/machine-unlearning
[70] Machine Learning Model Monitoring: What to Do In Production https://www.heavybit.com/library/article/machine-learning-model-monitoring
[71] Azure OpenAI in Azure AI Foundry Models model retirements – Azure OpenAI https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements
[72] azure-ai-docs/articles/machine-learning/concept-model-lifecycle-retirement.md at main · MicrosoftDocs/azure-ai-docs https://github.com/MicrosoftDocs/azure-ai-docs/blob/main/articles/machine-learning/concept-model-lifecycle-retirement.md
[73] Decoding MLOps: Key Concepts & Practices Explained – Dataiku https://www.dataiku.com/stories/detail/decoding-mlops/
[74] r/LeopardsAteMyFace: The Decommissioning Of AI https://www.forrester.com/blogs/r-leopardsatemyface-the-decommissioning-of-ai/
[75] MLOps: Continuous delivery and automation pipelines in machine … https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning
[76] Keeping Your Machine Learning Model in Top Form: A Guide to Post-Deployment Monitoring https://blog.gopenai.com/keeping-your-machine-learning-model-in-top-form-a-guide-to-post-deployment-monitoring-151a969ef027?gi=716de7a3e76e
[77] 11 MLOps Best Practices Explained in 2025 – Tredence https://www.tredence.com/blog/mlops-a-set-of-essential-practices-for-scaling-ml-powered-applications
[78] Development of a Machine Learning Model Using Limited Features … https://ascopubs.org/doi/10.1200/CCI.21.00163
[79] MLOps Checklist – 10 Best Practices for a Successful Model … https://neptune.ai/blog/mlops-best-practices
[80] Development of a Machine Learning Model Using Limited Features … https://pmc.ncbi.nlm.nih.gov/articles/PMC9067363/
[81] Predicting Patient Mortality for Earlier Palliative Care Identification in … https://ai.jmir.org/2023/1/e42253
[82] Optimized Retraining Guide for MLOps – barbara.tech https://www.barbara.tech/blog/optimized-retraining-guide-for-mlops
[83] Understanding Model Drift: Impact on Performance and Stability https://www.lyzr.ai/glossaries/model-drift/
[84] [PDF] Cost-Effective Retraining of Machine Learning Models – arXiv https://arxiv.org/pdf/2310.04216.pdf
[85] Cost-Effective Retraining of Machine Learning Models https://arxiv.org/pdf/2310.04216v1.pdf
[86] “© 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all https://opus.lib.uts.edu.au/bitstream/10453/167779/3/Concept%20Drift%20Detection%20Delay%20Index.pdf
[87] What is concept drift in ML, and how to detect and address it https://www.evidentlyai.com/ml-in-production/concept-drift
[88] Model Decommissioning in MLOps – Uniity Cloud https://uniity.cloud/platform-engineering/mlops/model-decommissioning-in-mlops/
[89] 10 MLOps Best Practices Every Team Should Be Using | Mission https://www.missioncloud.com/blog/10-mlops-best-practices-every-team-should-be-using
[90] MLOps foundation roadmap for enterprises with Amazon SageMaker https://aws.amazon.com/blogs/machine-learning/mlops-foundation-roadmap-for-enterprises-with-amazon-sagemaker/
[91] Navigating MLOps: Insights into Maturity, Lifecycle, Tools, and Careers https://ar5iv.labs.arxiv.org/html/2503.15577
[92] Machine Learning for Retirement Planning https://www.pm-research.com/content/iijretire/8/1/32
[93] Practitioners guide to MLOps: https://services.google.com/fh/files/misc/practitioners_guide_to_mlops_whitepaper.pdf
[94] [논문]Improving palliative and end-of-life care with machine learning … https://scienceon.kisti.re.kr/srch/selectPORSrchArticle.do?cn=NART105622168
[95] The Total Career Benchmark Model : A Pension Model for Retirement 20/20 https://www.soa.org/4934b9/globalassets/assets/library/newsletters/the-pension-forum/2012/pfn-2012-vol19-iss1-walker.pdf
[96] [PDF] 2017-22 Operational Measures Key Performance Indicator Summary https://www.calpers.ca.gov/documents/201712-pension-item-6-attach-1/download
[97] Model monitoring for ML in production: a comprehensive guide https://www.evidentlyai.com/ml-in-production/model-monitoring