naver.HOW data lake by AI

[카테고리:] 미분류

  • Mean Average Precision (mAP) 완벽 정리

    Mean Average Precision (mAP)는 객체 검출, 정보 검색, 추천 시스템 등에서 모델의 성능을 평가하는 핵심 지표입니다. 이 지표는 정밀도(precision)와 재현율(recall)의 균형을 고려하여 모델의 전체적인 성능을 하나의 수치로 요약합니다[1][2][3].

    기본 개념

    Precision과 Recall

    mAP를 이해하기 위해서는 먼저 precision과 recall의 개념을 명확히 해야 합니다[2][4].

    Precision (정밀도)는 모델이 positive로 예측한 것 중 실제로 맞는 비율입니다:
    $$ \text{Precision} = \frac{TP}{TP + FP} $$

    Recall (재현율)은 실제 positive인 것 중 모델이 올바르게 예측한 비율입니다:
    $$ \text{Recall} = \frac{TP}{TP + FN} $$

    여기서 TP는 True Positive, FP는 False Positive, FN은 False Negative를 의미합니다[5][6].

    Precision-Recall 곡선

    Precision-Recall 곡선은 다양한 임계값에서 precision과 recall의 관계를 시각화합니다[4][7]. 이 곡선은 모델의 성능을 종합적으로 평가할 수 있는 중요한 도구입니다.

    Average Precision (AP) 계산

    Average Precision (AP)는 하나의 클래스에 대한 precision-recall 곡선 아래의 면적을 나타냅니다[8][9]. AP는 다음과 같이 계산됩니다:

    $$ AP = \sum_{n} (R_n – R_{n-1}) \times P_n $$

    여기서 $$ P_n $$과 $$ R_n $$은 n번째 임계값에서의 precision과 recall입니다[10].

    11-point Interpolation (Pascal VOC)

    Pascal VOC에서는 11-point interpolation 방법을 사용합니다[11][12]. 이 방법은 recall이 0.0, 0.1, 0.2, …, 1.0인 11개의 고정된 점에서 precision을 측정합니다:

    $$ AP = \frac{1}{11} \sum_{r \in {0, 0.1, …, 1.0}} p_{interp}(r) $$

    여기서 $$ p_{interp}(r) = \max_{r’ \geq r} p(r’) $$입니다[12].

    All-point Interpolation (COCO)

    COCO에서는 101-point interpolation을 사용하여 더 정밀한 평가를 수행합니다[13][14]. 이 방법은 recall이 0.0, 0.01, 0.02, …, 1.0인 101개의 점에서 precision을 측정합니다.

    Mean Average Precision (mAP)

    mAP는 모든 클래스에 대한 AP의 평균입니다[15][16]:

    $$ mAP = \frac{1}{N} \sum_{i=1}^{N} AP_i $$

    여기서 N은 클래스 수이고, $$ AP_i $$는 i번째 클래스의 Average Precision입니다.

    객체 검출에서의 mAP

    IoU (Intersection over Union)

    객체 검출에서 mAP를 계산하기 위해서는 먼저 IoU를 통해 예측이 정확한지 판단해야 합니다[17][18]:

    $$ IoU = \frac{\text{Area of Intersection}}{\text{Area of Union}} $$

    일반적으로 IoU > 0.5이면 True Positive, IoU < 0.5이면 False Positive로 분류합니다[18][19].

    계산 과정

    1. 예측 신뢰도 순으로 정렬: 모델의 예측 결과를 신뢰도 순으로 정렬합니다[20][21].
    2. TP/FP 판정: 각 예측에 대해 IoU를 계산하여 TP/FP를 결정합니다[21][22].
    3. Precision-Recall 계산: 각 임계값에서 precision과 recall을 계산합니다[21][22].
    4. AP 계산: Precision-Recall 곡선 아래의 면적을 계산합니다[22].
    5. mAP 계산: 모든 클래스의 AP 평균을 구합니다[22].

    평가 표준의 차이

    Pascal VOC vs COCO

    Pascal VOC는 IoU 임계값을 0.5로 고정하고 11-point interpolation을 사용하는 반면, COCO는 IoU 임계값을 0.5에서 0.95까지 0.05 단위로 변경하며 101-point interpolation을 사용합니다[23][24].

    COCO의 평가 방식이 더 엄격하기 때문에, 일반적으로 COCO mAP가 Pascal VOC mAP보다 낮게 나타납니다[23][24].

    정보 검색에서의 MAP

    정보 검색 시스템에서 MAP은 검색 결과의 순위를 고려한 평가 지표입니다[25][26]:

    1. 쿼리별 AP 계산: 각 쿼리에 대해 관련 문서가 나타나는 위치에서의 precision을 계산합니다[27][28].
    2. MAP 계산: 모든 쿼리의 AP 평균을 구합니다[27][28].

    MAP@K는 상위 K개 결과만을 고려하는 변형입니다[29][30].

    추천 시스템에서의 MAP

    추천 시스템에서는 사용자별 추천 품질을 평가하기 위해 MAP을 사용합니다[31][28]:

    • 각 사용자에 대해 추천된 아이템 중 관련 있는 아이템의 순위를 고려하여 AP를 계산합니다.
    • 모든 사용자의 AP 평균이 MAP이 됩니다.

    실제 응용 및 벤치마크

    주요 데이터셋

    1. Pascal VOC (2007-2012): 20개 클래스, IoU 0.5 기준[24][32]
    2. MS COCO (2014-현재): 80개 클래스, 다양한 IoU 임계값[23][24]

    성능 지표 해석

    • mAP 0.5: IoU 0.5에서의 mAP
    • mAP 0.5:0.95: IoU 0.5~0.95 범위에서의 평균 mAP
    • mAP small/medium/large: 객체 크기별 mAP[23]

    한계 및 고려사항

    클래스 불균형

    mAP는 각 클래스에 동일한 가중치를 부여하므로, 클래스 불균형이 있는 데이터에서는 주의가 필요합니다[33][34].

    임계값 민감성

    IoU 임계값 설정에 따라 mAP 결과가 크게 달라질 수 있습니다[19][35].

    계산 복잡성

    특히 COCO 방식의 mAP는 계산이 복잡하고 시간이 많이 소요됩니다[13][14].

    구현 및 도구

    주요 라이브러리

    • pycocotools: COCO 형식 mAP 계산[13][36]
    • scikit-learn: 일반적인 AP 계산[10][37]
    • torchmetrics: PyTorch 기반 MAP 계산[38]

    계산 시 주의사항

    • 신뢰도 점수를 올바르게 사용해야 합니다[37]
    • 정의되지 않은 AP 값은 0으로 설정해야 합니다[35]
    • 결과 절단(truncation)이 성능에 미치는 영향을 고려해야 합니다[35]

    Mean Average Precision은 현재 컴퓨터 비전, 정보 검색, 추천 시스템 등 다양한 분야에서 표준적으로 사용되는 평가 지표입니다. 각 분야와 데이터셋의 특성에 맞는 적절한 계산 방법을 선택하고, 그 한계를 이해하여 올바르게 활용하는 것이 중요합니다.

    출처
    [1] Mean Average Precision (mAP) Explained: Everything You Need to … https://www.v7labs.com/blog/mean-average-precision
    [2] What is: Average Precision https://statisticseasily.com/glossario/what-is-average-precision/
    [3] “Mean average precision” (MAP) evaluation statistic – understanding good/bad/chance values https://stats.stackexchange.com/questions/92106/mean-average-precision-map-evaluation-statistic-understanding-good-bad-cha
    [4] mAP(mean average precision) – velog https://velog.io/@kwjinwoo/mAP
    [5] Average Precision – Testing with Kolena https://docs.kolena.com/metrics/average-precision/
    [6] Object Detection 과 Segmentation 평가 Metric 알아보기 https://glee1228.tistory.com/5
    [7] mAP(Mean Average Precision) 개념 이해하기 – velog https://velog.io/@kimkj38/Object-detection-metric-mAP-%EC%9B%90%EB%A6%AC-%EC%9D%B4%ED%95%B4%ED%95%98%EA%B8%B0
    [8] Average Precision | CloudFactory Computer Vision Wiki https://wiki.cloudfactory.com/docs/mp-wiki/metrics/average-precision
    [9] Evaluation metrics for object detection and segmentation: mAP https://kharshit.github.io/blog/2019/09/20/evaluation-metrics-for-object-detection-and-segmentation
    [10] mAP(Mean Average Precision) 정리 – DEVELOPER – 티스토리 https://ctkim.tistory.com/entry/mAPMean-Average-Precision-%EC%A0%95%EB%A6%AC
    [11] Evaluation 12: mean average precision https://www.youtube.com/watch?v=pM6DJ0ZZee0
    [12] [추천시스템][ML] 평가지표(Evaluation metric) : MAP(Mean Average … https://velog.io/@crosstar1228/ML-%ED%8F%89%EA%B0%80%EC%A7%80%ED%91%9CEvaluation-metric-MAPMean-Average-Precisionm
    [13] What is mean average precision (MAP) and how is it used in evaluation? – Zilliz Vector Database https://zilliz.com/ai-faq/what-is-mean-average-precision-map-and-how-is-it-used-in-evaluation
    [14] Average precision metric https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wxgov-average-precision-metric.html?context=wx
    [15] Object Detection 정의와 Evaluation metric (평가지표) https://ai-bt.tistory.com/entry/01-Object-Detection
    [16] What is Mean Average Precision (MAP)? https://milvus.io/ai-quick-reference/what-is-mean-average-precision-map
    [17] Packt+ | Advance your knowledge in tech https://www.packtpub.com/en-in/product/hands-on-computer-vision-with-tensorflow-2-9781788830645/chapter/object-detection-models-7/section/average-precision-and-mean-average-precision-ch07lvl1sec19
    [18] Performance Metrics Deep Dive – Ultralytics YOLO Docs https://docs.ultralytics.com/guides/yolo-performance-metrics/
    [19] What is Mean Average Precision (MAP) in recommender systems? https://milvus.io/ai-quick-reference/what-is-mean-average-precision-map-in-recommender-systems
    [20] Mean Average Precision – an overview | ScienceDirect Topics https://www.sciencedirect.com/topics/computer-science/mean-average-precision
    [21] C_2.08 Mean Average Precision (mAP) Explained – 위키독스 https://wikidocs.net/164371
    [22] Precision-Recall Curves https://www.scikit-yb.org/en/latest/api/classifier/prcurve.html
    [23] Precision Beyond Pixels: mAP Unveiled for Object Detection Assessment https://python.plainenglish.io/precision-beyond-pixels-map-unveiled-for-object-detection-assessment-e050564e3bef?gi=7f45340adc23
    [24] precision_recall_curve https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html
    [25] Evaluating Object Detection Models Using Mean Average Precision – KDnuggets https://www.kdnuggets.com/2021/03/evaluating-object-detection-models-using-mean-average-precision.html
    [26] Mean Average Precision (mAP): A Complete Guide – Kili Technology https://kili-technology.com/data-labeling/machine-learning/mean-average-precision-map-a-complete-guide
    [27] What is the Precision-Recall Curve? | Machine Learning Tutorial https://www.youtube.com/watch?v=Py8cTbFLy_A
    [28] mAPObjectDetectionMetric – Mean average precision (mAP) metric for object detection – MATLAB https://www.mathworks.com/help/vision/ref/mapobjectdetectionmetric.html
    [29] Precision-Recall Curve – ML – GeeksforGeeks https://www.geeksforgeeks.org/machine-learning/precision-recall-curve-ml/
    [30] What is mAP in object detection? https://stackoverflow.com/questions/59452897/what-is-map-in-object-detection/59493505
    [31] Precision-Recall¶ https://scikit-learn.org/0.21/auto_examples/model_selection/plot_precision_recall.html
    [32] Mean average precision (mAP) in object detection | SuperAnnotate https://www.superannotate.com/blog/mean-average-precision-and-its-uses-in-object-detection
    [33] What is Mean Average Precision (mAP) in Object Detection? https://blog.roboflow.com/mean-average-precision/
    [34] Precision-Recall Curve https://www.jmp.com/support/help/en/18.2/jmp/precisionrecall-curve.shtml
    [35] Object Detection의 정의와 Metric mAP(mean Average Precision) https://herbwood.tistory.com/2
    [36] average_precision_score — scikit-learn 1.7.0 documentation https://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html
    [37] Precision-Recall — scikit-learn 1.7.0 documentation https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html
    [38] What is Intersection over Union (IoU)? – Viso Suite https://viso.ai/computer-vision/intersection-over-union-iou/
    [39] mAP Calculator – PascalVOC2012 and COCO Standards – GitHub https://github.com/shreejalt/mAP-calculator
    [40] What is Mean Average Precision (MAP)? – Zilliz Vector Database https://zilliz.com/ai-faq/what-is-mean-average-precision-map
    [41] Intersection over Union (IoU): Definition, Calculation, Code – V7 Labs https://www.v7labs.com/blog/intersection-over-union-guide
    [42] Table 1. https://pmc.ncbi.nlm.nih.gov/articles/PMC9738404/table/sensors-22-09536-t001/
    [43] IoU, Intersection over Union 개념을 이해하자 – 딥러닝, 패기있게 https://ballentain.tistory.com/12
    [44] [개념 설명] Object Detection 평가 방법[2](IOU, AP, mAP) https://autonomous-vehicle.tistory.com/30
    [45] Mean Average Precision (MAP) for Information Retrieval Systems – Analytics Yogi https://vitalflux.com/mean-average-precision-map-for-information-retrieval-systems/
    [46] [Object Detection] IoU(Intersection over Union) & NMS(Non-Max … https://developer-lionhong.tistory.com/33
    [47] Mean Average Precision (mAP) Using the COCO Evaluator https://pyimagesearch.com/2022/05/02/mean-average-precision-map-using-the-coco-evaluator/
    [48] Mean Average Precision Made Simple With A Complete Guide https://spotintelligence.com/2023/09/07/mean-average-precision/
    [49] Intersection over Union (IoU) Explained – Ultralytics https://www.ultralytics.com/glossary/intersection-over-union-iou
    [50] Understanding Pascal VOC and COCO Annotations for Object Detection https://pub.towardsai.net/understanding-coco-and-pascal-voc-annotations-for-object-detection-bb8ffbbb36e3?gi=162b30d65851
    [51] 정보 검색(Information Retrieval) 평가 방법: MAP, MRR, DCG, NDCG https://modulabs.co.kr/blog/information-retrieval-map-ndcg
    [52] Intersection over Union (IoU) in Object Detection & Segmentation https://learnopencv.com/intersection-over-union-iou-in-object-detection-and-segmentation/
    [53] Large and suspicious differences on Pascal VOC 2012 and MS COCO 2017 validation sets (1 class) https://groups.google.com/g/caffe-users/c/mn6TNfoBInE
    [54] 정보 검색(Information Retrieval) 평가는 어떻게 하는 것이 좋을까?(2/2) https://lamttic.github.io/2020/03/20/01.html
    [55] C_2.01 IoU (Intersection over Union) and GIoU – 4. Object Detection https://wikidocs.net/177706
    [56] Mean Average Precision(mAP) on Object detection https://seongkyun.github.io/study/2019/01/15/map/
    [57] Understanding Detectron2 COCO Evaluator https://stackoverflow.com/questions/70959952/understanding-detectron2-coco-evaluator/72133351
    [58] Slide 1 https://web.stanford.edu/class/cs276/handouts/EvaluationNew-handout-1-per.pdf
    [59] GitHub – feiyuhuahuo/COCO_improved: An improved mAP measurement tool for COCO object detection and instance segmentation. https://github.com/feiyuhuahuo/COCO_improved
    [60] Retrieval Mean Average Precision (MAP)¶ https://torchmetrics.readthedocs.io/en/v0.8.2/retrieval/map.html
    [61] Mean Average Precision (mAP) in Object Detection – LearnOpenCV https://learnopencv.com/mean-average-precision-map-object-detection-model-evaluation-metric/
    [62] Efficient Graph-Friendly COCO Metric https://arxiv.org/pdf/2207.12120v1.pdf
    [63] [딥러닝] mAP(mean Average Precision)이란? – 테크 조랭이떡 https://zorang2.tistory.com/89
    [64] Under review as a conference paper at ICLR 2023 https://openreview.net/pdf/b64642f9ced9d99f97529b07fb5608d3f7f4aedb.pdf
    [65] Mean Average Precision at K (MAP@K) clearly explained https://towardsdatascience.com/mean-average-precision-at-k-map-k-clearly-explained-538d8e032d2/
    [66] DetectionMAP¶ https://www.paddlepaddle.org.cn/documentation/docs/en/2.4/api/paddle/fluid/metrics/DetectionMAP_en.html
    [67] canlion/COCO_evaluation: coco evaluation 간소화 – GitHub https://github.com/canlion/COCO_evaluation
    [68] The MAP metric in Information Retrieval Fault Localization (ASE 2023 – Research Papers) – ASE 2023 https://conf.researchr.org/details/ase-2023/ase-2023-papers/2/The-MAP-metric-in-Information-Retrieval-Fault-Localization
    [69] How to calculate mAP for detection task for the PASCAL VOC Challenge? https://datascience.stackexchange.com/questions/25119/how-to-calculate-map-for-detection-task-for-the-pascal-voc-challenge
    [70] COCO Evaluation metrics explained – Picsellia https://www.picsellia.com/post/coco-evaluation-metrics-explained
    [71] Mean Average Precision (MAP) in ranking and recommendations https://www.evidentlyai.com/ranking-metrics/mean-average-precision-map
    [72] [Evaluate Performance] mAP(mean Average Precision) https://dnai-deny.tistory.com/55
    [73] Keras documentation: Classification metrics based on True/False positives & negatives https://keras.io/api/metrics/classification_metrics/
    [74] What is Precision-Recall curve (PR AUC)? https://klu.ai/glossary/pr-auc
    [75] GitHub – Vinicius-ufsc/average_precision_for_multilabel_classification: Guide to average precision calculation https://github.com/Vinicius-ufsc/average_precision_for_multilabel_classification
    [76] ROC Curves and Precision-Recall Curves for Imbalanced … https://www.machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/
    [77] Ultimate Guide to PR-AUC: Calculations, Uses, and Limitations https://coralogix.com/ai-blog/ultimate-guide-to-pr-auc-calculations-uses-and-limitations/
    [78] Precision-Recall Curves and AUPRC | Confusion Matrix Metrics Part 9 | Machine Learning https://www.youtube.com/watch?v=fF0MvCPsEp8
    [79] Section 3.3 Interpolation Techniques https://confluence.ecmwf.int/display/FUG/Section+3.3+Interpolation+Techniques
    [80] Area under Precision-Recall Curve (AUC of PR-curve) and Average Precision (AP) https://stats.stackexchange.com/questions/157012/area-under-precision-recall-curve-auc-of-pr-curve-and-average-precision-ap
    [81] 17.5. Evaluation of retrieval model – non-interpolated and interpolated average precision https://www.youtube.com/watch?v=t7CL4NgopTc