22  Gleason Analysis

22.1 Primary Pattern

Note for Pathologists: Primary Gleason Pattern distribution across all interpreters (Pathologists with/without AI, AI model, Report, and Reference Diagnosis).

Pattern
Total
3 4 5
Interpreter



    P1_p 74 (54%) 33 (24%) 31 (22%) 138 (100%)
    P1AI_p 69 (50%) 51 (37%) 18 (13%) 138 (100%)
    P2_p 52 (38%) 70 (51%) 16 (12%) 138 (100%)
    P2AI_p 56 (41%) 76 (55%) 6 (4.3%) 138 (100%)
    P3_p 36 (26%) 68 (49%) 34 (25%) 138 (100%)
    P3AI_p 63 (46%) 68 (49%) 7 (5.1%) 138 (100%)
    P4_p 30 (22%) 97 (70%) 11 (8.0%) 138 (100%)
    P4AI_p 57 (41%) 73 (53%) 8 (5.8%) 138 (100%)
    AI_p 67 (49%) 69 (50%) 2 (1.4%) 138 (100%)
    Rep_p 39 (28%) 96 (70%) 3 (2.2%) 138 (100%)
    Gold_p 39 (28%) 96 (70%) 3 (2.2%) 138 (100%)
Total 582 (38%) 797 (53%) 139 (9.2%) 1,518 (100%)

22.1.1 Primary Pattern agreement

Primary Pattern agreement:: Pathologists no AI and Gold Standart

Note for Pathologists: Inter-rater agreement for Primary Pattern among pathologists (without AI) and the Reference Diagnosis.


 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      31.88406   
   Kappa                           0.4258547   
   z                                20.73969   
   p-value        < .0000001   
 ─────────────────────────────────────────────

Primary Pattern agreement:: Pathologists with AI and Gold Standart

Note for Pathologists: Inter-rater agreement for Primary Pattern among pathologists (with AI) and the Reference Diagnosis.


 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      53.62319   
   Kappa                           0.6116531   
   z                                26.47632   
   p-value        < .0000001   
 ─────────────────────────────────────────────

22.2 Secondary Pattern

Note for Pathologists: Secondary Gleason Pattern distribution across all interpreters.

Pattern
Total
3 4 5
Interpreter



    P1_s 67 (49%) 44 (32%) 27 (20%) 138 (100%)
    P1AI_s 44 (32%) 68 (49%) 26 (19%) 138 (100%)
    P2_s 57 (41%) 69 (50%) 12 (8.7%) 138 (100%)
    P2AI_s 34 (25%) 92 (67%) 12 (8.7%) 138 (100%)
    P3_s 31 (22%) 58 (42%) 49 (36%) 138 (100%)
    P3AI_s 29 (21%) 77 (56%) 32 (23%) 138 (100%)
    P4_s 38 (28%) 85 (62%) 15 (11%) 138 (100%)
    P4AI_s 49 (36%) 80 (58%) 9 (6.5%) 138 (100%)
    AI_s 29 (21%) 81 (59%) 28 (20%) 138 (100%)
    Rep_s 52 (38%) 49 (36%) 37 (27%) 138 (100%)
    Gold_s 52 (38%) 49 (36%) 37 (27%) 138 (100%)
Total 482 (32%) 752 (50%) 284 (19%) 1,518 (100%)

22.2.1 Secondary Pattern agreement

Secondary Pattern agreement:: Pathologists no AI and Gold Standart

Note for Pathologists: Inter-rater agreement for Secondary Pattern among pathologists (without AI) and the Reference Diagnosis.


 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      14.49275   
   Kappa                           0.2246169   
   z                                11.46534   
   p-value        < .0000001   
 ─────────────────────────────────────────────

Secondary Pattern agreement:: Pathologists with AI and Gold Standart

Note for Pathologists: Inter-rater agreement for Secondary Pattern among pathologists (with AI) and the Reference Diagnosis.


 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      22.46377   
   Kappa                           0.3303647   
   z                                16.52973   
   p-value        < .0000001   
 ─────────────────────────────────────────────

22.3 Gleason Score

Note for Pathologists: Gleason Score (Sum) distribution across all interpreters.

Score
Total
6 7 8 9 10
Interpreter





    P1_score 56 (41%) 26 (19%) 15 (11%) 27 (20%) 14 (10%) 138 (100%)
    P1AI_score 34 (25%) 45 (33%) 17 (12%) 40 (29%) 2 (1.4%) 138 (100%)
    P2_score 25 (18%) 59 (43%) 30 (22%) 20 (14%) 4 (2.9%) 138 (100%)
    P2AI_score 15 (11%) 60 (43%) 45 (33%) 18 (13%) 0 (0%) 138 (100%)
    P3_score 24 (17%) 19 (14%) 24 (17%) 59 (43%) 12 (8.7%) 138 (100%)
    P3AI_score 15 (11%) 62 (45%) 24 (17%) 35 (25%) 2 (1.4%) 138 (100%)
    P4_score 15 (11%) 38 (28%) 61 (44%) 22 (16%) 2 (1.4%) 138 (100%)
    P4AI_score 21 (15%) 64 (46%) 36 (26%) 17 (12%) 0 (0%) 138 (100%)
    AI_score 18 (13%) 59 (43%) 32 (23%) 29 (21%) 0 (0%) 138 (100%)
    Rep_score 16 (12%) 59 (43%) 25 (18%) 36 (26%) 2 (1.4%) 138 (100%)
    Gold_score 16 (12%) 59 (43%) 25 (18%) 36 (26%) 2 (1.4%) 138 (100%)
    paige_score 18 (13%) 59 (43%) 32 (23%) 29 (21%) 0 (0%) 138 (100%)
    report_score 16 (12%) 59 (43%) 25 (18%) 36 (26%) 2 (1.4%) 138 (100%)
    research_score 16 (12%) 59 (43%) 25 (18%) 36 (26%) 2 (1.4%) 138 (100%)
Total 305 (16%) 727 (38%) 416 (22%) 440 (23%) 44 (2.3%) 1,932 (100%)

22.3.1 Gleason Score Agreement

Gleason Score agreement :: Pathologists no AI and Reference Diagnosis

Note for Pathologists: Inter-rater agreement for Gleason Score among pathologists (without AI) and the Reference Diagnosis.

Quadratic Weighted Kappa (Gleason Score) - No AI
Rater1 Rater2 Kappa P_value Subjects
P1_score Gold_score 0.366 0 138
P2_score Gold_score 0.498 0 138
P3_score Gold_score 0.288 0 138
P4_score Gold_score 0.691 0 138 

 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      15.94203   
   Kappa                           0.3177971   
   z                                21.77379   
   p-value        < .0000001   
 ─────────────────────────────────────────────

Gleason Score agreement :: Pathologists with AI and Reference Diagnosis

Note for Pathologists: Inter-rater agreement for Gleason Score among pathologists (with AI) and the Reference Diagnosis.

Quadratic Weighted Kappa (Gleason Score) - With AI
Rater1 Rater2 Kappa P_value Subjects
P1AI_score Gold_score 0.757 0 138
P2AI_score Gold_score 0.710 0 138
P3AI_score Gold_score 0.568 0 138
P4AI_score Gold_score 0.696 0 138 

 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      38.40580   
   Kappa                           0.5418164   
   z                                33.91645   
   p-value        < .0000001   
 ─────────────────────────────────────────────

22.4 Grade Group

Note for Pathologists: Gleason Grade Group (1-5) distribution across all interpreters.

GradeGroup
Total
1 2 3 4 5
Interpreter





    P1_gg 56 (41%) 17 (12%) 9 (6.5%) 15 (11%) 41 (30%) 138 (100%)
    P1AI_gg 34 (25%) 35 (25%) 10 (7.2%) 17 (12%) 42 (30%) 138 (100%)
    P2_gg 25 (18%) 27 (20%) 32 (23%) 30 (22%) 24 (17%) 138 (100%)
    P2AI_gg 15 (11%) 41 (30%) 19 (14%) 45 (33%) 18 (13%) 138 (100%)
    P3_gg 24 (17%) 12 (8.7%) 7 (5.1%) 24 (17%) 71 (51%) 138 (100%)
    P3AI_gg 15 (11%) 48 (35%) 14 (10%) 24 (17%) 37 (27%) 138 (100%)
    P4_gg 15 (11%) 15 (11%) 23 (17%) 61 (44%) 24 (17%) 138 (100%)
    P4AI_gg 21 (15%) 36 (26%) 28 (20%) 36 (26%) 17 (12%) 138 (100%)
    AI_gg 18 (13%) 48 (35%) 11 (8.0%) 32 (23%) 29 (21%) 138 (100%)
    Rep_gg 16 (12%) 23 (17%) 36 (26%) 25 (18%) 38 (28%) 138 (100%)
    Gold_gg 16 (12%) 23 (17%) 36 (26%) 25 (18%) 38 (28%) 138 (100%)
Total 255 (17%) 325 (21%) 225 (15%) 334 (22%) 379 (25%) 1,518 (100%)

22.4.1 Grade Group agreement

Grade Group agreement: Pathologists no AI and Gold Standart

Note for Pathologists: Inter-rater agreement for Grade Group between pathologists (without AI) and the Reference Diagnosis.


 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      19.56522   
   Kappa                           0.3221551   
   z                                23.33324   
   p-value        < .0000001   
 ─────────────────────────────────────────────

Grade Group agreement: Pathologists with AI and Gold Standart

Note for Pathologists: Inter-rater agreement for Grade Group between pathologists (with AI) and the Reference Diagnosis.


 INTERRATER RELIABILITY

 Interrater Reliability                        
 ───────────────────────────────────────────── 
                  Fleiss' Kappa for m Raters   
 ───────────────────────────────────────────── 
   Subjects                              138   
   Raters                                  5   
   Agreement %                      23.91304   
   Kappa                           0.4773394   
   z                                34.82143   
   p-value        < .0000001   
 ─────────────────────────────────────────────