Free · no sign-up · nothing stored

Intraclass Correlation Calculator (ICC for Inter-Rater Reliability)

Which ICC should I use for inter-rater reliability?

If the same raters scored every subject and you want the result to hold for other raters, use the two-way random-effects, absolute-agreement ICC: ICC(2,1) for one rater's score, ICC(2,k) for the average of k raters. Report it with its 95% confidence interval. Koo and Li (2016) call values below .50 poor, .50 to .75 moderate, .75 to .90 good and above .90 excellent.

There are six ICCs and they can give very different answers from the same data: Shrout and Fleiss's own example runs from .17 to .91. Most mistakes in theses are picking the wrong one, or picking the highest.

Answer three plain questions about your design, paste the ratings, and the calculator gives the ICC you need with its confidence interval, all six for comparison, and a sentence you can put in your results.

Use the calculator All free tools

Built by Muhammad Shahzeb Anwar, MSc Industrial and Organizational Psychology. Rater reliability is checked wherever people score people: interviews, assessment centres, performance ratings, clinical scales.

The calculator

Your ratings

Runs in your browser. Nothing you type is stored or sent anywhere.

Consistency ignores a rater who scores everyone higher. One-way models always use absolute agreement.
The example is Shrout and Fleiss's (1979) 6 targets rated by 4 judges; replace it with your own. Separate with commas, tabs or spaces; a text label at the start of a row is ignored. No missing values. Nothing you paste leaves this page.

Choosing the ICC in three questions

1. Did the same raters rate every subject? If each subject had its own raters (different interviewers for different candidates), the model is one-way random, ICC(1). If the same raters rated everyone, it is two-way.

2. Are the raters a sample, or the only raters that matter? If you want the result to generalise to other trained raters, the raters are random (ICC(2)). If these particular raters are the only ones who will ever rate, they are fixed (mixed model, ICC(3)). The arithmetic of the point estimate is the same for random and mixed; the interpretation is not.

3. Must the scores match, or only the order? Absolute agreement counts a rater who scores everyone two points higher as disagreeing. Consistency does not. If a pass mark or cut-off is applied to the scores, use absolute agreement.

Finally, single or average: if in practice one rater will score each subject, report the single-measures ICC. If the final score will be the mean of k raters, report the average-measures ICC, which is always higher. Reporting the average when one rater will be used is the most common way an ICC is overstated.

Worked example: Shrout and Fleiss (1979)

Six targets were rated by four judges (the data filled in above). The six ICCs come out as .17, .29, .71, .44, .62 and .91. The gap between ICC(2,1) = .29 and ICC(3,1) = .71 is the judges' different average levels: judge 1 rated everyone high and judge 2 everyone low. They agree on the order of the targets (consistency) but not on the scores (absolute agreement). The calculator reproduces the published values to two decimals.

What a confidence interval adds

With six subjects the 95% interval for ICC(3,1) runs from about .34 to .95: the data are consistent with anything from poor to excellent. Koo and Li (2016) recommend judging reliability by the interval, and a common planning rule is at least 30 subjects and at least 3 raters. More subjects narrow the interval far more than more raters do.

Before you rely on this

What this will not tell you

This computes ICCs for complete data: every subject rated by every rater (or, for the one-way model, the same number of ratings per subject). It does not handle missing ratings, unbalanced designs, multi-level or generalisability-theory designs, or kappa-type agreement for categories (use Cohen's or Fleiss' kappa for those). Nothing you paste leaves your browser.

Questions

Questions people actually ask

What is a good ICC value?

Koo and Li (2016): below .50 poor, .50 to .75 moderate, .75 to .90 good, above .90 excellent. Judge by the 95% confidence interval, not only the point estimate.

What is the difference between ICC(2,1) and ICC(3,1)?

Both use a two-way design. ICC(2,1) measures absolute agreement and treats raters as random; ICC(3,1) measures consistency with raters fixed, so a rater who scores everyone higher does not lower it.

Should I report single or average measures?

Report single measures if one rater's score will be used in practice, and average measures only if the final score really is the mean of the k raters.

How many subjects do I need for an ICC?

Koo and Li suggest at least 30 heterogeneous subjects and at least 3 raters. With fewer subjects the confidence interval becomes too wide to say much.

Is ICC the same as Pearson correlation?

No. Pearson correlation ignores differences in level between raters and works for two raters only; the absolute-agreement ICC counts those differences and works for any number of raters.

Can an ICC be negative?

Yes, when the variation within subjects is larger than between them. Treat a negative ICC as zero reliability, and check the data for a reversed scale or data-entry error.

The papers behind the method

Sources

Formulas from Shrout and Fleiss (1979) and McGraw and Wong (1996); the worked example reproduces Shrout and Fleiss's Table 2 data and published values. References checked on 2 November 2026.

  1. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420-428: https://doi.org/10.1037/0033-2909.86.2.420
  2. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30-46: https://doi.org/10.1037/1082-989X.1.1.30
  3. Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163: https://doi.org/10.1016/j.jcm.2016.02.012
Citation

Cite this tool

Free to use and free to quote. If a figure here is useful in something you are writing, a link back is all that is asked.

Muhammad Shahzeb Anwar, MSc Industrial and Organizational Psychology. Intraclass Correlation (ICC) Calculator. Shahzeb Speaks. https://www.shahzebspeaks.com/intraclass-correlation-calculator/

Every free tool

The other free tools

The tools hub →

Each one answers a single question and refuses the ones it cannot. All free, none asks for an email address.

Pakistan employment law6 tools

Gratuity Calculator PakistanProvince by province, from the Standing Orders themselves, with the tax-free part. Final Settlement Calculator PakistanLast salary, unused leave, notice pay and gratuity in one statement. Minimum Wage Checker PakistanEach province's notified figure, the arrears it creates, and EOBI on top. Maternity Leave Calculator PakistanWeeks, dates, pay and paternity leave under each province's law. Harassment Act Compliance Checklist (Pakistan)Is your inquiry committee valid, and what else the 2010 Act requires. Punjab Labour Code 2026 CalculatorGratuity with the new caps, notice, unused leave and overtime under the 2026 Code.

India3 tools

Labour Codes Applicability Checker (India)Which duties under India's four Labour Codes apply at your headcount. Leave Encashment Calculator: New Labour Code (India)Annual leave earned, the 30-day carry-forward cap and encashment under the OSH Code. POSH Internal Committee Checker and Annual Report Builder (India)Internal Committee composition, the complaint timeline and the annual report.

UK employment law4 tools

Employment Tribunal Time Limit CalculatorThe new six-month limit, the old three-month one, and the Acas clock-stop. Unfair Dismissal Qualifying Period CheckerTwo years until 2027, six months after, with the statutory notice extension. All Reasonable Steps Checker: Sexual Harassment Duty (UK)Score your prevention against the EHRC's eight steps before 30 October 2026. 48-Hour Week Average CalculatorThe 17-week average, with leave put back in the way the regulations require.

United States5 tools

Pay Transparency Laws by State CheckerTick the states and cities a job touches; see what the posting must say. AI Hiring Laws by State CheckerNYC, Illinois, California, Colorado, Connecticut and more: what applies, and when. Non-Compete Laws by State Checker (2026)State bans, 2026 pay thresholds and the conditions on non-competes. WARN Act and State Mini-WARN CalculatorIs a layoff covered, how much notice, and the state rules on top. FMLA Eligibility CalculatorThe four federal tests, and how many of the 12 weeks are left.

Canada4 tools

Termination and Severance Pay Calculator CanadaStatutory notice and severance in all 14 jurisdictions, with group terminations. Vacation Pay Calculator CanadaVacation time and pay in all 14 jurisdictions, including PEI's 2026 change. Statutory Holiday Pay Calculator CanadaHoliday pay and pay for working the holiday, in all 14 jurisdictions. Job Posting Compliance Checker (Canada)Ontario's pay range cap, AI and Canadian-experience rules, plus BC, PEI and NB.

EU regulation and hiring law5 tools

EU AI Act HR and Recruitment CheckerIs your hiring or HR AI high-risk, banned or neither, and what applies when. EU Pay Transparency Directive CheckerWhich country has transposed it, what applies to you, and your first report date. EU Pay Transparency Gap CalculatorAll seven Article 9 indicators from one pasted list, and the 5% flag. NYC Bias Audit Impact Ratio CalculatorLocal Law 144's selection and scoring rates, by sex, race and both. Interview Question CheckerPaste your questions: which ones are unlawful, risky or fine, by country.

I/O psychology and psychometrics13 tools

Content Validity Ratio (CVR) CalculatorLawshe's CVR for every item, with exact critical values and the CVI. Aiken's V CalculatorAiken's V for each item, with its confidence interval and exact p-value. Standard Error of Measurement and Banding CalculatorSEM, SED and the band of scores that cannot be told apart from the top. Taylor-Russell CalculatorValidity, selection ratio and base rate into the share of hires who succeed. rwg and ICC Calculator (Within-Group Agreement)Within-group agreement and ICCs before you aggregate team or climate scores. Motivating Potential Score CalculatorThe Job Characteristics Model's MPS, and which redesign lever moves it most. Vroom Expectancy Theory CalculatorVroom's motivational force for several choices and outcomes, side by side. Item Analysis Calculator (Difficulty, Discrimination, KR-20)Difficulty, discrimination, distractors and KR-20 for a multiple-choice test, from pasted answers. 9-Box Grid Generator (Performance and Potential)Paste names and two ratings; get the plotted 9-box grid and the rating problems to fix first. Copenhagen Burnout Inventory (CBI) CalculatorThe free 19-item CBI with official scoring, for yourself or a whole team. Composite Reliability, AVE and HTMT CalculatorCR, AVE, MSV, Fornell-Larcker and HTMT for every construct, from loadings and correlations. eNPS Calculator with Margin of ErroreNPS with its margin of error, and whether the change since last time is real. Spearman-Brown Prophecy CalculatorHow reliable a longer or shorter test would be, and how many items reach a target.

US money and tax (2026)5 tools

Quarterly Estimated Tax Calculator 2026Self-employment and income tax for 2026, the safe-harbor amount, and what to pay each quarter. Mileage Deduction Calculator 2026The two 2026 IRS mileage rates, split by date, and how much of each gig payout to set aside. No Tax on Tips and Overtime Calculator 2026The 2026 tips and overtime deductions after the caps and the $150,000 phase-out. Dependent Care FSA vs Tax Credit Calculator 2026The $7,500 FSA against the 50% child care credit: what each comes to on your numbers. Parent PLUS Loan Calculator 2026-27Net price, the gap after the student's loan, Parent PLUS under the 2026 limits, and the payment.

UK money and tax (2026-27)7 tools

Making Tax Digital Income Tax CheckerThe £50,000, £30,000 and £20,000 tests on gross income, and your start date. Online Selling Tax Checker (UK)Trading or not, the £1,000 allowance, and what the platform report means. Payments on Account CalculatorWhat Self Assessment asks for in January and July, and the first-year surprise. 60% Tax Trap and £100k Childcare Cliff CalculatorAdjusted net income, the £100k cliff and the rate on your next £1,000. Landlord Tax Calculator 2027Your rental tax now, under the 22% law and the 24% Budget plan. Care Home Fees: How Long Will Savings Last?Months until savings reach £23,250, and the weekly top-up after that. UK Maternity Pay Calculator 2026-27Statutory Maternity Pay week by week from your start date, with the week-7 drop.

Canada money and tax4 tools

TFSA Contribution Room CalculatorYour TFSA room, the over-contribution tax, and your room on 1 Jan 2027. GST/HST $30,000 Small Supplier CalculatorOne quarter or four: when the $30,000 GST/HST limit makes you register. FHSA + HBP Down Payment CalculatorFHSA, the $60,000 HBP and savings against the minimum down payment, CMHC and the GST rebate. EI Maternity and Parental Leave Calculator (Canada)EI maternity and parental benefits per week and in total, standard or extended, with dates.

Online sellers3 tools

Etsy Profit CalculatorEvery Etsy fee by country, the profit, and what an Offsite Ads sale leaves. TikTok Shop Fee CalculatorFee, creator commission, returns and ads: what one order really leaves you. Reseller Profit CalculatorWhat a flip really makes after eBay, Poshmark, Mercari, Depop, Vinted, Facebook or Etsy fees.

Household planners4 tools

Wedding Cost Per Guest CalculatorFixed vs per-guest wedding costs: what each guest really adds and how many your budget covers. Balance Transfer 0% APR Payoff CalculatorWill a 0% balance be gone before the promo ends? The monthly payment that clears it. Christmas Savings CalculatorYour Christmas budget split into weekly, every-2-weeks or monthly amounts, with the dates. Puppy First-Year Cost CalculatorWhat a new puppy costs in year one: one-time costs, monthly budget, total.

Hiring and selection4 tools

Adverse Impact CalculatorThe four-fifths ratio, and how many selections would clear it. Selection Accuracy CalculatorWhat a hiring test is worth once the base rate is included. Utility Analysis CalculatorWhat a better hiring method is actually worth, in money. Range Restriction CalculatorWhy your own data makes your test look worse than it is.

Absence, pay, turnover and engagement4 tools

Bradford Factor CalculatorWhy five one-day absences score twenty-five times one long one. Pay Gap CalculatorMean against median, and what their disagreement tells you. Employee Turnover Cost CalculatorFive lines, and the largest is the one nobody invoices for. Survivorship Bias CalculatorDid the score rise, or did the unhappy people leave?

Reading a number without being fooled by it4 tools

Regression to the Mean CalculatorWhy the month after a record month always disappoints. Review Rating Confidence CalculatorIs 4.6 from twelve reviews better than 4.4 from three hundred? Combined Rate Trap CalculatorBoth groups improved and the total fell. Simpson's paradox. Multiple Comparisons CalculatorRun twenty tests and one will win on its own.

Studying I/O psychology methods?

I/O Psychology Prep is a free study app with flashcards on reliability, validity, selection and the rest of the I/O syllabus.

Get the free study app