Basic Psychometric Concepts

Psychometric methods are used to ensure the fairness and accuracy of the examination process. A psychometrically sound examination gives candidates taking the exam reassurance that they are being tested on content that is both appropriate and relevant for the credential they are seeking.[1]

Psychometrics is the science and technology of mental measurement. The science of testing individuals uses knowledge from a variety of disciplines, including: psychology and behavioral science, education, and statistics and, more recently, computer science. The individual professionals who are experts in testing are called psychometricians. Companies who employing psychometricians and provide testing services are referred to as testing agencies or testing companies.[2]

According to “The NOCA Guide to Understanding Credentialing Concepts”[3] professional psychometricians are required for the following tasks:

  • Design and analyze results of a job analysis or role delineation to define knowledge and/or skill associated with performance domains and tasks associated with the identified profession.
  • Establish examination specifications based on a job analysis or role delineation.
  • Select appropriate examination item format to meet measurement goals.
  • Facilitate examination development based on examination specifications and item writing principles.
  • Facilitate passing standard (‘cut score’) studies, such that the cut score is consistent with the purpose of the credential and the established standard of competence for the profession.
  • Advise on examination administration policies and procedures that are appropriate, standardized, and secure.
  • Analyze examination results using appropriate statistical methods.
  • Establish scoring and reporting procedures, and ensure the security and confidentiality of such scores and reports.
  • Ensure that the reported scores are sufficiently reliable for the intended purpose(s) of the examination.
  • Ensure that different forms of an examination assess equivalent content and that candidates are not disadvantaged for taking a form of an examination that varies in difficulty from another form.
  • Conduct ongoing research in the areas of reliability and validity.

Validity and Reliability:

The two most basic principles of psychometrics are validity and reliability and they are directly related to the critical issues of accuracy and fairness.

To say an exam is valid, is to say that it measures what it is intended to measure. Validity is a measurement and does not provide a simple “yes” or “no” response. Instead, evidence is collected by the psychometrican to support the validity of the exam results. As Dr. Fabrey states, “The most important principle of validity is that an examination is valid only for its intended purpose.” The most common method used to collect validity evidence for certification examinations is by documenting content validity.

To say an exam is reliable is to indicate the accuracy of the measurement across administrations of the exam. A reliable exam is free from measurement errors. [4]

Common Psychometric Terms:[5]

Content-related validity: One of the three traditionally-defined methodologies of collecting evidence in support of the validity of an examination; a job analysis provides the primary content validation strategy for credentialing examinations.

Decision consistency: A measure of reliability, decision consistency answers the question “If the same examinees were administered two equivalent forms of the same examination, what percentage would be classified the same way (i.e., pass or fail) on both examinations?”

Equating: A statistical procedure used to compare the difficulty of alternate forms of a test, such that no matter what form of a test is taken, candidates will be treated fairly and consistently; different equating designs and statistical procedures (e.g., classical and IRT) may be used for equating.

Item bias: A source of invalidity resulting from interpretations to be made from item performance being different for identifiable groups; for bias to exist, an item must have the potential to perform differently for different groups of examinees; this differential item functioning can be detected either during examination development or during the analysis of examination results.

Psychometrics: The measurement of some aspect of a mental process.

Reliability: The degree to which scores are free from errors of measurement; the degree of consistency, dependability, or repeatability of scores.

Test bias: A source of invalidity of examination scores resulting from interpretations to be made from test scores being different for identifiable groups.

Validity: The extent to which a test result measures what it purports to measure.

[1] Certification: A NOCA Handbook, Edited by Anne H. Browning, Alan C. Bugbee, Jr., and Meredith A. Mullins

[2] Starting a Certification Program, 2nd Edition, Larry Allan Early, Ph.D.

[3] The NOCA Guide to Understanding Credentialing Concepts, Cynthia C. Durley, M.Ed., MBA

[4] Certification: A NOCA Handbook, Edited by Anne H. Browning, Alan C. Bugbee, Jr., and Meredith A. Mullins

[5] Certification: A NOCA Handbook, Edited by Anne H. Browning, Alan C. Bugbee, Jr., and Meredith A. Mullins