How EDi credentials are assessed

Every scored lesson is a work task set in a fictional organisation: a brief, a set of source materials (a policy, an email thread, a spreadsheet, a report) and an AI assistant the participant works with in a chat. The participant produces a deliverable they would stand behind at work.^[14]

1. 1

   Work with a real AI

   The assistant is a real model, not a script. It is kept blind to the planted problems and the scoring notes, so the participant gets no help it would not get at work.

2. 2

   Catch what it gets wrong

   The materials hold planted problems: a figure the AI will misstate, a rule it will overlook, sensitive data it should not be given. Lures look suspicious but are correct, so flagging everything does not pay.

3. 3

   Scored, then questioned

   The transcript and deliverable are scored against the rubric by an automated judge, run several times and reconciled. On challenge lessons the participant then answers questions about their own work, with it out of view.

Whether the participant notices, checks and corrects what the AI gets wrong is the thing being measured. Each own-words question asks for a specific value from the record; answers that do not match it lower the verification score. Borderline results go to a person.

Each attempt is scored on eight criteria at four levels. Four describe how the participant worked with the AI, four how they reviewed the result. Two scores come out of this, working with the AI and reviewing the result, each 0 to 100, with the overall score as their mean.

A credential is earned by passing every lesson of its level. A lesson passes when its score of record meets the pass mark and, where set, its hurdles.

The level's challenge lesson, taken under assessment conditions, is judged on both dimensions: working with the AI and reviewing the result must each meet the certification's mark, and every planted problem must be caught. Participants are also expected to show the standard on their own: the mean of first attempts across the level must meet a floor, and a dimension score within a few points of its mark, over or under, is reviewed by a person before the credential is issued or refused. An attempt the integrity checks flag waits for a person as well.

Each certificate records which rule produced it.

3 scored attempts

per lesson on credentialed levels; further attempts are practice

24 hours apart

between scored attempts

A fresh scenario

every retake: same objectives and planted structure, new story

A second attempt is a second demonstration, not a second look at the same page. Practice attempts are built, scored and fed back in full, but sit outside the score of record. The entry-level credential, AI Baseline, keeps unlimited scored attempts with the best one counting.

The rationale: a bound on scored attempts keeps a pass from being the product of persistence alone, the gap between them gives learning a chance to happen,^[2] and fresh scenarios mean memorising a layout earns nothing. Practice stays open because retrieval practice with feedback is itself one of the most reliable ways to learn,^[12] and the certificate separately records how many attempts were used and the first-attempt score, so a reader can see how the standard was reached.

Credentials with an expiry date are renewed by passing the level's challenge lesson again. Holders are told 60 days before expiry; the retake is free, earlier attempts do not count against the limit, and a pass issues a new certificate under the rule in force at the time. Nobody is required to renew: an existing certificate stands until its expiry date.

Two versions are stamped on every certificate and never change afterwards: the rubric version (what was scored and how the levels are defined) and the decision-rule version (how scores became a pass). A certificate issued under v1 remains a valid v1 certificate; issued credentials are not re-scored or re-decided when the rule changes. Lessons passed before a rule change keep counting; new scored attempts follow the rule in force from that moment.

Holders of a v1 credential are offered a free challenge retake under v2; the v1 credential is not withdrawn.

We treat the assessment as a measurement and check it as one, following the argument-based approach to validation: state what a pass is meant to show, then collect evidence for each link in that claim.^{[1, 9]}

The design choices above draw on established measurement practice and on research into how people work with automated systems. These sources support the design; evidence that EDi's own scores mean what they claim comes from the evidence program above.

Validity as an argument

A score is valid for a stated use when evidence supports each step from the performance to the decision. The evidence program is organised that way.^{[1, 9, 10]}

Realistic work tasks

Performance on authentic tasks, with real materials and a real tool, represents the skill more fully than recall questions about it.^{[10, 14]}

Why checking is the core skill

People over-trust automated output, and AI help can lower the quality of work on tasks the AI handles badly. Catching AI errors is therefore what the credential certifies.^{[4, 5, 11]}

Planted problems and lures

Catching real problems while leaving correct details alone is a detection task. Counting both hits and false alarms separates real discrimination from flagging everything.^{[6, 13]}

An AI judge, run several times

Language-model judges agree well with people on many tasks but show known biases, so judgements are repeated, reconciled and checked against human scores.^[16]

Checking the judge against people

Automated scores are evaluated by exact and adjacent agreement with human raters, compared with how well human raters agree with each other.^[15]

Hurdles and pass marks

Some requirements are conjunctive by design: strength elsewhere should not compensate for missing them.^{[3, 7]}

Spaced, bounded retakes

Spacing between attempts improves learning, and practice with feedback strengthens it; bounding scored attempts keeps the pass decision meaningful.^{[1, 2, 12]}

Certification structure

Published scope, a documented decision rule, versioned criteria, renewal and a public verification page follow the elements expected of a scheme that certifies persons. EDi is not accredited to this standard.^[8]

1. 1.American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for Educational and Psychological Testing. Washington, DC: AERA. www.testingstandards.net/open-access-files.html
2. 2.Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380. doi.org/10.1037/0033-2909.132.3.354
3. 3.Cizek, G. J., & Bunch, M. B. (2007). Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. Thousand Oaks, CA: Sage. doi.org/10.4135/9781412985918
4. 4.Dell'Acqua, F., McFowland, E., Mollick, E. R., et al. (2023). Navigating the jagged technological frontier: Field experimental evidence of the effects of AI on knowledge worker productivity and quality. Harvard Business School Working Paper 24-013. doi.org/10.2139/ssrn.4573321
5. 5.Goddard, K., Roudsari, A., & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. doi.org/10.1136/amiajnl-2011-000089
6. 6.Green, D. M., & Swets, J. A. (1966). Signal Detection Theory and Psychophysics. New York: Wiley.
7. 7.Haladyna, T., & Hess, R. (1999). An evaluation of conjunctive and compensatory standard-setting strategies for test decisions. Educational Assessment, 6(2), 129–153. doi.org/10.1207/S15326977EA0602_03
8. 8.ISO/IEC 17024:2012. Conformity assessment: General requirements for bodies operating certification of persons. Geneva: International Organization for Standardization. www.iso.org/standard/52993.html
9. 9.Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. doi.org/10.1111/jedm.12000
10. 10.Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. doi.org/10.1037/0003-066X.50.9.741
11. 11.Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. doi.org/10.1177/0018720810376055
12. 12.Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. doi.org/10.1111/j.1467-9280.2006.01693.x
13. 13.Stanislaw, H., & Todorov, N. (1999). Calculation of signal detection theory measures. Behavior Research Methods, Instruments, & Computers, 31(1), 137–149. doi.org/10.3758/BF03207704
14. 14.Wiggins, G. (1990). The case for authentic assessment. Practical Assessment, Research, and Evaluation, 2(2). doi.org/10.7275/ffb1-mm19
15. 15.Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. doi.org/10.1111/j.1745-3992.2011.00223.x
16. 16.Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track. arxiv.org/abs/2306.05685