Human Review of AI Case Decisions

Study Overview

This study examines how people understand AI judgments and the explanations that accompany them. We are interested in whether readers find those materials clear, reasonable, and sufficient to support their own judgment when the system explains how it reasoned, what information it relied on, and how it compared the case against the rules. The study is not testing technical training or specialist knowledge; it focuses on how ordinary readers interpret and respond to the materials they see.

Research Problem

Many AI systems can produce answers quickly, but people may still not know what evidence supported the decision, whether important information was missed, or whether the conclusion is actually well grounded.

Research Focus

The goal is not only to ask whether an answer is correct. The study also examines whether an AI explanation truly helps people understand the result, form a judgment, and respond to it. In other words, we are not only interested in whether information is provided, but whether that information is meaningful and useful to the reader.

Example Source

The cases in this study are adapted from Nike Hong Kong's public online return and refund rules. They are research cases and do not contain real customer data.

What Each Case Contains

Each case includes the case facts, the relevant rules, the Initial AI decision, and the Review AI analysis of that decision.

Your Role

You do not need to act as Nike staff and decide the case from scratch. Your role is to judge whether the Review AI explains its reasoning clearly enough, whether the reasons are adequate, whether the analysis truly follows the rules and facts, and whether you would accept that explanation.

What You Will Answer

During the study, you will read short case materials and answer a few simple questions based on your understanding, such as whether the information is clear, whether the explanation is sufficient, and whether you accept the judgment.

How Your Responses Help

Your responses help researchers understand how ordinary readers interpret, experience, and respond to AI judgments and explanations. They also help identify which parts feel clear, where the explanation becomes confusing, and when a case should be handed to a human for further checking.

Summary

In short, the study is not only about whether an AI can give an answer. It asks whether people understand, trust, and use the answer more readily when the AI makes its judgment process visible.