Researcher preview of the Study 2 pilot, one sample problem

Try First, Then Ask: what students see and what we record

Pick a condition and work the problem on the left. The right side shows the data the website records. In the study, researchers assign each student one link to one condition. Students see only the student screen. They do not see these condition tabs or the researcher view.

Prior-knowledge checkShort quiz and AI habits

Class task[Six] problems like this one, in one condition

After the taskCorrect answers after each problem, then a metacognitive self-esteem measure

One week laterUngraded quiz in class, no AI

Student screen

Problem 1 of [6], independent-samples t-test

Scenario. A researcher compared test scores for two groups of students. Group A studied with practice questions. Group B reread their notes. Does the average score differ between the two groups? Use the information cards to answer.

Information cards

Click a card to read it. One card shows at a time.

How the four conditions differ

	No AI	Immediate AI	Try first	Scaffolded AI tutor
Problem and cards	Same scenario, same six cards, same questions
AI available	Never	Already showing with the problem	Appears after a first answer	From the start, as a tutor panel
What the AI gives	Nothing	Full worked answer	The same full worked answer	Feedback on each step, hints, never the final answer
Student must do first	All of the work	Nothing	Submit an answer and a one-sentence reason	Reply to the tutor's questions
AI type	None	Fixed text, generated and checked in advance	Fixed text, generated and checked in advance	Live LLM through an API
Extra data	Gives the no-AI baseline	Cards opened after seeing the AI, which shows checking	First answer, reason, final answer, whether it changed	Typed messages, step accuracy, hints used

Four sample students, same scoring

Invented examples to show the calculation. The no-AI baseline of 5 cards is a placeholder. In the study, it comes from the real no-AI group on each problem.

What the website records

Variable	What it means
`card_clicks`	Every card click, including repeats
`cards_total`	Different cards opened on the problem (0 to 6)
`card_seconds`	Seconds each card stayed open, card by card
`cards_before_ai`	Cards opened before the AI answer appears or the first tutor message
`cards_after_ai`	New cards opened afterward, which shows checking
`steps_saved`	(No-AI mean cards − cards_total) ÷ no-AI mean cards
`ai_opened_sec`	Tutor: seconds until the first message
`first_answer`, `final_answer`	Try first: both answers
`try_first_reason`	Try first: the one-sentence reason, coded later with ICAP (Chi & Wylie, 2014)
`answer_changed`	Try first: final answer differs from first answer
`tutor_messages`	Tutor: students' typed messages, coded later with the Study 1 codebook and with ICAP (Chi & Wylie, 2014)
`tutor_steps`, `hints_used`	Tutor: steps right or wrong, and hints needed
`accuracy`	Questions correct out of 3
`decision_confidence`	Confidence that the answer is correct, 0 to 100
`metacognitive_self_esteem`	[Six] items after the last problem, [1 to 7]. Samples: "I am good at thinking," "I feel good about my ability to think through problems"
`time_on_task`	Seconds on the problem

The website only saves the text. Trained coders code students' messages and reasons after data collection. The website does not code anything itself.

One week later

Students take an ungraded quiz in class through a lockdown browser. There are no cards and no AI. All items are new, and none repeats a practice problem. Each item has a 0 to 100 confidence rating.

Outcome	Sample item
Memory ([5] items, one per key concept)	What does Levene's test check? When do you read "Equal variances not assumed"? What does a p value mean? What p value makes a difference statistically significant? What do you conclude when p is .05 or higher?
Application ([5] items)	A new SPSS output and scenario: which row do you read, what is the p value, and what do you conclude?

Calibration = confidence − accuracy, averaged across items
Example: 90% confident on an item answered wrong: 90 − 0 = +90 (overconfident)
Example: 40% confident on an item answered right: 40 − 100 = −60 (underconfident)

Calibration uses the quiz and not the class task, because task accuracy can reflect help from the AI.

Prototype only. The sample output is computed from the means and SDs shown, except Levene's F and p, which are example values. The tutor replies here are scripted around four steps. For the pilot, API calls will make the tutor adaptive, with the same rules: give feedback on every reply, never repeat a question word for word, and never state the final answer. Study data will be stored on secure UM systems, and AI will run on a local LLM or a UM-approved setup. No data leaves this page.