Write a test for Harman

Harman is a set of written review methods that a language model reads before it answers. It was written for verbal and social-science reasoning. It asks questions like these: are two positions really opposed, or do they answer different questions? Are sources that say the same thing really independent of each other? Did a summary drop information that would change a decision? Is a piece of work that is called “done” actually done?

There is no ready question set that can test Harman properly. So we cannot currently test whether Harman makes a model better; we have not shown that it does. This page explains why, and what kind of test you could write for us.

Why we cannot test it

• Existing sets ask a different question. We screened 599 open sets that measure verbal reasoning. For 38 of the 66 kinds of decision Harman makes, we found no set at all. Most of the sets we did find either lack the input Harman works on or have correct answers that are not the kind of decision it makes. For example, testing whether sources are really independent needs one claim and several separate texts reporting it; critical-question sets have a single text.
• Models solve the sets we built without the method. We built six sets. To keep the correct answers beyond dispute, we kept only clear cases, and on four of the six sets strong models scored near the ceiling without the method. On such a set, a contribution from the method cannot show.
• Experts can give the correct answers to hard cases. The method could make a difference on borderline cases. Their correct answers have to come from people who know the field and are independent of us. We do not have such labels; that is what we are looking for.

Tests we are looking for

Each card gives a question, what one item looks like, who could provide the correct answer, and how many items are needed. Item counts are rough lower bounds; assumptions are below.

1. Are the two sides really opposed?

Item: a one- or two-sentence “X or Y” tension. Correct answer: one of five classes: a real opposition (one side makes up for what the other lacks) · two equivalent descriptions of the same thing · different levels or different questions · hidden options (a false dilemma) · no opposition. Labelled by two graduate researchers in philosophy or the social sciences. At least 30 per class, at least 150 in total. Known nearby sets label only false dilemmas (LOGIC, MAFALDA, CoCoLoFa, Faina).

2. Are sources that say the same thing really separate?

Item: one claim plus 3–6 texts that report it, with outlet names removed. Correct answer: for each text, was it derived from a shared source text or written independently; from this, the number of independent sources. Labelled by journalism researchers. At least 150 claim clusters. Nearby sets: METER (news-agency copy and newspaper stories), health news and press releases (MacLaughlin et al. 2020).

3. Is a term defined well enough to tell experiments apart?

Item: a research proposal plus 3–5 marked terms plus the planned measurements. Correct answer: for each term, which planned measurements this definition makes unnecessary (or none); if it stays vague, how many measurements would be wasted. Labelled by two researchers who have taught methods in that field. At least 150 terms.

4. What did the summary drop, and would it change a decision?

Item: a meeting transcript plus its summary. Correct answer: the dropped sentences, which of them change a decision, the current state or an open task, and whether the summary contains the decisions, their reasons and the open tasks. Meeting corpora with decision and action-item labels (AMI) are a possible start. At least 100 meetings.

5. Was written work that is called “fixed” really fixed; are open tasks named?

Item: a critique plus the “fixed” reply plus the new version of the text; or the “open items” section of a report. Correct answer: does the edit address the critique; for each open task, is the concrete observation that would close it written in one sentence. Nearby sets: ARIES (review comment ↔ edit), RevUtil (actionability of review comments). At least 200 items.

6. Is the conclusion broader than its evidence supports?

Item: a source (paper abstract) plus a derived text (press release, summary); or a study design plus its conclusion sentence. Correct answer: is there exaggeration (scope, causation, animal-to-human). Nearby sets: Wright & Augenstein 2021, Sumner et al. 2014, Yu et al. 2019, Peters & Chin-Yee 2025. At least 150 items.

7. Which stage of an inquiry does new evidence reopen?

Item: a one-to-three-page inquiry file (definitions, oppositions, evidence, failed attempts, open questions) plus one new observation. Correct answer: the stage or stages that must be reopened, or none; observations that change nothing must be included. At least 150 items, at least 50 of which change nothing.

8. Does a proposal stay in thought, or is it tied to a specific practice?

Item: a one- or two-page interpretation or proposed solution. Correct answer: three yes/no questions: does it present material conditions as a defect to be overcome; is the change tied to a specific practice, device or institution where it would be carried out or tested; does it say where and how it would be tested. Best format: pairs of “decorative” and “repaired” versions of the same text. At least 60 pairs.

9. How many separate tasks does a request contain; is it out of scope or unclear?

Item: a one-to-five-sentence user request, or a half-page research plan. Correct answer: the list of separate tasks (for a plan: separately testable causal claims), or “out of scope”, or “unclear: the missing question is this”. Caution: coders agreed on the number of hypotheses in preregistrations only 14% of the time (Bakker et al. 2020); test the agreement of your labelling guide first. At least 150 items.

10. How good is a proposed solution when there is no correct answer?

Item: a dilemma plus two proposed solutions (one from the model with the method, one without; which is which is hidden). Measure: blind expert preference and criteria written in advance (does the proposal really rule out an option; does it say which observation would prove it wrong; does it name what it keeps and what it loses). At least 3 experts per pair; about 195 pairs to detect a 60% preference, about 85 for 65%.

11. Does a finding carry over to a new sample and setting?

Item: the summary of an original study, with the replication result hidden. Correct answer: the outcome of large replication projects. Replications published after the model’s training date are especially valuable.

How we will run your test

• The same model runs three ways: without the method, with the method, and with a general instruction of equal length that contains no method. Two models of different sizes are used; results are reported per model.
• Each item is a separate call; the call does not see any other item, any label, or any other arm’s output.
• If a judge is needed, it comes from a model family other than the tested model’s. The judge does not know which arm’s answer it is reading; on 20 items we ask it to guess the arm and report how often it is right.
• The measure, sampling and decision rule are frozen before the first answer is scored.
• Decision rule: if the 95% interval of the difference between the method arm and the no-method arm lies entirely above zero, we say “the method adds value”; if the difference from the equal-length general-instruction arm also lies above zero, we say “the gain comes from the method”. These are two separate statements.
• If, in the pilot, the no-method model is within 5 points of the maximum score, the set cannot measure the question; we then choose a harder subset or a weaker model.

Assumptions for item counts: two-sided α = 0.05, power 0.80. For binary labels, comparing the method and no-method arms on the same items, detecting a 10-point difference needs 155–235 items and a 15-point difference 70–105 items.

How to send a contribution

A contribution contains:

1. Items: the input texts, each with an ID.
2. Correct answers, in a separate file. If you prefer, do not send us this file: send only its sha256 hash and keep the answers yourself. We send you the model outputs and you do the scoring. This shows that we never saw the answers.
3. A labelling guide (one page is enough) and labeller agreement: at least two independent labellers, Cohen’s kappa.
4. Source and licence. A short statement that the items contain no personal data.
5. Optionally, a 20-item pilot: we use it first to check whether the set can measure the question.

We are looking for verbal and social-science items rather than code or software-agent tests.